This paper revisits the task of end-to-end chart data extraction, aiming to simplify the input representation for more effective visual understanding and reasoning in chart understanding tasks. We introduce ChartLite, a streamlined dataset that focuses only on the essential chart elements needed for reasoning, in contrast to existing datasets that often include redundant or trivial annotations. With this dataset, we evaluate a range of models capable of converting chart images into structured JSON outputs without relying on OCR. Our experiments span multiple types of charts, including vertical bar, horizontal bar, dot, and line charts, and assess the capability of the model to extract key components such as chart type, bounding boxes, and element roles. The results highlight both the strengths and limitations of current approaches, particularly in handling complex visuals and varied textual content. We conclude with observations on current model capabilities and outline directions for improving chart understanding systems through better architectural design and task-specific training strategies.