A chest X-ray image report includes a text description of the abnormalities and normalities of a patient lung, heart, and chest. The report is an image-to-text translation based on the radiologist’s diagnosis of the X-ray view. It helps to communicate the diagnosis to no radiologist or medical experts and track the patient’s health condition in the future. The manual writing of reports is a tedious task that consumes time and is vulnerable to possible human errors. In this work, we propose a pipeline for building based-transformer models to process the task of automatic report writing. We included the experiment that justified the choice of the models and configurations of the feature extractor encoder, transformer attention layers, and training. We show that: 1) First, the pre-trained ViT feature extractor performs better than the CNN-based encoder (DensNet121). 2) Second, the dual-view input (Frontal and Lateral images) give better result than one input (Frontal or Lateral).