Brain imaging offers remarkable opportunities to study the human brain and yet is of limited utility in the everyday practice of clinical psychiatry. Clinical heterogeneity is one of the main obstacles preventing the translation of brain imaging from research to a clinical tool. There are obvious and marked differences among people with the same diagnosis of bipolar disorders (BD) and yet, brain imaging studies still typically search for the mythical average patient. Some brain imaging changes in BD reflect genetic risk, some are consequences of illness episodes or treatment, and others reflect the presence of comorbid conditions. A combination of these extra-diagnostic factors creates a unique set of brain alterations in a given individual with BD. I review research investigating these extra-diagnostic variables, including genetic liability, metabolic comorbidities, or medications as risk or preventive factors for brain changes in BD. The picture that is starting to emerge from brain imaging studies in BD is one of marked structural plasticity. Many of the brain changes found in adults with BD are not pre-existing. They emerge only later in the course of illness and may be made worse by medical comorbidities such as obesity or better by treatment with lithium (Li). Understanding neurobiological differences between individuals with the same diagnosis may ultimately teach us more about BD than looking for a one-size-fits-all brain change.