Introduction <p>The importance of respectful maternity care on optimal maternal outcomes is increasingly acknowledged globally. However, mistreatment and abuse are still experienced by women during hospital childbirth in many parts of the world, with sub-Saharan Africa being one of the places where it is most common. Interventions aiming to improve respectful maternity care must be able to assess the prevalence of responsiveness and mistreatment women experience. This is usually done with questionnaires, though these are not always validated. Scores to represent the level of responsiveness and mistreatment experienced can be created from questionnaire results and have many uses, though no score is consistently used in this field. A new questionnaire measuring responsiveness and respectful treatment was developed for use in the ALERT project, as a questionnaire covering both of these concepts did not exist. This study aimed to validate this questionnaire and to create a scoring method.</p> Methods <p>Psychometric analyses, including exploratory and confirmatory factor analysis, were performed on cross-sectional data collected from the ALERT study to identify and confirm underlying factors. Using these factors, simple summation and factor-weighted methods were used to create scores and their results compared.</p> Results <p>Six factors were identified: “Communication &amp; supportive care”, “Hospital environment”, “Maintained respect &amp; dignity”, “Social support”, “Maintained privacy &amp; confidentiality” and “Lack of physical &amp; verbal abuse”. The results of the two scoring methods developed were similar.</p> Conclusions <p>The responsiveness and respectful treatment questionnaire has high validity in the ALERT study population for the six factors identified. The two scoring methods developed are useful for different aspects of the ALERT intervention and can be used to facilitate comparisons or measure progress towards improving respectful maternity care in these settings.</p>