Exploring Differences in Value Functions Allowed by Ordinal Validation

Decision-based design often states an aspirational goal for value functions of achieving perfect ordinal consistency. How reliably such a standard can be achieved, however, is rarely addressed. Due to the multiple options available to an engineer regarding model form, model fitting procedures, training data, etc., there are often multiple value functions which could be developed for any particular problem. The extent to which those functions can be distinguished from one another depends on the exact validation procedures used to determine acceptability. This work utilizes a space launch vehicle simulation model to generate outcomes for the comparison of value functions. A training set of 12 outcomes are rank-ordered, and 250 models which produce the correct order of those 12 outcomes are generated. Relative preference of 2 separate alternatives is then compared across all 250 acceptable functions. This comparison is made with both certain and uncertain outcomes associated with the alternatives. In the base condition with certain outcomes, 64% of the models preferred Alternative A and 36% preferred Alternative B. With uncertain outcomes, relative preference depended on both the shape of the resulting distributions as well as the decision criterion used to characterize the distributions. These results demonstrate that functions which produce the same rank ordering of a training set are not guaranteed to have full ordinal consistency, highlighting the importance of validation procedures in engineering value modeling.