The Order of Merit Method and the Method of Paired Comparisons
Mabel Barrett
In this experiment, the order of merit method and the method of paired comparisons were applied to three series of materials involving judgments of varying subjectivity. The three series consisted of (1) weights, to be judged with respect to their heaviness, (2) specimens of handwriting, to be judged with respect to their excellence, and (3) propositions of varying validity, to be judged with respect to the subject's degree of belief in the fact stated.
The results were used as data by which to compare the relative efficiency of the two methods with regard to statistical investigation of judgment. Seven main problems are suggested, each of which involves a basis of comparison between the two methods.
I. The variability of each specimen in the series from the average position accorded to that specimen, and the consequent average
( 383) variability of the series. In the case of weights, this average variability is, by the order of merit method, slightly greater than by the paired comparisons method, and in handwriting judgments the exact opposite is true. These averages in isolation might indicate that the one method is particularly favorable to judgment of weight, the other to judgment of handwriting—or the one method to the one group of subjects, and the other method to the other group. These hypotheses are, however, invalidated by the exceedingly high correlation between the two methods for any one type of judgment, and by a comparison of the variabilities in handwriting and beliefs, where the judgments were performed by the same group of subjects. The average of these variabilities for the three types of judgment shows a difference of only .02 between the two methods. The differences in isolated cases may be due to the materials themselves or to the groups themselves apart from any consideration of method. They are very evidently not due to the methods.
II. The second problem is the correlation of the average order with the objective order of the series, by the two methods. In judgment of weights this correlation is exactly the same, and in hand-writing almost exactly the same for one method as for the other. The difference in the latter case is only .003. In the case of beliefs there is no objective order.
III. The correlation between the arrangements of a given series by the one method and by the other averages .987 for the three types. This indicates that it is unnecessary to employ either one of these methods, which for any reason is less to be preferred, if we consider them with respect to the general results obtained by both.
IV. The individuals of the group correlate as well with their average in the one method as in the other. The differences between the average correlations by the two methods lie in every case within the limits of the probable error.
V. An individual who stands high in correlation with the group arrangement by one method also tends to stand high in that correlation by the other method. This relation is expressed by the correlation + .72 in the ease of handwriting and beliefs. In the case of weights, the relation is a random one, +.01. The individual differences in correlation with the average, are, by the paired comparisons method, so insignificant as to make the order of correlations subject to chance and very unreliable.
VI. The order of merit method shows a random relation (–.01) between an individual's judgment of handwriting and the same individual's judgment of beliefs. This result accords with the results obtained by other investigations of this sort of problem. In the paired comparisons method this correlation is expressed by –.35.
( 384) This represents the first and only discrepancy between the equal efficiency of the two methods in this experiment.
VII. A comparison of the groups which performed the one method first with the groups which performed the other method first shows that the method which is employed first does not tend in any way to improve the judgments made by the method which follows it a month later,
On the basis of the efficiency of the two methods for statistical investigation of judgment we may conclude that the one method is in no way to be preferred to the other. From the point of view of convenience, labor, and time required, the order of merit method is by far the more satisfactory of the two.
