Professor Cattell's Studies by the Method of Relative Position
Harry L. Hollingworth
IN the physical and exact sciences, in which the phenomena to be measured are usually in themselves quantitative in character, measurement is facilitated by the development of standardised spatial and temporal scales. When the facts to be measured are in themselves spatial and temporal changes, these scales may be applied directly and quantitative treatment is at once possible. In other cases the phenomena may be indirectly measured by means of correlated spatial and temporal changes.
In economic science, and in the processes of exchange, where the phenomena to be measured are not spatial or temporal facts, but are rather of the nature of subjective values, measurement and quantitative treatment are made possible by the development of a medium of exchange, to the units of which are attached values which parallel those of the facts to be measured.
In the biological and social sciences, when quantitative treatment is at all possible, it is either of spatial or temporal changes directly or is in terms of the frequency of occurrence of qualitative facts which are in themselves determinable and identifiable.
The psychologist finds it necessary to deal with phenomena which are neither directly nor indirectly spatial or temporal, but are essentially of an intensive or qualitative character, and which can not be easily related to objective scales. In this science measurement first began in the case of those experiences which have as their immediate cause or correlate external series of facts either in themselves immediately spatial or temporal (as in subjective estimates of duration, or of linear magnitude), or series of facts which could be themselves measured by the indirect use of objective standards (as in judgments of weights, lights, sounds, etc.) The quantitative treatment of these psychological processes led to that branch of experimental psychology which is known as psycho-physics. Closely related to psychophysics are the psychometric measurements which are interested in the temporal relations of processes which are in other respects not measured.
Of much greater difficulty is the measurement of psychological facts which are neither in themselves immediately quantitative,
( 76) nor yet directly related to phenomena which can be quantitatively described. Affective experiences such as pleasantness, preference, interest; more complex facts such as feelings of similarity and difference, confidence, belief, conviction; social or personal evaluations such as eminence, usefulness, character; literary and artistic facts such as beauty, eloquence, style; educational facts such as legibility of penmanship, ability in drawing, excellence in composition; or traits which have high industrial or commercial importance such as skill, leadership, zeal, persuasiveness; all these are experiences or traits for which it is highly desirable to develop precise methods of measurement, analysis and description. Such facts as these are distinctly psychological, either in that they are known only to the immediate experience of an individual and show large dependence on individual differences (as preference, beauty, etc.) or in that their quality arises directly out of their effect on the consciousness and behavior of other individuals, this effect not being susceptible of spatial and temporal measurement. They are, in other words, subjective facts, for the measurement of which objective scales either can not or do not exist.
There is also another type of subjective fact represented by graded series of stimuli in which the steps are so small that different individuals' judgments or the same individual's judgments on different occasions differ measurably with respect to the position to be assigned a given item in the graded scale. In this case there may or may not exist quantitative methods of determining the objective relations of the different items, but in any case the diverse judgments do not always agree with the known relations. Psychologically the items take their place in a qualitative series, and in an order which varies with the occasion of the determination.
An early contribution of Professor Cattell, and one which has proven of unusual value to psychological procedure, presented a method of converting such a qualitative series into a series of quantitative differences. It is obvious that a method of treating facts which are subjective chiefly because of their imperceptible differences will be serviceable also in the measurement of facts which are subjective for widely different reasons.
Thus “if one endeavors to arrange and rearrange in serial order a number of given objects, the positions successively given them will vary somewhat as they would vary if the arrangements had been made one each by different observers. If we undertook to arrange ten times a series of grays in order of brightness, we should no more get the same order each time than we should get identical orders from ten different subjects. . . . The judgments of the
( 77) same individual at different times are theoretically quite comparable to those of different individuals regardless of the factor of time.
"In this way may be illustrated a continuum between the subjective and objective classes of judgment. In the cases of grays, weights or lines, we assume a certain standard which we term the objective order, and which we determine through photometry or some analogous method. Because we have such methods we do not need to have recourse to individual judgments to determine objective values, and these individual judgments give us a part of the personal equation . . . . On the other hand we have such subjective judgments as preferences in sculpture, painting or music. In the first class we may arrange individuals in precise order for accuracy of discrimination; in the second, one may with equally good taste vary his preferences within a considerable range. So far as any distinction on a statistical basis is possible, we might consider as subjective those types in which the various judgments of the individual formed a species of their own, varying from each other considerably less than from an equal number of judgments made by different individuals; and consider as objective those in which an individual would vary from his own independent judgments about as much as the variation of an equal number of judgments by different observers. . . . The two categories would almost certainly be continuous."[1]
Since Professor Cattell's original account of his first application of the method of relative position is relatively inaccessible to the general reader, that account is here reproduced. It consists of that portion of the larger article [2] relating to the use of the order of merit method.
"In the experiments on the intensity of light, I used various methods. These included a method of securing a given physical difference by altering the angle of incidence of the rays of light, which constitutes, I believe, a new form of photometer, having certain advantages. I found it, however, most convenient to use gray surfaces, reflecting a known percentage of the light. These surfaces were washed with Indian ink. The ink was made as black as possible, and one piece of drawing paper was washed; then a drop of water was added to the ink, another piece was washed, and the process was continued until the wash became imperceptible. In this way 211 shades between black and white were obtained.
( 78) These grays (in cards 5 cm. sq.) were then tested by various photo-metric methods. The series proved to have a fairly regular gradation, except that the steps increased toward the white end, the gray 95 steps from white being midway between white and absolute black.
"Such a series of grays (a hundred grays forming equal physical steps could be selected that would be preferable to this series of 211 cards) is suitable for testing the accuracy of discrimination and its variation under different circumstances, as with the intensity of stimulus, in memory, etc. The physical differences are smaller than can be perceived, and if an observer arranges the grays as nearly as possible in the order of brightness, errors occur which measure the accuracy of discrimination. Thus in ten trials—the first two by the present writer and the others by eight different observers—the errors of displacement were as follows: 6.63, 7.97, 6.04, 6.2, 7.44, 7.81, 8.29, 9.68, 11.5. The arrangement can be made in about an hour, and, as there are over 200 separate judgments, the average has a probable error of only about 0.5. Observers differ within the extremes of about 1: 2 which agrees with other determinations. For the more accurate observers the error is six cards, or about 1/35 of the range between black and white. The degrees that appeared just noticeable when these were selected were between 13 and 32, but errors occurred in the case of the larger number.
"When the results of the ten series are combined and the relation of the error of displacement to the intensity is considered, it is found that while the error increases with the stimulus, it does so more slowly than required by Weber's law. The average displacement for each of seven groups of 30 cards (allowance being made for the longer steps between the brighter cards) was, approximately, 13.61, 10.6, 12.14, 10.03, 6.46, 5.09, 2.98.
"In the table are given: (I.) the approximate percentage of light reflected from the middle gray in each of the seven groups, that is the magnitude of the stimulus; (II.) the error of observation in terms of percentages of light between white and absolute black; (III.) the error of observation in terms of the magnitude of the
( 79) stimulus to the nearest whole fraction; (IV.) the error of observation in terms of percentages of the magnitude, and (V.) the error of observation in terms of percentages of the square root of the magnitude.
"It appears, consequently, that as the stimulus increases the error of observation increases, but not so rapidly as the stimulus, being about 1/7 with the weakest and 1/16 with the strongest stimulus. It increases much more nearly in direct proportion to the square root of the stimulus, in accordance with the hypothesis I have proposed in place of Weber's law. This hypothesis—‘The error of observation tends to increase as the square root of the magnitude, the increase being subject to variation whose amount and cause must be determined for each special case’—has not yet received the consideration which the experimental evidence and theoretical explanation seem to me to warrant."
In a later reference [3] we find these experiments succinctly summarized in the following words:
"Some two hundred shades of gray were made, giving approximately equal differences in illumination between white and black. In such a series the grays toward the white end appear more alike than those toward the black end, and two adjacent grays are indistinguishable. Psychologically it is a qualitative series. If now the grays are arranged in the order of brightness a number of times by the same or different observers and the average position in the series of each gray is determined, the mean variation is inversely proportional to the psychological differences between the grays. There is thus determined the quantitative difference in the perception and its relation to the physical differences between the lights. The same methods have been used in the Columbia laboratory to measure the validity of beliefs, the beauty of pictures, differences in traits of character, literary skill and efficiency in various performances."
This use of the method of relative position in psychophysics represents its application to the measurement of facts which are subjective chiefly because of their small differences in perception. The judgments are objective, in the sense that their correctness is determinable by comparison with photometric or similar measurements. From this to the application of the method to the measurement of facts not determinable by such objective means, and subjective not only because of the amounts of difference presented, but more especially because of individual differences in the judg-
( 80) -ments of various observers, was a most natural step. Professor Cattell took this step when in 1906 he published his study of American men of science. (The study was begun in 1902 and a report of the data concerning the group of psychologists was published in 1903.) As characteristic of his point of view in psychology the introductory words to this report are of special interest.
"The psychologist, like the student of other sciences, can view his subject from different standpoints and pursue it by various methods. He may get what knowledge he can of mental processes by introspection, or he may use objective methods. He may confine himself to the `inner life,' or he may study the individual in all his psychophysical relations. He may give verbal descriptions, or he may make measurements. He may describe static mental life, or he may study the lower animals and human beings from a dynamic and genetic point of view. He may attempt to determine facts and laws that hold for mental life in general, or he may attend to individual differences. He may ignore the practical applications of his science, or he may investigate them. Psychology has until recently concerned itself chiefly with the first of these various alternatives. But its recent progress and future development seem to the present writer to depend particularly on the second. In this case, our two main methods, which can often be combined, are experiment and measurement in the laboratory, and the inductive and statistical study of groups of individuals. In recent years great progress has been made in both directions. Experimental psychology has become a science coordinate with the other great sciences, and statistics have been extended to include sociological and moral phenomena."[4]
As material for intensive studies of groups of individuals, and the nature and origin of their differences, three groups of people were selected. They were, (a) a thousand students of Columbia University, selected on the basis of their willingness to submit themselves, upon entrance and graduation, to a series of mental and physical tests; (b) the thousand most eminent men and women in history, selected on the basis of the amount of space given them in several standard biographical dictionaries and encyclopedias: (c) a thousand American men of science, selected on the basis of their ranking, by the method of relative position, in their respective sciences, at the hands of ten leading workers in each field. It is the study of this third group which is of interest in the present connection. It may be well to give a general account of this application of the order of merit method, in the form of selected quotations from Professor Cattell's own accounts of it.
( 81)
"Many of the problems that the writer had in view in the present research might be solved by the study of any group of a thousand American men of science, so long as they had been objectively treated. The objective selection of a group sufficiently large for statistical treatment is however essential. As cases can be quoted to illustrate the cure of nearly every disease by almost any medicine, so examples can be given in support of any psychological or sociological theory. The method of anecdote, as used by Lombroso, may be readable literature, but it is not science. A thousand names might have been selected by lot from all the scientific men of the country, assuming a list to have been available, but a group of the thousand leading men of science arranged in the order of merit has certain advantages. Information in regard to them can be better obtained than in the case of those who are more obscure. Correlations can be determined between degrees of scientific merit and various conditions. The comparison with a similar group selected ten or twenty years hence, or with a similar group of British, French or German men of science, would give interesting results. The list itself, if printed after an interval of twenty years, would be a historical document of value. Lastly, the data can be so used as to carry quantitative methods a little way into a region that has hitherto been outside the range of exact science. It is this last problem which I wish to take up in this paper."[5]
"In selecting a group of a thousand scientific men, the number in each science was taken roughly proportional to the total number of investigators in that science . . . . The individuals were selected by asking ten leading representatives of each science to arrange the students of that science in the order of merit. There were for each science slips made with the names and addresses of all those known to have carried on research work of any consequence. The total number assigned a position was 2,481, distributed among the sciences as follows: Mathematics, 201; physics, 261; chemistry, 389; astronomy, 165; geology, 257; botany, 213; zoology, 290; physiology, 101; anatomy, 89; pathology, 251; anthropology, 72; psychology, 192. These numbers included duplications when a man was given a place in more than one science....
"The ten positions assigned to each man were averaged, and the average deviations of the judgments were calculated. This gave the most probable order of merit for the students of each science, together with data for the probable error of position of each individual. The students of the different sciences were then
( 82) combined in one list by interpolation, the probable errors being adjusted accordingly. The list contains 1,443 names, of whom the first thousand are the material used in this research.” [6]
"The average of ten judgments is not necessarily more correct than any one of these judgments; the conditions are similar to observations in the exact sciences. One good observation may have more validity than the average of a number of observations made under less favorable conditions. But if ten scientific men concerning whose competence it is not possible to discriminate in advance make a judgment, we may take the average as the most probable value. If we had but a single judgment we should not know its validity, but with ten judgments the probable error can be calculated. These probable errors tell us not only the limits within which the place of an individual in the series is likely to be correct, but also measures the differences between the individuals."
"The method enables us to measure not only differences in scientific merit, but also the accuracy of judgment of those who make the arrangements. It would be possible to determine whether those more eminent would have the more accurate judgments, at what age the individuals are most competent and the like. As a matter of fact, the judgments in the present case were made by those most eminent in each science who were willing to undertake the task."[7]
The primary results of this study may be illustrated (a by the) data concerning the distribution of the leading thousand men of science; (b) by the account of the intensive study of the members of one of the scientific groups, the psychologists; and (c) by the comparison of the first arrangement with a second similar arrangement made after an interval of several years.
(a) "In the accompanying curve . . . is shown the distribution of the thousand men of science. The 1,000 scientific men are divided into ten groups, the range of eminence or merit covered by each hundred being proportional to the space it occupies on the axis of the abscissas, and the number of each degree of ability being proportional to the ordinates. The range of merit covered by each hundred becomes smaller and there are more of each degree cl merit as we pass from the first to the second hundred and so on ice the first five hundred, after which the differences become very small. The first hundred men of science cover a range of merit about equal to that of the second and third hundreds together, and this again is very nearly equal to the range covered by the remaining
( 83) seven hundred. The average differences between the men in the first hundred are about twice as great as between the men in the second and third hundreds, and about seven times as great as between the men in the remaining groups. Or the differences among the first hundred are almost exactly ten times as great as among the last five hundred, who differ but little among themselves. It would be desirable to compare this distribution with that of the probability integral and with the salaries paid to scientific men, but the data are not as yet at hand."[8]
(b) The study of the group of the fifty leading psychologists yields many results of immediate interest. It also illustrates the
<insert figure 1>
attempt to secure quantitative treatment, statistical analysis and mathematical exactness, so characteristic of all of Professor Cattell's work and teaching.
"In order to illustrate further the serial distribution and the probable errors, I have made a diagram for the fifty psychologists. The grade of each, no judgments being omitted, is shown by the
( 84) vertical mark, and the length of the line indicates the probable error or range within which the chances are even that the true position falls. Thus the psychologist who stands first on the list was . . . given this position by the independent judgment of all. The psychologist who stands second has, as shown on the diagram, a position of 3.7 and a probable error of 0.5, i. e., the position of 3.7 is the most probable, but the true position is equally likely to be within the short horizontal line, between 3.3 and 4.2, or outside it. It must, however, be remembered that the chances of the true position being far outside the range of this line decrease very rapidly. Over it is roughly drawn the bell-shaped curve of the normal probability integral. The true position is along the base line covered by this curve, and the chances of its being at any
<insert figure 2>
given point are proportional to the ordinate or height of the curve above the base line. There is only one chance in about six that the true grade is above 2.7 or below 4.7, and only one chance in about 150 that the true grade is above 1.7 or below 5.7. It will be seen from the diagram that while the positions of the psychologists II., III. and IV. are the most probable, the relative order is not determined with certainty. On the other hand the chances are some 10,000 to one that each of these psychologists stands below I. and above V."
"It is evident that the probable errors increase in size as we go down the list. The curve of distribution drawn over No. XL. indicates that the chances are even that the true position falls between the grades of XXXIV. and L. and that there is one chance in four that he does not belong among our fifty leading psychologists.
( 85) The increase in the size of the probable errors is irregular, it being more difficult to assign a position to some men than to others."
"It will be noted that the psychologists fall into groups, the first twenty being set off from the next group, though the two groups are bridged over by three cases. At this point also the probable errors become almost suddenly about three times as large. There are altogether about 200 psychologists in the country, and it looks as if the first tenth forms a separate group of leaders. There is a similar, though less marked group of the first twenty astronomers, but these groups seem to be partly accidental. There is, however, as shown below, an inflection point in the curve of distribution after about the first tenth of our scientific men. The first twenty psychologists fall into four distinct groups, and there are groupings in the other sciences. They do not, however, appear to be sufficiently marked to lead us to distinguish species, such as men of genius and men of talent. It is, however, possible that the complicated conditions may ultimately be analyzed so as to give such groups."
"The probable errors not only tell the accuracy with which the psychologists can be arranged in the order of merit, but they also measure the differences between them. This, indeed, I regard as the most important result of this paper, as science is advanced chiefly by the extension of quantitative methods, and it might not have been foreseen that it would be possible to measure degrees of scientific merit. Our data are concerned with the recognition of scientific performance, not with abstract ability, if such a thing is conceivable. Merit is in performance, not in non-performance, and expert judgment is the best, and in the last resort the only, criterion of performance."
"The difference in scientific merit between any two of the psychologists whose positions and probable errors are shown in the chart is directly as the distance between them and inversely as their probable errors. If two of them are close together on the scale, and if the probable errors are large, the difference between them is small, and conversely."
"If the psychologists II. and III. were separated by 0.5 and their probable errors were 0.5, as is approximately the case, then the difference between them is so small that there is one chance in four that the position of III. is above the grade of II. If again the psychologists XL. and XLIX. were separated by 6 and their probable errors were 6, as is again approximately the case, then there is again one chance in four that the true position of XLIX.
( 86) is above the grade of XL. The difference between II. and III. is thus about the same as that between XL. and XLIX."
"If we take the fifty psychologists in groups of 10, and thus partly eliminate the chance variations, the average probable errors of the five groups are 0.7, 1.8, 4.2, 5.8, 6.2. These probable errors are subject to a correction for the range covered by the grades. Thus the first ten cover a range of about eleven points, and the last ten a range of about six points, and the differences between the psychologists at the top of the list would be nearly twice as great as between those at the bottom of the list if the probable errors were the same. When we take account of both factors, the probable errors in the five groups are 0.6, 1.9, 1.8, 6.4, and 10.7. While the probable errors are determined with a considerable degree of exactness, which is itself measured, the ranges covered by the grades seem to depend on the special conditions in the science; they are not the same in the different sciences, and their validity cannot be determined with any exactness. Subject, however, to a considerable probable error, the range of merit covered by the fifty psychologists is inversely as the figures given, and reduced to a scale of 100 would be: 55.6, 17.5, 18.5, 5.2 and 3.2."
"Thus we can say that the psychologists at the top of the list are likely to differ from each other about 18 times as much as the psychologists at the bottom of the list. We have no zero point from which we can measure psychological merit. Men who are 6 ft. 2 in. tall are likely to differ from each other about ten times as much as men who are about 5 ft. 8 in. tall, though the difference in their height is only as 68 :74. Even though we assumed the zero point to be where psychological performance begins or at the survival minimum of human ability, we should only obtain relative differences."[9]
(c) About seven years after the selection of the group of scientific men "it seemed desirable to repeat the process of determining the thousand leading scientific men in the United States. It is worth while to learn what changes have taken place in the composition of this group and in the distribution of the scientific men among various institutions and in different parts of the country. A list of scientific men as nearly contemporary as might be was alsowanted for some further studies of the conditions of heredity and environment which are favorable to scientific productivity. The methods used . . . were substantially the same as before and need not be redescribed in detail."[10]
( 87)
On the basis of the two arrangements seven years apart, and with the aid of additional information as to the distribution of the men, etc., an attempt was made to throw light on a great many questions of importance to science in general, and of particular interest to the science of education. The changes in position during the seven year period; the correlation between age and advancement; the distribution of scientific men in various institutions and localities; the relative strength of the various departments and scientific faculties of different institutions; the productivity of scientific men in different sections of the country and in different educational centers; the correlation between eminence and the age of receiving the various degrees; the comparison of the conditions of scientific success in the different sciences; these and other equally interesting questions received such illumination that it is quite impossible to summarize the results here. The influence of these studies of Professor Cattell on the general interests of education in America can not be easily estimated, and all of the original reports merit careful and detailed study by both students and administrators. Of the various contributions which the studies have made, the purely methodological portions are not the least important.
The application of the order of merit method was of course but incidental to the larger purposes of these studies. The general motive and point of view are well expressed in the following characteristic words.
"It is surely time for scientific men to apply scientific methods to determine the circumstances that promote or hinder the advancement of science. We should begin where and when we can; even though the results of the first efforts may appear somewhat trivial, we may proceed in the confident belief that in the end the advancement of science will become an applied science."[11]
"It is not altogether without interest to find that it is possible to reduce to order facts which might be supposed to be outside the range of the natural and exact sciences. The present articles are, however, only a beginning of a study of scientific men as a group and of the conditions on which scientific performance depends. We have in a large measure explored the material world and subdued it to our uses; it is now our business to secure an equal increase in our knowledge of human nature and to apply it for our welfare. If he is a benefactor to mankind who makes two blades of grass grow where one grew before, his services would be immeasurably
( 88) greater who could enable two men of science to flourish where there had been but one"[12]
The method of measurement by relative position in a series has been adopted by a number of Professor Cattell's students and colleagues, as well as by several other workers, in the investigation of facts to which objective scales can not be easily applied. These various studies may be roughly grouped under the headings individual differences, education, esthetics, judgment, commerce, and methodology.[13]
In the field of individual differences, Sumner has studied age, sex, and class differences in the ability to arrange one's beliefs along a scale of certainty. Norsworthy has studied personal estimates of individual differences in ability and character, and the conditions and variability of these estimates. Kuper has used the method in investigating the interests of children and the dependence of these interests on such factors as age, sex and nationality. Downey has employed the method in her study of family resemblance in handwriting and the degree to which such resemblance is due to heredity. On the basis of the same method Fernald has devised an ethical discrimination test which he has used in the examination of defective delinquents.
Applications of the method in education are represented by Thorndike's scales for the measurement of the legibility of penmanship and achievement in drawing and by the scale devised by Hillegas for the measurement of excellence in English composition. Yerkes has used the method as a suggestive method of instruction and of class exercise in psychology.
In esthetics Wells has used the method in his statistical study of literary merit, with special reference to the possession of such qualities as charm, clearness, originality, euphony, wholesomeness, etc. The same investigator has used picture postal cards as stimuli in a study of the variability of judgments of preference and beauty. The present writer has used the method in a study of the types and tendencies of comic situations and Downey has employed it in her study of the imaginal reaction to poetry and the variability of affective judgments.
In commerce the method was given an interesting and useful application by E. K. Strong, Jr., and the present writer, when it
( 89) was found that the laboratory study of advertisements, sales letters, etc., affords significant measurements of "pulling power" which correlate closely with the actual value of the various appeals as determined by the number of replies or inquiries, the cost per inquiry, the number of sales, etc. This and related work led to the establishment, by a group of practical business men, of a research fellowship in applied psychology in Columbia University, which has now been continued for three successive years.
In the field of judgment, Wells, Downey and the writer have made numerous studies by the method of relative position, in which, aside from the measurements and their usefulness in general and applied psychology, a number of problems concerning the judgments themselves have arisen. Among these problems may be mentioned as typical examples, the variability of judgment in different parts of the series, the certainty of individual preferences and aversions, group differences in likes and dislikes, individual differences in consistency of judgment and in judicial capacity, the correlation between these two characteristics, consistency and judicial capacity in different situations, the relation of variability to the length of the series, the influence of the form of expression on the outcome of judgments, and the more general problem of securing quantitative criteria of the degree of subjectivity possessed by various types of judgments.
From the purely methodological point of view Barrett has made an empirical comparison of the relative advantages of the order of merit method and the method of paired comparisons, demonstrating the numerous technical superiorities of the former method. The present writer has proposed a modification of the strict order method, designated the group method. This method, in addition to being simpler and more easily carried out, correlates closely with the results from the same observers by the strict order method and gives opportunity to observe changes in the evaluation of the series as a whole. Thorndike has discussed certain of the statistical features of the method of measurement by relative position, and the possibility of transmuting such a qualitative series into a quantitative distribution on the basis of the curve of the probability integral.
Several studies are now in progress in which the method of measurement by relative position is being used in the study of such topics as belief, judgment, eminence, affective qualities, special abilities, individual differences, and a number of specific problems in applied psychology.
The method has proven to be a real contribution to the tech-
( 90) -nique of experimental, statistical and applied psychology. It is especially useful in that type of psychological investigation which is more interested in outcome than in content; in the way in which mind works, rather than in what is in the mind at the moment of its operation.
H. L. HOLLINGWORTH
COLUMBIA UNIVERSITY
BIBLIOGRAPHY ON MEASUREMENT BY RELATIVE POSITION
(THE ORDER OF MERIT METHOD)
Barrett, Mabel, The Order of Merit Method and the Method of Paired Comparisons, Jour. Phil., Psych., and Sci. Meth., July 3, 1913, 382-4.
Cattell, James McKeen, The Time of Perception as a Measure of Differences in Intensity, Philos. Studien, 19, 63-68, 1902.
———, A Statistical Study of Eminent Men, Pop. Sci. Mo., 53, 357, 1903.
———, Statistics of American Psychologists, Amer. Jour. Psychol., 14, 310-328, 1903.
———, A Statistical Study of American Men of Science, Science, N. S., XXIV, No. 621, 658-665, Nov. 23, 1906; No. 622, 699-707, Nov. 30, 1906; No. 623, 732-742, Dec. 7, 1906.
———, A Further Statistical Study of American Men of Science, Science, N. S., XXXII, No. 827, 633-648, Nov. 4, 1910; No. 828, 672-688, Nov. 11, 1910.
———, American Men of Science. (In the appendix of this volume the two foregoing studies are reprinted, in the first edition (1906) the first study, and in the second edition (1910) both the first and the second studies.)
Downey, June E., A Preliminary Study of Family Resemblance in Handwriting, Bull. No. 1, Dept. Psychol., Univ. of Wyoming, 1910.
———, The Imaginal Reaction to Poetry,—Bull. No. 2, Dept. of Psychol., Univ. of Wyoming, 1912.
Fernald, G. E., The Defective Delinquent Class, Differentiating Tests, Amer. Jour. of Insan., 69, 125-142, 1912.
Hillegas, Milo B., A Scale for the Measurement of Quality in English Composition by Young People.—Teachers College Record, XIII, 4, Sept. 1912.
Hollingworth, H. L., Judgments of the Comic, Psych. Rev., 18, 132, 1911.
———, Judgments of Persuasiveness, Psych. Rev., 18, 234, 1911.
———, The Influence of Form and Category on the Outcome of Judgment (With Margaret Hart Strong), Jour. Phil., 9, 513, 1912.
———, Advertising and Selling, New York, Appletons, 1913.
———, Experimental Studies in Judgment, ARCHIVES OF PSYCHOLOGY, No. 29, 1913.
Kuper, Gertrude M., Group Differences in the Interests of Children, Jour. Phil.. July 4, 1912, 376-379.
Norsworthy, Naomi, Validity of Judgments of Character, Essays in Honor of William James, 1908.
Strong, E. K., Jr., The Relative Merits of Advertisements, ARCHIVES OF PSYCHOLOGY, No. 17, 1911.
———, The Relative Merits of Advertisements, Advertising and Selling, 1911.
———, Application of the "Order of Merit Method" to Advertising, Jour. Philos Oct. 26, 1911, 600-606.
———, Psychological Methods as Applied to Advertising, Jour. of Educational Psychol., Sept. 1913, 393-404.
Sumner, F. B., A Statistical Study of Belief, Psych. Rev., 5, 616-631, 1898.
Thorndike, E. L., Handwriting,
———, Mental and Social Measurements, 2d ed., 1913.
———, The Measurement of Achievement in Drawing. Teachers College Record, XIV, 5, Nov. 1913.
Wells, F. L., A Statistical Study of Literary Merit, ARCH. Psychol., No. 7, 1907.
———, On the Variability of Individual Judgments, Essays in Honor of William James, 511, 1908.
Yerkes, R. M., Introduction to Psychology, Ch. XIV., 1911.
