Measuring the Effect of Motion Pictures on the Intellectual Content of Children
George D. Stoddard
This brief report is devoted to a description of the purposes and methods utilized by P. W. Holaday,[1] under the direction of the writer, in one part of the broad inquiry into motion-picture influences. The restricted purpose was to ascertain the effect of viewing theatrical films under ordinary conditions on children's information and to analyze their comprehension and retention of material.
The original plan called for two approaches: Type X studies devoted to the measurement and analysis of what children learn from the movies, and Type Y studies which attempted to show the change produced by this new (movie-induced) knowledge on the general mass of information possessed by the children.
The simplest way to clarify the difference between these two approaches is to insert here a condensed version of the original scheme of attack:
Type X, Study A: The measurement of factual information gained by children from a particular film.
Aims
To discover:
- The extent of the children's knowledge of the film
- The curve of forgetting with respect to this knowledge
- Age-level differences
- Mental-level differences
- Comparison with adult knowledge of the same film
- The type of knowledge most (least) often gained and retained; such as character details, episodes in love scenes, what happened to the "villain," details of setting, customs, names of actors, etc.
- The accuracy of the knowledge gained and retained
(205)
Method
- Select stock types of pictures to be used as stimuli; e.g., (a) comedy-drama (love motif predominant) ; (b) "spook"; (c) wild west; (d) South-Sea romance; (e) slapstick comedy; (f) screen version of a classic; (g) war.
- Four or more stenographers are to take down as elaborately as possible all the factual elements in the picture and these notes are to be checked for accuracy and completeness by the research director and his assistants. Probably two viewings of each picture will be desirable.
- From these records an objective type of information test is to be built. This is given to all the subjects the morning following the movie (without previous announcement of the test). Test items cover in detail such points as the setting, actions, results, scenes, etc., but only from the standpoint of simple content.
- The test is repeated without warning one week and one month later.
- All tests are scored and analyzed in accordance with the statements under "aims" above. Types of knowledge are not segregated in the test, but they are in its analysis.
- The same subjects should be tested on several types of pictures, but with sufficient intervals between "significant" pictures to prevent any preparatory measures on their part.
Materials
- Objective tests (to be devised) : (a) Each test must fall within the reading range of the subjects.
- Intelligence test and school records of the children.
Subjects
- A group of at least fifty children at each age level in each age research unit. (Given the same picture and the same test, units from various sections could be combined.)
- Suggested age levels: (a) age 8 (7 1/2 to 8 1/2) ; (b) age 12(11 1/2 to 12 1/2);©) age 16 ( 15 1/2 to 16 1/2).
- A group of fifty adults: teachers, graduate students, parents.
Type X, Study B. The measurement of the comprehension and interpretation of a film on the part of children.
Aims
To discover:
- The extent of the children's comprehension of the film
- Temporal changes in this comprehension
- Children's interpretation of various actions and ideas; (a) extent and kind
(206)
- Age-level differences
- Mental-level differences
- Types of interpretation: (a) similar to adults; (b) fantastic; (c) close to intent of the film (if discoverable).
Method
Same as in Study A, but test must be of a different type, and great care must be taken to secure essential agreement among adults as to what the correct (or at least common) comprehensions and interpretations are.
Type Y, Studies A and B: The measurement of changes produced in children's knowledge (including comprehension and interpretation) by films.
Aims
To discover:
- The extent of revision of factual information in the light of the film
- Duration of this revision
- Age-level differences
- Mental-level differences
- Types of knowledge changes ; e. g., in new concepts of foreigners, Hollywood, countries, customs, etc.
Method
- A detailed analysis in advance of a film to record all possibilities for new knowledge to be gained from it.
- A test devised related to the chief points and given to children in advance of attendance at the film. The test is not on the film content, but on the information which is likely to be affected by this known film content.
- Attendance at the film.
- Retest to discover changes produced: (a) next morning; (b) one week and one month later.
- By "change" is meant: (a) new knowledge; (b) in-creased accuracy (or inaccuracy) in old knowledge; (c) lapse of old knowledge.
- Illustrative types of knowledge: (a) vocabulary; (b) historical events and persons; (c) film industry and personnel; (d) ways people live; (e) geographical; (f) knowledge of validity of screen events.
In the actual prosecution of the research, certain modifications proved to be necessary. For example, it proved infeasible to test the children the day after the show and again one month later with a view to measuring the retention from the original showing of the films. The testing
( 207) the day after tended to impress the children unduly, with the result that further tests were rendered somewhat in-valid. Hence groups were matched on school grade, intelligence, and reading ability. This necessitated rather large samplings of children. In the total Iowa sampling nine hundred observers assisted in one or more of sixteen tests. Extension of the work to Ohio towns in 1930-1931 added substantially to the population, which may be said to rep-resent adequately the large and small towns of these two States. It may be inferred that the sampling is adequate for unselected American-born, white, city school children.
It was found also that, for technical reasons, the true-false type of test does not lend itself well to a study of retention. Perhaps the most unexpected revision of all lay in the necessity for extending the testing up to seven months later in order to carry out the curve of forgetting to points of significant drops. For certain pictures it was evident that no fixed duration of time could be counted upon to erase all mental effects. Finally, the talkies suddenly displaced the silent movies after considerable work had been done and complicated the problem of picture analysis. However, the changes necessitated in this connection may be ascribed to "an act of God."
The machinery of transforming the paper plan of re-search to a working system within the customary frame-work of school child and motion-picture exhibitors is not to be viewed lightly. The researcher was compelled to gain access to the films in advance of public showing in a community; to appease the theater owners; to finance children's expenditures; to secure parent and teacher coöperation; to bar automatically intergroup discussion of pictures; to gear up personnel in such a way as to extract, in one showing of a film, all the essential points of setting, plot, characters, costumes, incidents, and conversations.
Pictures were viewed in cities earlier in the booking routine or were "previewed" by special arrangement. Theater owners were cheered by the sale of strings of
( 208) tickets (which were later dispensed by the researcher). In many cases admission for the children was secured for five cents. A good rapport was established with parents and teachers in the name of scientific inquiry, although few parents had any objection to movie-going. After a night showing children were examined in school early the following morning, before interchange of ideas would be likely to take place. In the matter of film analysis, the plan gradually evolved from the taking down of every-thing by expert stenographers to an allocation of subject-matter tasks to experienced observers. These observers often saw the picture two or three times before the notes were assembled.
Questions formulated on the basis of these data were later reviewed by the director of the project. The usual methods of determining reliability were employed and all tests were revised in the light of preliminary findings. Multiple response and completion tests were finally adopted, of which the items below are typical:
Multiple response, specific item (Type X)
The actress who played the part of Betty was (1) Dolores Costello, (2) Ruth Chatterton, (3) Evelyn Brent, (4) Greta Garbo, (5) Myrna Loy.
Completion, specific item (Type X)
The money to start the tearoom was furnished by ———
Multiple response, general item (Type Y)
In England, army officers are usually (1) gentlemen who joined because they needed money; (2) soldiers who were promoted for bravery; (3) soldiers promoted for having been in the army a long time; (4) gentlemen who joined because they liked the life; (5) gentlemen who were forced by the government to join the army.
Multiple-response items were answered by underlining one of the statements; completion items, by writing in the missing word or phrase.
The median reliabilities of the Iowa tests as finally ad-ministered varied from .67 to .92. They may be considered satisfactory for short tests designed for group com-
( 209) -parisons. Attempts to secure valid and reliable essays or reports from the school children proved fruitless. It was shown that laconic "compositions" often concealed an immense amount of actual information which could be elicited by objective testing methods.
In contrast to reliability, there are no "usual" techniques for establishing the validity of a test; that is to say, the extent to which a test really measures what it purports to measure. There were not even precedents in motion-picture material, but the following devices were employed to make test performance mirror the under-lying state of affairs:
- Films were checked in such a way as to ensure a spread of questions over the entire picture.
- At least three people observed each picture and contributed to the notes.
- Observers read novels from which the movies had been adapted, together with appropriate works in history and geography. In special fields, university experts were consulted. (These precautions apply to the formulation of "general" or Type Y questions; i. e., content which may conceivably be affected by what is seen in the movies.)
- The place of the correct answer in multiple response questions was fixed to give a random distribution. "Trick" items were avoided.
- Items were placed in ten categories on the basis of three judges, as follows: emotional (except fighting, mystery, romance), humorous, mysterious, revue, crime, fighting, romance, drinking, general conversation, general action. Test time in each category was closely related to the corresponding film time.
A consideration of the findings is not in order here. Suffice it to say that the specific knowledge of children and adults is greatly increased by motion pictures and that their general information is significantly affected by what is seen in the pictures. Retention is high over the period of seven months covered in this project.
NOTE: An article by Professor Frank N. Freeman on the measurement of the effectiveness of a film upon the care of the teeth will appear in the January issue of THE JOURNAL.
