Story Comprehension Experiment

Neil McIntyre
School of Informatics
University of Edinburgh

Thanks for doing our experiment!

Please read through the instructions below before starting. If you have any queries about this experiment then feel free to send email to the address given at the bottom of the page.

You must be a native English speaker to participate in this study. This experiment requires a Java compatible web browser and Java has to be enabled. You will also have to fullscreen your web browser to properly view the experiment.

Depending on the hardware and browser you use, and on your net connection, the execution of the experiment may be slow at times.

Instructions

In this experiment you will be asked to read a set of short computer generated stories. Each story will be 5 sentences long and will contain only a couple of characters. After reading each story you will assess its quality along three dimensions: fluency, coherence and interest. For each dimension you will provide a rating on a scale from 1 to 5.

Fluency refers to the individual sentences of the story, whether they are grammatical and in well-formed English or just gibberish.

• If the sentences are grammatical, the you should rate the story high in terms of fluency.
• If the sentences are something like word salad, then you should give the story a low number.

• If the text is almost impossible to understand, then you should give it a low number.
• If the text is readily comprehensible, well-organised and doesn't require any effort on the reader's part, then you should give it a high number.

• Use high numbers if you find the story original.
• On the other hand it the story is rather tedious, you should use low numbers.

Suppose you were given the following story:

Then, you may rate it high in terms of fluency (e.g., 4 or 5) as the sentences are well-formed and overall grammatical. This story would be also given a high coherence number (e.g., 4 or 5) as it progresses in a logical order, focuses on the main characters and generally makes sense. You may give this story a medium score as far as interest is concerned (e.g., 3), as it is familiar and describes a rather predicable series of events. However, if you are not familiar with the story at all, you may give it a high number.

Now, take the following example:

This story is much harder to read than the previous example. It contains unusual sentence structures and the individual words do not make sense together. So you would rate this story low in terms of fluency (e.g., 1 or 2). Taken in its entirety, this story also makes little sense. It is unclear why the characters are behaving as they are and there is no strong progression of actions. The story seems rather unfocused, the dragon appears in the first sentence and then only again at the end. Overall the story is not coherent and would receive a low score (e.g., 1 or 2). This story is not very interesting either as it is impossible to picture exactly what is happening. The disjoint action sequences are too confusing to capture the reader's attention. So interest here would receive a low rating (e.g., 1 or 2).

A story can receive high ratings on some dimension and low ratings on others; it is not necessary that your ratings for fluency, coherence, and interest are all either high or low.

Procedure

When you start the experiment below you will be asked to enter your personal details. Next, you will be presented with 12 short stories to evaluate in the manner described above. Once you have completed your rating, click the button at the bottom of that page to advance to the next story.

Things to remember:

• Full-screen your web browser before starting the experiment.
• Keep this page open so that you can refer back to it if you are at all unsure of how to rate a story.
• Higher numbers represent a positive opinion of the story and lower numbers a negative one.
• Do not spend too long analysing the stories; you should be able to rate them once you have read them for the first time.
• There is no right or wrong answer, so use your own judgement when rating each story.
• 
  

Personal Details

• your name and email address;
• your age and sex;
• whether you are right or left handed (based on the hand you prefer to use for writing);
• the academic subject you study or have studied (or your current occupation in case you haven't attended university);
• under `Language Region', please specify the place (city, region/state/province, country) where you have learnt your first language.
• enter the code provided at the end of the experiment into the Mechanical Turk HIT.

The experiment will take approximately 15 minutes. Once the experiment is over you will be redirected to a new web page which will inform you that your data has been transferred to the experimenter. So, please do not surf away from the experiment page unless you are told the experiment is finished. This page will also provide you with the validation code required by Mechanical Turk. To begin, click on the link below.