About us

Presentation Outline

What is IBEX and PCIbex?

IBEX

• (I)nternet-(B)ased (EX)periments developed by Alex Drummond.

• Refers to both the framework for scripting experiments and the platform for hosting experiments.

• IBEX the platform "IBEXfarm" has been discontinued in late 2021, but the IBEX framework survives and is supported by extensions and spinoffs.

PCIbex

• PennController for IBEX developed by Jeremy Zehr and Florian Schwarz at the University of Pennsylvania.

• A platform for hosting experiments that supports and extends IBEX

• Also introduces its own, alternative framework for scripting experiments.

Navigating the PCIbex platform

• Go to the PCIbex farm https://farm.pcibex.net

• Sign up with an email (or Log in)

• Once logged in, click Empty project under Start a new project

• Change the name of the project at the top to something unique (e.g., "SICOL2022")

• Your new project will appear on the main page

PCIbex Dashboard Interface

Experiment file structure

Experiment file structure

The script

When scripting with IBEX, we often only need one file: main.js

At creation, the default file looks like this:

This is a snippet of code from the PCIbex scripting framework, which is not the focus of the workshop today (we will briefly revisit PCIbex at the end).

Writing your own script

To demonstrate the IBEX scripting framework, we use our own template file.

The following IBEX code just works thanks to PCIbex's backward compatibility.

This script creates an acceptability judgment experiment.

A simple layout

A simple layout

Walkthrough of the components

Our first experiment

Study: We are interested in how people recover from garden-path sentences.

"While Anna dressed the kitten paid attention."

... *[VP dressed the kitten], ...

... ✓[VP dressed], the kitten ...

Hypothesis: Verbs that are frequently transitive make recovery (reanalysis) more difficult, compared to verbs that are frequently intransitive.

(transitive-biased) - "While Anna trained the kitten paid attention."

(intransitive-biased) - "While Anna dressed the kitten paid attention."

Prediction: Lower acceptability ratings in the transitive-biased condition (gp.trans) than in the intransitive-biased condition (gp.intrans).

Trial Syntax

For each trial in our acceptability judgment experiment, we write this code:

[["Condition name", Item Set #], "Task Type", {s: "Sentence"}]

This is a list (array) of three elements (clearer with spacing):

We have one big list (bracket) which contains three elements:

1. Another list, consisting of the name of the condition and item set number

2. A string specifying the type of task (also called Controllers or Elements)

3. A curly bracket (braces), which has "s" and the sentence separated by a colon

Translating the design to code

Example 1:

In the gp.trans condition, the first stimuli in our acceptability judgment experiment is the sentence "While Anna trained the kitten paid attention".

Translating the design to code

Example 1:

In the gp.trans condition, the first stimuli in our acceptability judgment experiment is the sentence "While Anna trained the kitten paid attention".

Translating the design to code

Example 1:

In the gp.trans condition, the first stimuli in our acceptability judgment experiment is the sentence "While Anna trained the kitten paid attention".

Translating the design to code

Example 1:

In the gp.trans condition, the first stimuli in our acceptability judgment experiment is the sentence "While Anna trained the kitten paid attention".

Translating the design to code

Example 1:

In the gp.trans condition, the first stimuli in our acceptability judgment experiment is the sentence "While Anna trained the kitten paid attention".

Translating the design to code

Example 1:

In the gp.trans condition, the first stimuli in our acceptability judgment experiment is the sentence "While Anna trained the kitten paid attention".

This can be put into a single line:

[["gp.trans",1], "AcceptabilityJudgment", {s: "While Anna trained the kitten paid attention."}]

Translating the design to code

Example 2:

For the gp.intrans condition, the first stimuli in our acceptability judgment experiment is the sentence "While Anna dressed the kitten paid attention".

This can be put into a single line:

[["gp.intrans",1], "AcceptabilityJudgment", {s: "While Anna dressed the kitten paid attention."}]

Another example set

Item set #2:

(transitive) - "Since Dave improved the department was satisfied."

(intransitive) - "Since Dave worried the counselor devised a plan."

[["gp.trans",2],
  "AcceptabilityJudgment",
  {s: "Since Dave improved the department was satisfied."}]
[["gp.intrans",2],
  "AcceptabilityJudgment",
  {s: "Since Dave worried the counselor devised a plan."}]

Putting together the stimuli

Wrap in var items = [ ... ] and separate each item by a comma:

Good scripting habits:

• Grouping item sets together and separating sets with new line

• Adding comments (starts with two or more slashes //)

• Saving often! (scripting outside of PCIbex using an editor is also an option)

Practice and Fillers

Practice Trials:

• Presented at the beginning, accompanied by instructions and feedback

Filler Trials:

• Mixed in with critical trials, to distract or exclude participants

We want both practice and filler trials to be invariant across conditions

To do that, we don't specify Item Set # for give practice and filler trials, as that gets used for counterbalancing (more later):

["practice-1",
 "AcceptabilityJudgment",
 {s: "The car drove like a dream."}]

["filler-good-01",
 "AcceptabilityJudgment",
 {s: "When Harry fell, the audience was shocked."}]

["filler-bad-01",
 "AcceptabilityJudgment",
 {s: "When Tyler sneezed the driver, he passed a tissue."}]

["filler-catch-01",
 "AcceptabilityJudgment",
 {s: "Please select 4 for this sentence."}]

Putting together the Body

Important: The ordering of the trials here is just for human readability. We haven't yet told the program what order to present them in!

Defining the Sequence

 "practice-1",
 "gp.trans",
 "gp.intrans",
 "gp.trans",
 "gp.intrans",
 "filler-good-01",
 "filler-bad-01",
 "filler-catch-01"

Defining the Sequence

seq(
 "practice-1",
 "gp.trans",
 "gp.intrans",
 "gp.trans",
 "gp.intrans",
 "filler-good-01",
 "filler-bad-01",
 "filler-catch-01"
)

Defining the Sequence

var shuffleSequence = seq(
 "practice-1",
 "gp.trans",
 "gp.intrans",
 "gp.trans",
 "gp.intrans",
 "filler-good-01",
 "filler-bad-01",
 "filler-catch-01"
)

Defining the Sequence

var shuffleSequence = seq(
"practice-1",
 "gp.trans", 
 "gp.intrans",
 "gp.trans", 
 "gp.intrans",
"filler-good-01",
"filler-bad-01",
"filler-catch-01"
)

Defining the Sequence

var shuffleSequence = seq(
 "practice-1",
"gp.trans",
"gp.intrans",
"gp.trans",
"gp.intrans",
"filler-good-01",
"filler-bad-01",
"filler-catch-01"
)

Defining the Sequence

var shuffleSequence = seq(
 "practice-1",
"gp.trans",
"gp.intrans",
"gp.trans",
"gp.intrans",
 "filler-good-01",
 "filler-bad-01",
 "filler-catch-01"
)

1. Sequence: multiple selection

To save us from writing repetitive code, we use startsWith()

var shuffleSequence = seq(
"practice-1",
"gp.trans",
"gp.intrans",
"gp.trans",
"gp.intrans",
"filler-good-01",
"filler-bad-01",
"filler-catch-01"
)

var shuffleSequence = seq(
startsWith("practice"),
startsWith("gp")
startsWith("filler")
)

The function startsWith() matches all names that starts with the given string.

2. Sequence: randomization

To mix critical and filler trials in random order, we use rshuffle()

var shuffleSequence = seq(
 startsWith("practice"),
startsWith("gp")
startsWith("filler")
)

var shuffleSequence = seq(
 startsWith("practice"),
rshuffle(
   startsWith("gp"),
   startsWith("filler")
  )
)

By wrapping both the critical trials (gp...) and the filler trials (filler...) in rshuffle(), they are mixed together and presented in random order.

We can also write this out more compactly:

var shuffleSequence = seq(
 startsWith("practice"),
rshuffle(startsWith("gp"), startsWith("filler"))
)

3. Sequence: counterbalancing

The set number in our critical trials are automatically used for counterbalancing:

To alternate items between participants, we use a special counter "trial":

["setcounter", "__SetCounter__", { }]

3. Sequence: counterbalancing

The counter must be defined in items and added to shuffleSequence:

var items = [
  ...
  ["setcounter", "__SetCounter__", { }], // The counter is defined
  ...
]
var shuffleSequence = seq(
  "setcounter", // The counter is incremented at the start
  ...
)

Sometimes you want to increment counter in the middle of the experiment:

var shuffleSequence = seq(
  "intro"
  "consent",
  "setcounter", 
  ...
)

Body and Sequence together

Just need one more step: Settings

Settings

Consist of miscellaneous options that we can put at the top of the script.

For example, using the defaults variable, we can specify parameters for the design of the tasks that we use in the experiment, like Acceptability Judgment.

More details in the AcceptabilityJudgment section of the documentation.

The defaults variable

Some options for Acceptability Judgment:

var defaults = [
    "AcceptabilityJudgment", {
       as: ["1", "2", "3", "4", "5", "6", "7"],           
       presentAsScale: true,                             
       instructions: "Use number keys or click boxes to answer.",    
       leftComment: "(Bad)",
       rightComment: "(Good)"
    }]

Another defaults example

Suppose you'd like to show participants multiple sentences but have them only rate the acceptability of one of the sentences.

var defaults = [
    "AcceptabilityJudgment", {
       ...                     
       instructions: "Rate how natural Speaker B's response sounds."
    }]

[["ConditionA", 1],
 "AcceptabilityJudgment",
{s: ["div",
       ["p", "Speaker A: Who left this sandwich on the table?"]
       ["p", "Speaker B: Fred did."]
     ]}]

Settings (Miscellaneous)

You can also change other options, such as showing a message at the end:

var completionMessage

// A message to show to participants at completion
var completionMessage = "Thank you for your participation!"

And whether to show a progress bar:

var showProgressBar

// Show a progress bar at the top? (true/false)
var showProgressBar = false

You can learn more about these various elements in the Miscellaneous options section of the IBEX documentation.

A minimal experiment

We now have a minimally working experiment!

Interim Summary #1

What we've covered:

• We have a templatic syntax for creating stimuli

• We store all the materials for our experiment inside items

• We specify the order of presentation inside shuffleSequence

• We can tweak various aspects of the experiment, like defaults for the task design.

A few more things we want to know:

• How can we show messages and text to participants?

  • Introduction page, consent form, directions, etc.

• How can we extend this workflow for other experimental designs?

  • Self-paced reading, comprehension tasks, etc.

The "Message" controller

The "Message" controller shows text on a new page.

It is a list of 3 elements, similar to the "AcceptabilityJudgement" items:

["Condition name", "Message", {html: text}]

• Condition name is used to reference the trial in sequencing

• "Message" is a type of task that just shows text on a screen

• Inside of the curly braces {} we can add text in the html parameter:

["intro", "Message", {html: ["p", "Welcome to the experiment!"]}]

Notes on html:

• The code ["p", "<your text here>"] prints a paragraph of text.

• The "p" is an HTML tag and there are many others (most common: "div", "strong", "em"). They don't usually get too complicated.

Message examples

["intro", "Message", {html: ["p", "Welcome to the experiment!"]}]

["intro", "Message", {html:
    ["div",
      ["p", "Welcome to the experiment!"],
      ["p", "Here's another paragraph."],
      ["p", "These paragraphs are all wrapped inside \"div\"."]
    ]
}]

["consent", "Message", {
  html: ["p", "Do you consent?"],
  consentRequired: true,
  consentMessage: "I consent."
}]

["move_on", "Message", {
  html: ["div", 
          ["p", "The option for \"transfer\" is \"keypress\""],
          ["em", "Press any key to continue."]
        ],
  transfer: "keypress"
}]

["sep", "Message", {
  html: ["em", "Press any key to continue."],
  transfer: "keypress"
}]

var shuffleSequence = seq(
  "practice",
  sepWith("sep", rshuffle(startsWith("gp"), startsWith("filler")))
)

Putting everything together

We now have a complete experiment with several Message controllers added.

Getting the experiment up

So we have a new script, but how do we host the experiment?

1. Go to the PCIbex farm website and log in.

2. Click into (or create) your experiment and edit the main.js file.

3. Click Share in the menu bar to the right and copy the Demonstration link (for testing and development) or Data-collection link (available after publishing)

Demo: https://farm.pcibex.net/r/YBYAwi

Published: https://farm.pcibex.net/p/HCRZqI

A different experiment

Suppose that after a pilot experiment, we find acceptability judgments to be inappropriate for answering our research question.

We want a finer-grained measure of recovery difficulty, so we'd like to change the experiment to self-paced reading and look at differences in reading time.

Given our existing template, we take the following steps:

1. Go to the documentation and find a Controller for self-paced reading.

2. Specify the design of that controller in the defaults variable.

3. Change our trials in items from "acceptabilityJudgment" to that Controller.

4. Make changes to the text of the "Messages" items (e.g., directions).

"DashedSentence" Controller

The "DashedSentence" Controller creates self-paced reading trials.

We can simply replace "acceptabilityJudgment" with "DashedSentence" in items:

var items = [
  ...
  [["gp.trans",1], "DashedSentence", {s: "While Anna trained the kitten paid attention."}],
  [["gp.intrans",1], "DashedSentence", {s: "While Anna dressed the kitten paid attention."}],
  [["gp.trans",2], "DashedSentence", {s: "Since Dave improved the department was satisfied."}],
  [["gp.intrans",2], "DashedSentence", {s: "Since Dave worried the counselor devised a plan."}],
  ...
]

And specify appropriate settings for "DashedSentence" in defaults:

var defaults = [
    "DashedSentence", {
       mode: "self-paced reading",
       display: "dashed"
    }]

Our second experiment

Finally, after re-writing some of the "Message" items, we have a new experiment!

Question and Form Controllers

[["YesNo_example",1],
"Question",
{q: "Was the kitten trained?", as: ["Yes", "No"]}]

[["FC1_example",1], "Question", {
  instructions: "Choose the more natural sentence.",
  as: ["Who did you see that ate bread?",
       "What did you see the girl who ate?"]}]

[["FC2_example",1], "Question", {
  q: "While Anna dressed the baby ______",
  instructions: "Choose the more natural continuation.",
  as: ["started to cry.", "he started to cry."]}]

[["FR_example",1],
 "Form", {html: ["div",
   ["em", "Fill out a continuation for the sentence fragment:"],
   ["p", "While Anna dressed the baby ______"],
  ["textarea"]
]}]

Where to find the data

The data collected in the experiment can be found on the experiment page, in the Results tab to the right side of the dashboard.

It gives you a preview of the results and options to download or delete the data that's been collected so far.

The results.csv file

If you download the data, you get back a csv file called results.csv, meaning that each line contains a set of values that are separated by a comma.

Lines that start with a pound symbol "#" are comments that include metadata.

The ones that start with numbers tell us what variable the columns correspond to.

The variables

All IBEX experiments return these 7 variables:

1. Time ("Results reception time")

2. Participant ID ("MD5 hash of participant's IP address")

3. Controller ("Controller name")

4. Item number ("Order number of item")

5. Inner element number ("Inner element number")

6. Condition ("Label")

7. Item set ("Latin Square Group")

The variables

In our case, we only care about 4 of these:

1. Time

2. Participant ID: A unique ID for each participant

3. Controller: The type of task/trial (e.g., AcceptabilityJudgment)

4. Item number

5. Inner element number

6. Condition: The condition label for the trial (practice-1, gp-trans, ...).

7. Item set: The item set number (1,2,3,...).

The last two variables uniquely identify the stimuli defined in items:

[["Condition", Item set], "Controller", {s: "Sentence"}]

[["gp.trans", 1], "DashedSentence", {s: "..."}]

The variables

You also get other variables depending on the Controller

For self-paced reading with "DashedSentence", we get 5 more variables:

1. Word number - The index of the word in the sentence.

2. Word - The text of that word.

3. Reading time - Reading time for that word.

4. Newline? - 0 or 1 indicating whether there was a line break.

5. Sentence - The text of the sentence.

These are also outlined in the documentation for "DashedSentence", so you know what variables to expect beforehand.

For an actual analysis of the results, we need the results to be imported somewhere as a data frame, preferrably in a rectangular table format where rows are observations and columns are variables.

Importing - Excel

Steps for importing the data into Excel:

1. Copy the text of the results file from IBEX

2. Paste into the first column of an Excel spreadsheet

3. Highlight that column and click Data tab -> Filter

4. Click on the dropdown arrow that appears at the top cell

5. Click Text Filters -> Begins With... and type in "#"

6. Go to Home tab -> Find & Select, check "Visible cells only", and click OK

7. Right click on any part of the sheet and select Delete Row

8. Click the first column again and go to Data tab -> Text to Columns

9. Check "Delimited", "Comma", then "General" for each prompt

10. Add an empty row at the top and manually type in the column names from the comments of the original results file

Importing - Excel

Complex results

Some Controllers return multiple lines of results, with differing number of columns

For example, each AcceptabilityJudgment trial is a combination of Message and Question controllers under the hood, so it logs two lines of data:

In more complex cases like these, using a script (R/Python) is recommended

Importing - R

Steps for importing into R:

1. Download the results.csv file from PCIbex

2. Open it with the read_pcibex() function from the read_pcibex.R script

source("read_pcibex.R")
results <- read_pcibex("SPR_results.txt")

  # A tibble: 6 × 5
    Controller_name Label      Word_number Word   Reading_time
    <chr>           <chr>            <dbl> <chr>         <dbl>
  1 DashedSentence  practice-1           1 The             132
  2 DashedSentence  practice-1           2 car             115
  3 DashedSentence  practice-1           3 drove           110
  4 DashedSentence  practice-1           4 like            112
  5 DashedSentence  practice-1           5 a               118
  6 DashedSentence  practice-1           6 dream.          117

source("read_pcibex.R")
results <- read_pcibex("Acceptability_results.txt")

  # A tibble: 6 × 4
    Label      Latin_Square_Group Sentence_or_sentence_MD5   Answer
    <chr>      <chr>              <chr>                       <dbl>
  1 practice-1 NULL               The car drove like a drea…     NA
  2 practice-1 NULL               <NA>                            5
  3 gp.intrans 2                  Since Dave worried the co…     NA
  4 gp.intrans 2                  <NA>                            6
  5 gp.trans   1                  While Anna trained the ki…     NA
  6 gp.trans   1                  <NA>                            5

Task

The experiment script incomplete.js has some missing pieces.

Task

Directions:

1. Download incomplete.js

2. Create a new experiment on your PCIbex account called "WorkshopExercise" and replace the contents of main.js with incomplete.js.

3. Follow the directions in the script to fill in the blanks.

4. Complete as many of tasks as possible (8 total).

If you finish, save the edits, refresh, and preview the experiment in the bottom panel or click Open in new tab. Check to see that your experiment looks similar to the complete version - https://farm.pcibex.net/p/ZvbOXD.

Make sure to reference the IBEX documentation!

https://github.com/addrummond/ibex/blob/master/docs/manual.md

What about the PCIbex framework?

For several of the standard experimental paradigms in linguistics, the PCIbex framework directly ports controllers from IBEX. So for tasks like self-paced reading, you get the same experiment from a different style of code.

You can clone the PCIbex version of the workshop's SPR experiment here - https://farm.pcibex.net/r/nYsovX

PCIbex has great support for experiments involving multimedia (audio, video, etc.). The PCIbex documentation is a good place to get started - https://doc.pcibex.net

IBEX is always good to know due to the sheer amount of linguistics experiments already written in IBEX.

PCIBex is not difficult to pick up after IBEX, should you choose to learn it.

Thank you!

Many thanks to...

• Brian Dillon (UMass Amherst) for permission to use materials from the LSA Minicourse "Doing Experiments for Linguists" (Brian Dillon & Rodica Ivan)

• Jon Sprouse (UConn) for permission to use materials from LSA 2016, Experimental Syntax Workshop

  • Link to Jon's course on Introduction to Experimental Syntax Methods: https://sprouse.uconn.edu/courses/experimental-syntax

• Florian Schwarz and Jeremy Zehr for permissions to use materials from their PCIbex workshops.

• Attendees of the 2021 BK Winter School Workshop on Experimental Linguistics/Syntax at Sungkyunkwan University for valuable feedback on the workshop materials.