Study approval process

Remember, before your study can go live on Lookit, there are three things you’ll need to do:

  1. Sign and return the access agreement. This covers your entire lab. You only have to do this once.
  2. Get IRB approval for your study at your own institution.
  3. Get peer feedback on the study, and then submit it to Lookit for approval. This is the part we’ll discuss here!

Why is there a study review process?

Because Lookit is a shared platform with a shared reputation, it benefits all of us to uphold high standards for all Lookit studies. Families who have a good experience - the study is fun, it runs smoothly, the instructions are easy to follow, they get a gift card promptly or get quick friendly answers to their questions, their child isn’t upset by the stimuli - will come back. Positive media coverage likewise benefits all of us.

To make sure all studies meet community standards, we use two forms of review before studies are posted: another researcher outside your lab providing feedback (“peer review”), and Lookit staff reviewing studies before approving them for posting (“admin review”). While this may feel like “extra steps” compared with in-lab research, it’s an opportunity to improve both the participant experience and the science - saving you time later. So far, researchers who were frustrated by the extra steps have been convinced once they see how much their studies improve in the process.

What are the rules for external studies and ‘live’ sessions?

From the beginning of Lookit through most of 2021, all studies listed on Lookit were also run within Lookit, using the Lookit experiment runner. In contrast, the new “External Study” type recruits participants via Lookit but the experiment itself takes place elsewhere. This opens up the possibility for researchers to run “asynchronous” studies with any desired platform (e.g., Qualtrics) or “synchronous” studies (e.g., scheduled Zoom sessions).

In conjunction with the Children Helping Science/Lookit merger, we are expediting reviewing for external studies. Under this process, peer review is optional (but highly recommended) and admin review will be expedited; however, all external studies must still meet basic requirements.

You are also strongly encouraged to review the study guidelines that we use for reviewing Lookit experiments, and are always welcome to give and solicit peer review for any study type on Slack. This is probably the best way to learn about what features make studies successful on our platform, and we recommend that all new researchers contribute peer reviews so that you have seen at least one other remote study. Graduate students, and undergraduate students who are conducting mentored research, are considered peers and do not need to get special permission to contribute peer reviews.

If you don’t have IRB approval or an access agreement yet

You can still go ahead with steps 1 and 2 below so you’re ready to go as soon as possible! The “Lab” your study is associated with on Lookit will have to be approved to test before you’ll be able to submit in Step 3, which means you’ll need the access agreement set at that point.

Step 1: study preparation and self-review

First, get your study working smoothly, the way you want it to, and make any revisions from within-lab feedback. Please work through the self-review checklist before requesting peer review!

Step 2: peer review

Next, post a preview link for your study on the Lookit Slack #researchers channel to gather some feedback from researchers outside your group. This is like getting a bunch of fresh eyes at a lab meeting made up of researchers who work online! Check out the Slack channel to see many previous examples of how to post review requests, including links to your study and to the peer review form. Notice also that whenever someone posts a study review request, you will also see them volunteering to review other recently posted studies - trading reviews in this way is how we keep the system going!

The goal of peer review is to have someone go through the study exactly as a family browsing CHS would do. This means that you should always direct reviewers to the Lookit preview link, even if your study takes place externally. For instance, although much of our peer review documentation assumes that studies are asynchronous, for a synchronous study you should expect to demonstrate your entire procedure, from scheduling all the way to debriefing. Researchers who act as reviewers will start from a Lookit study link, sign up for a session and communicate with the experimenter exactly as a participant would, and attend & record a video chat session including all warmup, consent, and debrief that a participant would experience.

Peer reviewers can reference the peer review checklist and will be looking for things like:

  • Typos
  • Overall flow of the study - are there any abrupt transitions? do they feel like they know what to expect?
  • Places where your instructions are confusing, overwhelming, condescending, etc.
  • Cases where your stimuli or prompts might be unintentionally confusing, upsetting, or unnecessarily non-inclusive (e.g., you’ve assumed that each child has a mother and father at home, that a child who’s experienced the birth of a sibling still has that sibling, etc.; you’ve prompted children to describe something bad that happened to them assuming they’re going to think about stuff like their popsicle falling)
  • Whether your study description accurately reflects their experience
  • Experimental design issues that happen to occur to them (this isn’t primarily a scientific review)

With this feedback you can revise and improve your study, often the instructions and parent-facing text in particular.

You should also expect to return the favor and review other labs’ studies, as part of participating in the platform.

Step 3: Lookit submission and approval

When your study is ready and you’ve responded to the peer feedback, you can submit your study for Lookit approval so it can go live. This is the point where you’ll need to have your access agreement set up.

When you submit the study, you will be prompted to note any non-standard elements that require specific approval per the Terms of Use (e.g., integration of additional information about participants from another source) as well as what changes you made based on peer feedback.

At this point, Lookit staff will review your study, focusing primarily on the participant experience. This is also when any custom code you’re using will be reviewed for security or functionality issues.

Initial internal review can generally be completed within 10 business days. Revisions may be requested before the study can be approved to run. To minimize the number of rounds of review needed, researchers are strongly encouraged to polish their studies as much as possible before submitting - please don’t use us as a proofreading service!

Outcomes of Lookit review

In general, although we technically reserve the right not to host work at our discretion, the outcomes of the Lookit admin review process are “accept” and “revise and resubmit.” We’ll work with you to get your study ready to go.

In rare cases, a study may be fine to run on Lookit, but in the judgment of Lookit admin staff, not a good idea to advertise publicly via Lookit. (E.g., a study that is ethically designed, but stands an unusual chance of putting parents off coming back for other studies.) In this case a “partial approval” - to collect data, but not make discoverable - may be the final outcome. We consult with working group members before making such a decision to ensure it reflects community norms, and will devise a more formal process and expanded guidelines if this occurs more often.

Reapproval after changes

Most changes to studies require re-approval. If your study is active, paused, or approved, and you make changes to it, it will be automatically rejected and you’ll need to resubmit. If your study has already been approved, you’ll see a warning to this effect when you click “save” letting you know which fields will require re-approval to change.

For minor changes (“we fixed a typo”, “we clarified instructions”, “we’re stopping data collection for some conditions”, etc.) approval is quick - you do not go back into the same queue as for initial submissions. We approve studies the same day (often within 1 business day, but no promises).

If you want, you can let us know you’d like the study restarted in addition to reapproved. By default we just approve it and you can restart it when you’re ready.