Study approval process¶
Remember, before your study can go live on Lookit, there are three things you’ll need to do:
- Sign and return the access agreement. This covers your entire lab. You only have to do this once.
- Get IRB approval for your study at your own institution.
- Get peer feedback on the study, and then submit it to Lookit for approval. This is the part we’ll discuss here!
Why is there a study review process?¶
Because Lookit is a shared platform with a shared reputation, it benefits all of us to uphold high standards for all Lookit studies. Families who have a good experience - the study is fun, it runs smoothly, the instructions are easy to follow, they get a gift card promptly or get quick friendly answers to their questions, their child isn’t upset by the stimuli - will come back. Positive media coverage likewise benefits all of us.
To make sure all studies meet community standards, we require two forms of review before studies are posted: at least one other researcher outside your lab will provide peer feedback, and Lookit staff will review as well. While this may feel like an “extra step” compared with in-lab research, it’s an opportunity to improve both the participant experience and the science - saving you time later. So far, researchers who were frustrated by the extra steps have been convinced once they see how much their studies improve in the process.
What about external studies and ‘live’ sessions?¶
From the beginning of Lookit through most of 2021, all studies listed on Lookit were also run within Lookit, using the Lookit experiment runner. In contrast, the new “External Study’’ type recruits participants via Lookit but the experiment itself takes place elsewhere. This opens up the possibility for researchers to run “asynchronous” studies with any desired platform (e.g., Qualtrics) or “synchronous” studies (e.g., scheduled Zoom sessions).
Study review for asynchronous studies that take place outside of Lookit are subject to the same review process as Lookit-internal studies. Again, this is because we all share a reputation, and positive or negative experiences that affect a family in one study affect all of us. When you ask for peer review, you must have the ‘Lookit part’ of your study created and set up exactly as your participants would experience it; researchers who act as reviewers for it should start at Lookit and run through the entire study experience exactly as a participant would.
Study review for synchronous (video) studies are also subject to peer and internal Lookit review. Right now, much of our peer review documentation assumes that studies are asynchronous, but you should expect to demonstrate your entire procedure (from scheduling all the way to debriefing) to at least one other researcher during this process. Researchers who act as reviewers should start from a Lookit study link, sign up for a session and communicate with the experimenter exactly as a participant would, and attend & record a video chat session including all warmup, consent, and debrief that a participant would experience.
If you don’t have IRB approval or an access agreement yet¶
You can still go ahead with steps 1 and 2 below so you’re ready to go as soon as possible! The “Lab” your study is associated with on Lookit will have to be approved to test before you’ll be able to submit in Step 3, which means you’ll need the access agreement set at that point.
Step 1: study preparation and self-review¶
First, get your study working smoothly, the way you want it to, and make any revisions from within-lab feedback. Please work through the self-review checklist before requesting peer review!
Step 2: peer review¶
Next, post a preview link for your study on the Lookit Slack #researchers channel to gather some feedback from researchers outside your group. This is like getting a bunch of fresh eyes at a lab meeting made up of researchers who work online!
Peer reviewers can reference the peer review checklist and will be looking for things like:
- Overall flow of the study - are there any abrupt transitions? do they feel like they know what to expect?
- Places where your instructions are confusing, overwhelming, condescending, etc.
- Cases where your stimuli or prompts might be unintentionally confusing, upsetting, or unnecessarily non-inclusive (e.g., you’ve assumed that each child has a mother and father at home, that a child who’s experienced the birth of a sibling still has that sibling, etc.; you’ve prompted children to describe something bad that happened to them assuming they’re going to think about stuff like their popsicle falling)
- Whether your study description accurately reflects their experience
- Experimental design issues that happen to occur to them (this isn’t primarily a scientific review)
With this feedback you can revise and improve your study, often the instructions and parent-facing text in particular.
You should also expect to return the favor and review other labs’ studies, as part of participating in the platform.
Step 3: Lookit submission and approval¶
When your study is ready and you’ve responded to the peer feedback, you can submit your study for Lookit approval so it can go live. This is the point where you’ll need to have your access agreement set up.
At this point, Lookit staff will review your study, focusing primarily on the participant experience. This is also when any custom code you’re using will be reviewed for security or functionality issues.
Initial internal review can generally be completed within a week. Revisions may be requested before the study can be approved to run. To minimize the number of rounds of review needed, researchers are strongly encouraged to polish their studies as much as possible before submitting - please don’t use us as a proofreading service!
Outcomes of Lookit review
In general, although we technically reserve the right not to host work at our discretion, the outcomes of the Lookit admin review process are “accept” and “revise and resubmit.” We’ll work with you to get your study ready to go.
In rare cases, a study may be fine to run on Lookit, but in the judgment of Lookit admin staff, not a good idea to advertise publicly via Lookit. (E.g., a study that is ethically designed, but stands an unusual chance of putting parents off coming back for other studies.) In this case a “partial approval” - to collect data, but not make discoverable - may be the final outcome. We consult with working group members before making such a decision to ensure it reflects community norms, and will devise a more formal process and expanded guidelines if this occurs more often.
Reapproval after changes¶
Most changes to studies require re-approval. If your study is active, paused, or approved, and you make changes to it, it will be automatically rejected and you’ll need to resubmit. If your study has already been approved, you’ll see a warning to this effect when you click “save” letting you know which fields will require re-approval to change.
For minor changes (“we fixed a typo”, “we clarified instructions”, “we’re stopping data collection for some conditions”, etc.) approval is quick - you do not go back into the same queue as for initial submissions. We approve studies the same day (often within 1 business day, but no promises).
If you want, you can let us know you’d like the study restarted in addition to reapproved. By default we just approve it and you can restart it when you’re ready.