That’s so Meta
Thanks to new lab member Rachel Duquette for writing this blog post about meta analysis!
When I was a kid, I remember doing experiments to verify science I had learned in school. I would roll a marble down a ramp and change the angle of the ramp to see how far the marble would go (and whether I could get the cat to chase it). Whether it’s launching a gummy bear with a popsicle stick, building a pendulum with paper clips, or dissolving sugar cubes in water, most 4th grade science is pretty easy to verify.
Once you move past the basics of science — the things we’re already pretty sure we know, the things you learn in school — and get into more complicated topics, things get a lot less simple. At the level of scientific research, the questions you ask become much more complicated and the answers much less clear.
With that uncertainty, it becomes much more common for different studies to come up with conflicting results. This is especially a concern in research concerning people. A marble will almost always roll farther if you increase the angle of the ramp, but every person is different. Plus, in questions of memory and language, there are so many different factors at play that it’s difficult to isolate just a few variables. But when different studies produce different answers, how do you even begin to go about evaluating them to figure out the truth? One answer is a meta-analysis.
What’s a meta-analysis?
Instead of conducting research with people or mice or marbles, a meta-analysis is a study on other studies. A meta-analysis starts with a topic or a question, some sort of relationship you’re looking to test. After gathering a bunch of already published studies on that same question, you use statistics to calculate their effect sizes. There are different ways to do this, but the idea is to tell a) what each study says and b) how sure we are that that study is right.
In general, a study with more participants will have a lower standard error, meaning it’s more likely that the result from that study is accurate. Conversely, a study with only few participants will be more prone to error.
As an example, think about flipping a coin. If you only flip the coin 4 times, it might be heads every time. From that, you could conclude that this coin always lands on heads — but in reality, it could just as easily have landed on tails every time. If you flip the coin 100 times, however, in the end, the coin will probably have landed on heads about half the time and on tails about half the time. A larger sample size increases the accuracy of the study.
For this reason, results from large studies are typically weighted more in the comparison of a meta-analysis, while smaller studies are weighted less. This helps evaluate the true effect.
Why do a meta-analysis?
Combining a bunch of studies into a meta-analysis can help investigate all sorts of potential problems in collecting and interpreting data.
For the same reason that they’re more prone to error, small studies are less able to get a statistically significant result. There are a variety of ways to calculate significance, but generally, they depend on some concept of statistical power. This is just a mathematical way to express the effect of the sample size. A study with lots of participants has high statistical power, while a small study has low statistical power.
This helps prevent false positives — finding a relationship when there really isn’t one — but it also can cause false negatives — finding no result when there really is one — especially if the overall effect is small. A small study might have evidence of a relationship between two things, but because there were so few participants, the effect wasn’t statistically significant. By combining data from multiple studies, a meta-analysis has high statistical power, so it can bring out effects like this.
The larger sample size in a meta-analysis also makes for a more diverse sample, meaning it’s a better representation of a whole population. In a single study, all the participants might share some characteristics, like geographic area, language, education level, or political opinions, for instance. Any of those factors could influence how they respond to the study.
By combining many studies, a meta-analysis reduces the chance that variables like these impact the data. This means that the conclusions of a meta-analysis are more generalizable: we can say with more certainty that they’re true for all people in a population.
Meta-analyses can also check for biases within studies. If the distribution of study results is skewed, which means there are a lot more studies finding positive effect than negative ones, it’s possible that something called publication bias is involved. Publication bias is the phenomenon where studies that find a negative effect or fail to find an effect at all — basically, that don’t support the anticipated result — are less likely to be published.
Let’s return to our coin-flipping example. If only the experiments where more heads than tails were flipped were recorded, you would conclude that heads are flipped more often than tails — when in fact both were equally likely, and the data was being manipulated. A graph called a funnel plot in a meta-analysis can help detect publication bias.
If many studies report just barely significant results, another type of mischief, called p hacking, might be at play. Because of how most statistics work, the more comparisons you test on a set of data, the more likely you are to find something that’s statistically significant but not actually true. In some instances, researchers will collect a ton of data and just run different tests until they find some result — never mind if it’s really accurate or not. By plotting the distribution of different p values in a p curve, a meta-analysis can look for this problem.
Our Meta-Analysis
In the Language Learning Lab, we’re working right now on a meta-analysis on the effect of working memory maintenance on long term memory. The question is this: if you hold something — a number, a word, a picture — in your head for a short period of time, will that help you remember it later?
This is an issue that’s been debated for a long time in the psychology community. There have been many studies finding that the two things are totally unrelated, that there’s no relationship, but many others that found the opposite. By conducting a meta-analysis, we hope to discover why this conflict exists and — hopefully — what the real answer might be.
For more information on meta-analyses and some examples, check out MetaLab!
