Wait, Can I Actually Trust This? (A Guide for People Who Doomscroll)

Okay so picture this. You’re scrolling, half paying attention, and someone posts “studying with music makes you smarter, it’s SCIENCE” with a screenshot of something that looks vaguely official. And some part of your brain goes… does it though? Not disagreement, just a tiny alarm going off. A little internal “hm.”

That “hm” is the whole point of this post. You already have it. Most people do. You just haven’t been given the words for what it’s checking.

Let’s run that music claim through the wringer, because it’s a perfect little case study and nobody involved smokes anything.

First question your brain should ask: who’s talking?

Say the music claim comes from your friend, who read it in a group chat, which got it from an account that posts “facts,” which credits a “study.” Follow that chain far enough and it basically evaporates. Nobody in it researches anything for a living. It’s just vibes, forwarded.

Now imagine instead it comes from a sleep and learning researcher at a university, explaining their own published work. Same sentence, wildly different weight. Weighing credentials like that isn’t snobbery. It’s a useful shortcut: does this person know this stuff, and do they have anything to gain by you believing them?

That second half matters more than people give it credit for. If the “study” showing music helps you focus was paid for by a headphone company, you don’t have to assume they lied: you just have to notice they had a reason to want a specific answer, and read a little more carefully because of it. A source can be nice and still be interested. Both things can be true.

Second question: did they check the thing they’re claiming?

Here’s where it gets sneaky, because this is the part that even honest, well-meaning people mess up constantly.

Say researchers really do run a study on music and studying. They play music, then ask students afterward, “did that feel helpful?” Everyone says yes, it felt great, very chill. Headline gets written: “Music Helps You Study, Confirmed.”

Except… they measured how it felt, not how much you learned. Those are not the same thing. Feeling relaxed and retaining information are cousins, not twins. A study that wants to prove “music helps you learn” needs to test learning: quiz scores, recall, something measurable, not vibes after the fact. If it tests the wrong thing, it doesn’t matter how careful or honest everyone was. It’s still answering a different question than the one in the headline.

This mismatch happens constantly and it’s rarely on purpose. A school wants to know “did students learn from this assembly,” so they ask “did you enjoy the assembly.” Enjoyment and learning are not interchangeable: plenty of things are enjoyable and teach you nothing, and plenty of things are boring and extremely educational (looking at you, some textbooks). Measuring the wrong thing, confidently, is one of the most common ways a completely honest, completely accurate result still ends up misleading.

• +A stopped clock is right twice a day−It's also useless as a clock, because being occasionally correct isn't the same as measuring time. Same idea: a method that sometimes lands on a true answer by accident still isn't testing what it claims to test.
• +This has an actual name−Psychologists call this a construct validity problem: whether a measurement captures the thing you meant to study, rather than something else that happens to correlate with it. You set out to measure focus and somehow ended up measuring "did people have a nice time."

The construct validity idea traces back to Cronbach & Meehl, 1955, and the reliability-vs-validity framework used throughout this post follows the classic breakdown in Trochim’s Research Methods Knowledge Base.

Third question: would this still be true tomorrow?

Let’s say the music study did measure the right thing: real quiz scores, not vibes. Great. One more question: did they run it once, on twelve of their friends, on a random Tuesday? Or did other researchers, in other places, with other students, try it and land in the same neighborhood?

One result is a data point. It’s not nothing, but a single run can go sideways for a hundred boring reasons that have nothing to do with music: everyone was tired that day, the quiz was easy, half the group already knew the material. Genuine confidence shows up when the result keeps happening, on repeat, with people who have zero connection to the original researchers and zero reason to fudge it in the same direction. This isn’t hypothetical: when a large team of researchers tried to independently re-run 100 published psychology studies, only about 36% produced a statistically significant result the second time, even using the original materials (Open Science Collaboration, 2015).

This is also, weirdly, where your gut already has good instincts. You probably already trust “three different friends independently told me this restaurant is good” more than “one guy said so once.” That’s the same muscle. You already have it.

A slightly unhinged but genuinely useful mental picture

Imagine a bathroom scale that’s broken in a very specific way: it adds exactly ten pounds to every single reading, for everyone, every time, forever. Step on it a hundred times and it’s ten pounds off, a hundred times in a row.

That scale is, in a weird sense, extremely trustworthy: it’s shockingly consistent. And it is also completely wrong, every single time. Consistency and correctness are just not the same property, and a broken tool can absolutely nail one while failing the other.

So what do you do with all this

Next time something crosses your feed sounding extremely certain of itself, you don’t need a whiteboard and a research degree. Just ask, in whatever order feels natural: who’s telling me this, and do they have a reason to want me to believe it? Did they check the thing they’re claiming, or something that just looks similar? And has this held up more than once, for more than one person, or am I looking at a sample size of “some guy”?

You already ask versions of these questions constantly: about restaurants, about relationship advice, about whether that “everyone’s doing it” thing your friend swears by actually holds up. Turns out that instinct has a name. Three names, technically. Now you’re just doing it on purpose.

Want to see what happens when literally nobody asks these questions, for fourteen years straight, on a national scale? That’s the cigarette case study. Or if you’re more curious about why bad arguments feel so convincing in the first place, that’s cognitive bias 101.