Google interviews vs the LSAT, and why AI will save us from both

If you apply to Google, there’s a less than 1% chance that you’ll get a job offer¹. If you apply to Stanford Law School, your chance of acceptance is just over 6%². With 3 million annual applicants, Google should probably try charging $85 per application.

My money’s on Stanford being harder. But I think both processes are deeply flawed. Because standardized tests don’t test what we think.

The flaw with standardized tests is part of a profound societal problem that we’ve known about for almost 40 years. But a solution is close…

In 2009, I was disillusioned with my tech career. In a moment of weakness, I decided to apply to law school. The first thing you do when applying to law school is take a practice LSAT. I scored a 161, 80th percentile. That’s pretty good, but not good enough to get into schools I wanted to go to.

So I took a small group LSAT prep course. They teach you how the test is constructed and how to beat it. 20 hours of prep material and several practice tests later I was scoring 173, 99th percentile.

So, what was the LSAT testing? My innate abilities? Nope. My level of preparation? Sort of. But if I’d just put 20 hours of study in, I’d have done worse. It was testing how well I was prepared.

It turns out this phenomenon in learning has been quantified since 1984. It’s named for the problem it unearthed: Bloom’s 2 sigma problem³. Benjamin Bloom discovered that students tutored 1:1 using certain techniques perform 2 standard deviations above the mean.

They go from the 50th percentile, to the 98th!

The problem is that despite this incredible discovery, no society can scale 1:1 tutoring at a reasonable cost. My group LSAT course cost me $1000, for example. That’s not even 1:1 time.

I never attended law school, because in 2010 I decided to look for work in the US and interviewed at Google. I put as much effort into preparing for those interviews as the LSAT. I read the infamous Steve Yegge blog post⁴ and I spent evenings grinding out data structures and algorithms textbook problems. I even solved problems on my bedroom mirror to mimic doing it on a whiteboard (in the times before Covid, on-site interviews were… on-site).

In the 10+ years I spent at Google, I conducted over 100 interviews myself, and I sat on many hiring committees as a manager. I don’t think their interview meant to become a standardized test⁵, but Google recruiting is a well oiled machine and that’s basically what it became.

In 2010 I did 5 interviews. 2 went great, 1 went good, 1 was okay, and 1 went poorly. Knowing what I know now, it was a borderline case and I barely got in. What I’ve never told anybody, is that one of the “great” interviews was a question I had seen before during practice. I hadn’t memorized the solution, but I knew up front it was a dynamic programming question. That luck was what tipped my getting in.

I really should have gotten a 1:1 tutor to be safe.

And that’s exactly what others do:

• I personally gave mock interviews and coaching to interns we wanted to see re-hired
• Stanford offers a course, CS 9⁶, that includes coaching from ex Google employees on how to pass interviews
• Like the LSAT, an entire industry has popped up around tech interview preparation⁷

So what’s really being tested here? Innate ability? Maybe, in some cases. But somebody with access to 1:1 tutoring who moves up two standard deviations is hard to keep up with.

I believe that talent can be found anywhere⁸. But access to a great tutor is not universal. If the goal is finding the best future lawyer to represent Stanford an as alumnus, or the best software engineer to work at Google, this pesky Bloom’s 2 sigma problem is obscuring the view.

So how do we fix that?

Many people recognize the problem. Google engineers volunteer their time trying to level the playing field. They offer interview preparation at schools that don’t have a CS 9 course. They reach out in places where opportunities for tutoring aren’t keeping up with the potential talent available. It’s noble and it’s good work, but it’s not scalable.

Marc Andreessen recently wrote a piece called “Why AI Will Save the World”⁹. He predicts that every student will have an “AI tutor that is infinitely patient, infinitely compassionate, infinitely knowledgeable, infinitely helpful”.

AI tutoring is a solution that scales. It solves Bloom’s 2 sigma problem.

I believe it will change the world.

It might even fix standardized testing.