Principal Engineers and their million dollar microbiomes

Principal Engineers at Facebook, Google, and Amazon¹ make over a million dollars per year in total compensation. What are those companies paying so much for?

One way to answer that question is to look at the interview process. You’ll notice heavily overlapping interview questions with the Staff Engineer level role that only pays half as much. It’s a fairly standardized process:

• Live coding to show basic competencies
• System design to show ability to architect systems
• Leadership/behavioural interviews to gauge work style and experience

But the higher you go on the technical leadership ladder, the harder (and thus more inaccurate) interviewing gets. Because the hiring manager isn’t really looking for somebody who knows the right answers to those questions.

System design interviews are especially easy to fake answers to. There are hundreds of YouTube videos you can watch to prepare². They’ll refresh you on the Gang of Four Design Patterns, summarize Designing-Data Intensive Applications, and give you the textbook answers on why you should use a SQL database vs a document store³.

Leadership interviews might be worse. They’ll select for wordcels with well prepared anecdotes. During the layoffs of the GFC, I got access to “Career Transition Services” as part of a severance package. A coach there helped me curate five stories about past work I’d done, making them possible to spin into answers to dozens of common leadership/behavioural interview questions. Frankly, they could have been completely made up.

But a Principal Engineer isn’t a wordcel with good memorization skills. Principal Engineers are the ones who make the biggest, most consequential, technical decisions at a company. These can define how tens or hundreds of millions of dollars are spent on compute or engineering salaries. Principal Engineers are the tiebreaker on the most contentious detailed decisions. They need to be able to move from high level technical vision, to being in the weeds during the toughest parts of implementation. They’ll prototype solutions to problems that have no textbook answer, unblocking paths to entirely new technical or business opportunities.

To steal from Amazon’s leadership principles⁴, they are right, a lot. And they do it under time pressure, technical complexity, and ambiguity.

They have million dollar gut instincts.

One of the most common pieces of advice I give to top performers is “trust your gut”. There are many situations where the data is confusing, untrusted, or hard to gather. Situations where nobody has seen the issue before, or multiple paths look equally good or bad. That’s where you need to lean on a technical leader with a well refined gut instinct.

So how do you build one? What is the fecal transplant of software engineering leadership?

Unless there’s a breakthrough in the study of the human microbiome, the only working prescription I’ve seen is experience. The best Principal Engineers have spent more time training their guts. They’ve worked on multiple interesting projects, in multiple technical domains. They tend to work more. They’re incredibly curious. They usually have several active prototypes to explore new problems.

They get their reps in.

So, if you’re a hiring manager looking to spend a million dollars to gain access to a well formed microbiome, what should you do?

Promote from within: With a current employee, you get to see a track record. Are they right, a lot? They’ve also been training on more relevant problems.

Use your network: Nothing you learn in an interview can make up for first hand knowledge from a trusted reference. They’ve seen the track record, over time and up close.

Get more detailed: Make your technical interviews more detailed. Have part of it be hands on so they zoom into details. Don’t ask questions that can be answered with “a distributed lock” or “add a caching layer”.

Look beyond the resume: This doesn’t always apply, but look for any public work they have. Read their publications, check out their personal or open source projects, and watch technical presentations they’ve given.

Don’t settle: You’ll be under immense pressure to fill the role. The lack of this person is probably holding you back. But filling it wrong will be worse. If you need great instincts on filesystems design, don’t hire somebody who is an expert on networking routing just because the company and title on their resume is impressive.

Or… maybe just ask for a stool sample?