I gave the feedback at one Google interview that they should send Google employees through to see how many get hired. Good to see they basically tried that.
The conclusion at the end that bringing someone on board is the ideal method is true I'm sure, but even that runs into the issue that employee evaluation is an even worse situation than the interview process.
You can openly see some managers panic when they realize they have no idea what their employees have been doing for the last 6-12 months when they're asked to provide feedback.
> I gave the feedback at one Google interview that they should send Google employees through to see how many get hired. Good to see they basically tried that.
They did, but not with the intention of doing anything about the problem.
This is a question of reliability, the conceptual 'correlation' of a measurement instrument with itself when measuring the same thing.
Reliability is one of two major concepts in psychometrics, the other being validity, the conceptual correlation between a measurement instrument and that part of reality that you're hoping to measure.
The question behind validity is "I want to know X; if I measure Y, how helpful will that be?". And the question behind reliability is "if I measure Z, how accurate will that measurement be?"
Yegge calls out both concepts explicitly, though not by name, in this essay:
>> The outcomes from interviewing are statistically terrible. Google did wave upon wave of analysis over the years, and all the results were incredibly depressing.
>> [reliability] To name just a few off the top of my head: interviewers barely agreed with each other. Put the same candidate in front of two of our sharpest people and you’d routinely get a confident “strong hire” from one and a flat “no” from the other.
>> [validity, though the 'problem' here is strongly confounded by a restriction of range issue] And once people were actually on the job, their interview scores told you next to nothing about how they’d do
>> [reliability] Hell, some of our star performers failed their Google interviews four or five times, finally got in after 2+ years...
>> [validity] ...and then outshone everyone else.
The discussion of how interviewing outcomes are statistically terrible would benefit from naming the ways in which they're statistically terrible. Knowing the problem you have is an important step toward solving it.
(And as a side note, the last I heard from Google, you're not allowed to interview more often than once a year. Interviewing five times in two years would seem to violate that policy.)
It is a basic theorem that the validity of any instrument is bounded above by the square root of the reliability. It isn't possible for an unreliable instrument to be tightly correlated to reality, because it is, by definition, not tightly correlated with anything. That's what it means to be unreliable.
Thus, any company that wanted its hiring process to be good would necessarily be extremely concerned with making that process accurate; you need to come to the same decision when you assess the same person. This is something that interviews cannot achieve except at extreme cost. You'd need far more than five interviews to get a reliable assessment from them, despite the claim in this essay that "any more than four interviews and you're just playin' with your food". Of course, the Google interviews aren't supposed to be reliable anyway, so in that sense the claim is probably accurate.
The prescription Yegge offers is valid. Multi-month work assessments will give you a strong, reliable, and valid signal. They're also very expensive.
Another thing the essay completely glosses over is that this problem has been recognized for a long time, and we already know how to do assessments that are reliable, valid, and cheap to perform. They're called standardized tests.
I couldn't find the text of this joke, attributed to Dirac. I'll paraphrase.
A man walks into a pet store. There's a parrot for $100, it says this parrot speaks perfect English . The one next to it is $1,000, and says, This parrot speaks 12 languages fluently.
Then there's a bedraggled looking, droopy, parrot, and its label simply says One Million Dollars.
Does it sing opera and has successfully run for President? the man asks with a sneer.
This parrot, says the store owner, _thinks_.
That's what this entire post is about - how to evaluate people with a series of attributes, score, correlate, blah blah blah.
Hire them, see if they think. If they don't, fire them. It's cheaper than this credential/signal rigmarole, most of which is about CYA legal b+llsh1t. Yes, it's a simplistic strategy and it doesn't work for Shoogle, Banthropic, Goober, whatever. You know what, boo f*cking hoo. You're a trillion dollar company, suck it up. You have a zombie horde at your doors and you're just upset the "true gems" are hard for you to spot amongst the slavering masses. You're going to heartlessly lay them off anyway in a few years. You SHOULD feel this pain and anguish of having to sort through them, constantly regretting all your choices. That's the only way to have balance in the Universe.
There’s an old Malcom Gladwell podcast episode, I think the show was Revisionist History, where he says he’s an interview nihilist. As long as the person seems reasonably capable, and can probably do a bit of what you need, hire them. Interviews are so hard to get right that what you’re saying ends up being most effective.
His idea has some merit but will require the old system to completely crash out before anything new will be considered and I'm not sure if it will crash or just keep limping along. If it really does crash out hopefully we will see multiple new strategies emerge as there are many possible options once the current one is off the table.
As yegge mentioned, there might be more appetite for trying out this idea now because there are many engineers who are currently unemployed. Offering them short co-op could be beneficial to both the engineer and prospective employer
The most talent dense place I've worked had a dead simple process - two one hour chats, one with your potential manager/team and one with the CTO or CEO. If things didn't work out, well you got sent away, probably happened twice per month. There was a particular meeting the CTO/CEO used and if you saw someone meeting with them there on a Friday afternoon, you would not see that person on Monday.
The place was not big - never got much beyond 100 engineers, but produced dozens of founders, VPs, GMs etc at well known companies, as well as engineers with very notable OS projects and lots of high-placed engineering in FAANG companies.
I gave the feedback at one Google interview that they should send Google employees through to see how many get hired. Good to see they basically tried that.
The conclusion at the end that bringing someone on board is the ideal method is true I'm sure, but even that runs into the issue that employee evaluation is an even worse situation than the interview process.
You can openly see some managers panic when they realize they have no idea what their employees have been doing for the last 6-12 months when they're asked to provide feedback.
> I gave the feedback at one Google interview that they should send Google employees through to see how many get hired. Good to see they basically tried that.
They did, but not with the intention of doing anything about the problem.
This is a question of reliability, the conceptual 'correlation' of a measurement instrument with itself when measuring the same thing.
Reliability is one of two major concepts in psychometrics, the other being validity, the conceptual correlation between a measurement instrument and that part of reality that you're hoping to measure.
The question behind validity is "I want to know X; if I measure Y, how helpful will that be?". And the question behind reliability is "if I measure Z, how accurate will that measurement be?"
https://en.wikipedia.org/wiki/Reliability_(statistics)
https://en.wikipedia.org/wiki/Construct_validity
Yegge calls out both concepts explicitly, though not by name, in this essay:
>> The outcomes from interviewing are statistically terrible. Google did wave upon wave of analysis over the years, and all the results were incredibly depressing.
>> [reliability] To name just a few off the top of my head: interviewers barely agreed with each other. Put the same candidate in front of two of our sharpest people and you’d routinely get a confident “strong hire” from one and a flat “no” from the other.
>> [validity, though the 'problem' here is strongly confounded by a restriction of range issue] And once people were actually on the job, their interview scores told you next to nothing about how they’d do
>> [reliability] Hell, some of our star performers failed their Google interviews four or five times, finally got in after 2+ years...
>> [validity] ...and then outshone everyone else.
The discussion of how interviewing outcomes are statistically terrible would benefit from naming the ways in which they're statistically terrible. Knowing the problem you have is an important step toward solving it.
(And as a side note, the last I heard from Google, you're not allowed to interview more often than once a year. Interviewing five times in two years would seem to violate that policy.)
It is a basic theorem that the validity of any instrument is bounded above by the square root of the reliability. It isn't possible for an unreliable instrument to be tightly correlated to reality, because it is, by definition, not tightly correlated with anything. That's what it means to be unreliable.
Thus, any company that wanted its hiring process to be good would necessarily be extremely concerned with making that process accurate; you need to come to the same decision when you assess the same person. This is something that interviews cannot achieve except at extreme cost. You'd need far more than five interviews to get a reliable assessment from them, despite the claim in this essay that "any more than four interviews and you're just playin' with your food". Of course, the Google interviews aren't supposed to be reliable anyway, so in that sense the claim is probably accurate.
The prescription Yegge offers is valid. Multi-month work assessments will give you a strong, reliable, and valid signal. They're also very expensive.
Another thing the essay completely glosses over is that this problem has been recognized for a long time, and we already know how to do assessments that are reliable, valid, and cheap to perform. They're called standardized tests.
I couldn't find the text of this joke, attributed to Dirac. I'll paraphrase.
A man walks into a pet store. There's a parrot for $100, it says this parrot speaks perfect English . The one next to it is $1,000, and says, This parrot speaks 12 languages fluently.
Then there's a bedraggled looking, droopy, parrot, and its label simply says One Million Dollars.
Does it sing opera and has successfully run for President? the man asks with a sneer.
This parrot, says the store owner, _thinks_.
That's what this entire post is about - how to evaluate people with a series of attributes, score, correlate, blah blah blah.
Hire them, see if they think. If they don't, fire them. It's cheaper than this credential/signal rigmarole, most of which is about CYA legal b+llsh1t. Yes, it's a simplistic strategy and it doesn't work for Shoogle, Banthropic, Goober, whatever. You know what, boo f*cking hoo. You're a trillion dollar company, suck it up. You have a zombie horde at your doors and you're just upset the "true gems" are hard for you to spot amongst the slavering masses. You're going to heartlessly lay them off anyway in a few years. You SHOULD feel this pain and anguish of having to sort through them, constantly regretting all your choices. That's the only way to have balance in the Universe.
There’s an old Malcom Gladwell podcast episode, I think the show was Revisionist History, where he says he’s an interview nihilist. As long as the person seems reasonably capable, and can probably do a bit of what you need, hire them. Interviews are so hard to get right that what you’re saying ends up being most effective.
Edit: Didn’t link it initially because I thought it would be hard to find. Turns out it’s not. https://www.pushkin.fm/podcasts/revisionist-history/hamlet-w...
[dead]
His idea has some merit but will require the old system to completely crash out before anything new will be considered and I'm not sure if it will crash or just keep limping along. If it really does crash out hopefully we will see multiple new strategies emerge as there are many possible options once the current one is off the table.
As yegge mentioned, there might be more appetite for trying out this idea now because there are many engineers who are currently unemployed. Offering them short co-op could be beneficial to both the engineer and prospective employer
The most talent dense place I've worked had a dead simple process - two one hour chats, one with your potential manager/team and one with the CTO or CEO. If things didn't work out, well you got sent away, probably happened twice per month. There was a particular meeting the CTO/CEO used and if you saw someone meeting with them there on a Friday afternoon, you would not see that person on Monday.
The place was not big - never got much beyond 100 engineers, but produced dozens of founders, VPs, GMs etc at well known companies, as well as engineers with very notable OS projects and lots of high-placed engineering in FAANG companies.
And good riddance.