SUBMIT


  close
The data science interview process is something that we have seen evolve over the last 5-10 years, taking on several shapes and hitting specific fads along the way. Back when DS got popular, the process was a lot like every other interview process - questions about your resume, some questions about technical topics to make sure that you knew what a person in that role should know, etc. Then came the "well, Google asks people these weird, seemingly nonsensical questions and it helps them *understand how you think!".* So that became the big trend - how many ping pong balls can you fit into this room, how many pizzas are sold in Manhattan every day, etc. Then came the behavioralists. Everything can be figured out by asking questions of the format "tell me about a time when...". Then came leetcode (which is still alive). Then came the FAANG "product interview", which has now bred literal online courses in how to pass the product interview. I hit the breaking point of frustration a week ago when I engaged with a recruiter at one of these companies and I was sent a link to several medium articles to prepare for the interview, including one with a line so tone-deaf (not to be coming from the author of the article, but to be coming from the recruiter) that it left me speechless: >As I describe my own experience, I can’t help thinking of a **common misconception** I often hear: it’s not possible to gain the knowledge on product/experimentation without real experience. I firmly disagree. I did not have any prior experience in product or A/B testing, but I believed that those skills could be gained by reading, listening, thinking, and summarizing. I'll stop here for a second, beacause I know I'm going to get flooded hate. I agree - you can 100% acquire enough knowledge about a topic to pass "know" enough to pass a screening. However, there is always a gap between knowing something on paper and in practice - and in fact, that is *exactly* the gap that you're trying to quantify during an interview process. And this is the core of my issue with interview processes of this kind: if the interview process is one that a person can prepare for, then what you are evaluating people on isn't their ability to the job - you're just evaluating them on their ability to prepare for your interview process. And no matter how strong you think the interview process is as a proxy for that person's ability to do the actual job, the more efficiently someone can prepare for the interview, the weaker that proxy becomes. To give an analogy - I could probably get an average 12 year old to pass a calculus test without them ever actually understanding calculus if someone told me in advance what were the 20 most likely questions to be asked. If I know the test is going to require taking the derivative of 10 functions, and I knew what were the 20 most common functions, I can probably get someone to get 6 out of 10 questions right and pass with a C-. It's actually one of the things that instructors in math courses always try (and it's not easy) to accomplish - giving questions that are not foreign enough to completely trip up a student, while simultaneously different enough to not be solvable through sheer memorization. As others have mentioned in the past, part of what is challenging about designing interview processes is controlling for the fact that most people are bad at interviewing. The more scripted, structured, rigid the interview process is, the easier it is to ensure that interviewers can execute the process correctly (and unbiasedly). The problem - the trade-off - is that in doing so you are potentially developing a really bad process. That is, you may be sacrificing accuracy for precision. Is there a magical answer? Probably not. The answer is probably to invest more time and resources in ensuring that interviewers can be equal parts unpredictable in the nature of their questions and predictable in how they execute and evaluate said questions. But I think it is very much needed to start talking about how this process is likely broken - and that the quality of hires that these companies are making is much more driven by their brand, compensation, and ability to attract high quality hires than it is by filtering out the best ones out of their candidate pool.