Technical Tests: You’re Doing It Wrong, Part 1

Because so many of us have experienced both sides of the interview table, the London Java Community has a slight obsession with discussing approaches to recruitment. The nice thing about these conversations is you see both points of view—the candidates and the techies who are responsible for hiring.

Our most recent conversation was (again) about the value of technical tests during the interview process. I’m not sure there’s another topic that generates such diverse and yet all “correct” points of view. If there’s one thing I’ve learnt from seeing this conversation again and again, it’s that there is no One True Way to test a candidate’s technical ability. If there was, we’d all be doing it.

What do you mean, Technical Test?

First let’s lay out what constitutes technical tests. From my own experience, I’ve done:

• Coding test to be completed at home which either gets you to the next round or not
• Coding exercise completed at home, with additional pair programming on it in interview
• Pair programming in the office on a standard exercise
• Pairing at a computer on a real project.
• Remote coding using google docs or collabedit. These usually focus on small, contained problems like data structures and design patterns.
• Whiteboard exercises in the interview room
• Completing a non-specific-language programming test
• Critiquing existing code, for things like code style or concurrency bugs
• Working through a question with no ‘correct’ answer – “How many liters of petrol are sold each year in the UK?”
• Answering a series of questions on a specific API or syntax (includes things like “what’s the third normal form of this database schema”)
• Certification-style multiple choice questionnaire
• Working through designing an imaginary system

Obviously there are questions in the face to face which are designed to probe your technical ability, but I’m looking specifically at things that could be called a test.

The good news is that as an employer, you have a range of different options open to you for testing the technical competence of applicants. The bad news is that a) each of these takes a different amount of effort from the candidate and from the hiring organization and b) as a hiring organization, you really need to know what it is you’re looking for and what your passing criteria should be.

Who do you want to hire?

Let’s address the second point first. It really is amazing and surprising that companies get this so wrong. We all want to be Google, or Facebook. But copying their style of interviews (especially if they turn out to be… less than optimal) will not turn a company into Google or Facebook. Even Google isn’t really the image you have of Google. But every company is different, and the best way to sell that company and to find, attract, and hire the best people, is to understand the company, the team, the environment and the role. Setting and meeting expectations is a good way to ensure that when a candidate says yes, they know what they’re getting in to. It also means that if the candidate has been judged against criteria that really are important to getting the job done, there’s a good chance that when the candidate is offered and takes the job, they’re the right person for that role.

How much effort is it really?

Knowing what you’re recruiting for and testing candidates based on those criteria is one thing. Now let’s look at the other point I mentioned—time and effort required.

In the table below, I’m using a scale of None -> Low -> Medium -> High -> Highest. Where there are two entries (e.g. Low/Medium), the first is if the test is done at home, the second is if they must travel to a location for interview. Any travel (especially somewhere like London) will obviously substantially increase the time and effort required.

Time cost should be fairly obvious. Value gained for the candidate is how much they learn about the organization/role they are interviewing for. Value gained for the employer is how much they will learn about how well the candidate can do the role (assuming it’s a programming role). Prep is the up-front preparation for the test – picking the test or test materials, and training people on how to evaluate the candidate’s submission. This is more of a one-off cost than an ongoing cost, although new interviewers will need to be trained.

I believe that a company’s selection of technical test looks at columns three and four. This is understandable—after all, they want to get the best value from the least effort. Developers are usually required to mark/assess a candidate’s technical ability, and a developer’s time is expensive. So in London there is a trend towards the first tech test – set the candidates an exercise to be completed at home, and use it as a way to determine if they should make it to the next round of the process. Often the multi-guess certification-style test will be used too, but usually onsite so the candidates can’t cheat.

Why the best developers don’t do technical tests

The problem is that organizations don’t always look at the costs to the candidate. Maybe because they see there are a lot of candidates out there and they believe they can play a numbers game—for every candidate who doesn’t have the time or inclination to complete the tech task, there are several who are willing to. But what organizations may not not realize is the true cost to themselves. Candidates are not all created equal. What if the ones that don’t have time / can’t be bothered to take the tests are the best candidates? How likely is this?

Pretty likely.

Why?

• The most experienced developers have families, life outside of work, or have simply been through enough tech tests in their lives that they just don’t want to do another made-up test for a company they’ve never heard of.
• The developers who are most active in the open source community are being snatched up by companies like MongoDB, who are looking at developers’ real-life code in GitHub and judging them based on real examples of actual working projects. They can offer them a job without forcing them to spend their free time on a code that will never be used.
• The technologists who can explain tech concepts to other developers at conferences, or via their blogs, are being offered jobs at companies who see value in being the type of company that adds to the overall technical community.

That’s just a few examples. However, it gives you an idea of why some organizations might be missing out on technical talent, simply by having a process that’s the most convenient for them without considering the opportunity cost of losing good candidates.

Forget “superstar” developers, most developers aren’t doing your test

We’ve talked about the type of tests. Let’s talk about when the test is set, because this contributes to the opportunity cost.

As an employer, it’s easy to pick a test that suits your timescale best. Often the temptation (and seemingly logical thing to do) is set the tech test right after the CV screening, i.e. before the candidate speaks to a human in the company. This keeps the cost to the company very low, because you don’t need to invest any time in phone screens or face to face interviews with a candidate who doesn’t know the difference between an if statement and a for loop. So you ping out your standard technical test, and only those developers who can really code will make it onto your premises. Right?

But lets think about the candidate again. For them it’s a numbers game too. They’ll probably be applying for a number of jobs at the same time. They’ve sent a CV to them all, and hopefully a customized cover letter. But now they have to complete a different technical test for every one of them. In my experience from both sides of the process, a technical test can take somewhere between 1 and 12 hours, the average time people spend on them is probably round about 4 hours. If a candidate is going to spend 4 hours on each test to do it properly (or at least, well enough to not be ashamed of it), they’re going to select the companies they most want to work for, and do those tests first. They might get around to the other tests, they might give them 30 minutes or so, or they might not bother.

There are lots of things that can make a candidate choose your company as a place they want to work, but you can almost guarantee that unless you are Google or Facebook, not bothering to speak to them personally before sending them your aimed-at-all-developer-levels test is going to move you down their list—if you don’t care about them, why should they care about you? There are a lot of candidates out there, but there are a LOT of jobs. And in somewhere like London, there are a lot of interesting places to work, in both the enterprise and startup spaces.

So once again the cost/value metric is not working out for the recruiting organization—by picking the least cost option to them, by setting the test early in the process, they could be losing good quality candidates. But by setting it late in the process, they could be wasting their own valuable time. It’s a difficult balance to get right. As a simple metric though, the more well known and desirable you are as an organization, the earlier you can afford to set the test—if candidates already want to work for you, they will spend the time and effort. If you’re an unknown company, or maybe don’t have that developers-aspire-to-work-there image, you need to have a more personal approach with your candidates to make them want to work for you.

When coaching interviewers, a previous boss of mine said he wanted all candidates to feel excited about the company, to want to work there, as even if they didn’t get the job they would tell all their friends what a great place it was. Do not underestimate the value of word of mouth – even in a city like London, the tech community is a small world.