How To Setup an Effective Coding Test (a Guide for Hiring Managers)

Deepti Chopra -

Assessing coding skills has now become an indispensable part of the technical hiring process- the volume of resumes you get from candidates who can’t code is overwhelming. We know resumes are a bad predictor of performance and in most cases Github activity is not a good measure of a developer skills either.

At Adaface, our subject matter experts design coding tests by carefully understanding your role and requirements. We sat with them to breakdown their process to set up coding tests that are in line with job requirements to reduce false positives and false negatives.

How to setup an Coding Test?

There are two important parts to keep in mind while setting up a coding test:

  • Questions that are in line with the job requirements
  • A form factor that does not intimidate candidates

Questions

The biggest problem with coding tests today is that they’re not always aligned with what the role requires. An Android developer does not necessarily need to know how to invert a binary tree. Or maybe they do. But the point is that if you’re using that on a coding test for that role, you should be very deliberate about it.

Here’s one way to think about what should go on a coding test:

Questions where you might still hire the person even if they couldn’t solve it/ got it completely wrong are pointless at the screening stage, such as a coding test.

The idea of a coding test is to eliminate candidates who are clearly not a good fit for that role. We’ve found that the most useful way to do it would be to use 2 must-be-able-to-code questions, one easier than the other and give some benefit of doubt to candidates who solve it partially. This eliminates candidates who basically can’t code (quite a lot of them), and then you can take the rest to the final rounds of interview.

Now to decide what those must-be-able-to-code questions should be, here’s what you need to answer first:

  1. Is algorithmic skill needed to be successful in that role? Put another way: does the role focus on lowering response time per API? Streamlining data for faster execution? Better frontend componentization for faster page loads? If the answer to this is an absolute yes, then- What level of algorithmic skills do you require? Would basic optimization skills be sufficient or do you need expertise with niche data structures/ algorithms?
  2. Do they need to have experience with a particular programming language or can someone smart pick it up on the job?
  3. Do they need to ship fast or be more meticulous and careful?

Your answers to these questions decide what questions go in the coding tests and what goes into in-person interviews. Here are few example questions (and the scenarios in which those questions would make sense on a coding test):

Find any one local minima in an array

  • Checks if candidate can work with basic data structures/ basic optimizations and write code to cover multiple edge cases.
  • Reflects on-the-job work: False
  • Useful for: Fresh grad roles

Debug code to validate or invalidate a series of zipcodes based on given constraints

  • Checks if candidate can understand code written by someone else and work with multiple loops/functions.
  • Reflects on-the-job work: True
  • Useful for: Software engineering roles requiring work on existing codebase (for candidates with 1–3 year experience)

Parse series of logs and extract metrics

  • Checks if candidate can understand data format, parse logs and calculate metrics
  • Reflects on-the-job work: True
  • Useful for: Data analysis/ backend roles

Merging sorted lists

  • Checks if candidate can use fundamental data structures effectively and think around time complexity optimizations.
  • Reflects on-the-job work: False
  • Useful for: Software engineering roles with focus on time complexity optimizations at API or database level for 1–3 year experience candidates

Top K candidates in a stream of data with space constraints

  • Checks if candidate can use slightly advanced data structures like priority queues effectively and work with constraints.
  • Reflects on-the-job work: True
  • Useful for: Backend engineers with 1–3 year experience for roles that involve system design

Find the shortest path for the delivery boy to reach customer

  • Checks if candidate can use advanced data structures like trees/ graphs effectively and work with advanced algorithms.
  • Reflects on-the-job work: True
  • Useful for: Algorithm design roles (at companies like Uber)

PS: the section above is meant to demonstrate the kind of questions you should be asking, not the exact questions themselves. Some of them are too popular for it to be a good idea to put them on a coding test.

Form factor

Once you have finalized questions, the other thing to keep in mind is the form factor. Here are the important things you need to consider for an effective coding test:

  1. Shorter assessments (45–60 mins) to make sure engineers are investing as less time as possible, while still enough to give them a fair shot.
  2. Easy-to-read questions because programmers should spend time solving the question, not understanding it.
  3. Granular scoring (as opposed to binary scoring). At Adaface, we implement this with hints for each question 🤓.
  4. Working boilerplate code with parsed inputs, because we’ve all spent too much time getting stuck here.

At Adaface, we set up custom coding tests tailored to expectations of the role. Our bot, Ada answers questions they have about the role/ company, guides them through each technical question interactively and creates a scorecard for the hiring manager to review at the end of the conversation.