An Engineering Leader’s Guide to Assessing Candidates in 2023

If you’re reading this guide, we’ll assume you’re no stranger to interviewing and assessing candidates for your team. You don’t need us to tell you making the wrong decision is high-stakes, can become costly and will make or break your team.

The clock is ticking, and you only have 60 minutes. How do you make the best use of this time to identify a brilliant engineer and more importantly, how do you assess technical talent in a hiring landscape that’s constantly evolving?

About our Expert 

Adam Mullen is an experienced Technical leader with an impressive track record spanning 15 years in Product Engineering. Having held senior positions with the likes of Booking.com and most recently Vinted as Director of Engineering, Adam’s leadership experience is well-rounded and versatile.

At Vinted, Adam was tasked with building their Product Engineering hub in Berlin, which meant optimising the hiring process for technical roles and defining how engineers would be assessed based on their technical abilities.

In this guide Adam talks us through his step-by-step process and advice on assessing candidates in the modern age of tech hiring.

To ensure nobody in the process wastes time, a productive and informative briefing call with TA is essential to avoid a pipeline of irrelevant candidates. First things first: align with your team. Collaborate closely with your team on what they need from this hire. Next, your brief to talent acquisition should answer the following questions:

What problem/s will this team member solve?

What will this person support the existing team with? 

What does this employee’s day-to-day look like in order to fill this gap?   

What is our process for this hire and what is the intention of each interaction?

Instead of trying to train TA on how to evaluate the candidates’ technical experience, it’s essential that TA leave the call asking ‘Is this someone from a personal standpoint I am happy to progress to they next round’. 

At a certain point of each unsuccessful process it’s essential to revisit the original brief to TA. “Elimination is elegance” says Adam, “sometimes you need to question whether the role is wrong or whether you’re fishing with the wrong bait”. Oftentimes you discover during interviewing that a skill is not necessary. This is the point at which you should be sharing this feedback with your talent team to adjust the role’s requirements.

Don’t underestimate the importance of team buy-in when it comes to overall process and technical assessment. Make your own life easier by training up your successor to understand and challenge your hiring process. “It’s like pair programming but for recruitment” says Adam. “If your successor is clear on what to look for and what makes a good developer great, they can support you in managing this process across various roles.” 

Hiring can be laborious for the whole team. You can give your team some time back by setting up automation to assess technical assignments or set up multiple choice assignments where possible.

Score cards that work

Opinions on scorecards are divided. For Adam, subjectivity runs loose in unstructured interviews. Questions and scoring systems often depend on interviewers’ moods or biases with some interviews taking an hour while others might span only a few minutes. So what’s the deal with scorecards? Should you be using them or not? 

➡️ Read Adam’s advice on how to make an interview scorecard for tech interviews. 

Tech tests are alive and well. However, Adam stresses that assignments should always be based on a real-life example from within your codebase. Over the years, Adam’s adapted the technical test so that his team spends no more than 15 minutes on reviewing the assignment. How? With a little something he likes to call ‘spot the difference’. 

Can they code? 

Across all technical roles they provide candidates with an extract of the code base that includes hidden problems to identify. Found 17/20 of the problems? A good engineer. Found 23/20 – a great engineer! Doing a simple technical test that doesn’t require weeks from a candidate, still allows you to assess whether they know how to read code and it shows that you respect their time. 

At this point you need to have alignment with your team on who goes through to the next round – agree with the rest of your team which scores are a pass and which is a fail. Remember to ttribute a score to each “improvement point” to keep the assessment fair and consistent. 

How well can they code?
Happy days, your candidate can code! Moving onto the next round means that the candidate needs to motivate and discuss their assignment in a technical follow up. 

At this point you can explore how candidates got to certain problems, how they knew about the error, did it relate to a previous experience or problem they’ve had to solve? Why do they think x,y,z is important to consider? If a candidate can justify their solution and communicate their reasoning effectively, it’s safe to say you can work with them.

It’s important that candidates feel like they have control over the process, says Adam. “I think building a relationship with the candidate as a hiring manager instills a lot of trust in the company and leadership.”

Calendar integrations are your friend
Give the candidate the option to check your availability and choose a slot which works best based on their availability, then make sure you have your integrations to Greenhouse or Recruitee set up to automate and streamline scheduling and tracking.”

Leaving everything to your TA team to manage from a process point of view is not effective.

“As hiring manager, your role is to build rapport and monitor the candidate’s feedback about the stakeholders they are meeting with, and doing communication handovers and check-ins like “Jessy will be in touch with you to schedule your Technical” or “How was that call with Sheldon?”. 

When it comes to compensation, it’s important to talk about improvement areas during the offer stage, says Adam. Especially when building a new tech hub, you want to set a base pay per experience level. If your compensation structure pays 75k for a senior engineer but their expectation is more, talk through the candidate’s improvement areas in order to reach the next experience level. If the candidate runs a special project for instance, they could qualify for a bonus and if you clearly define a development plan, a promotion. Something that comes in handy when managing candidate expectations is a Role and Responsibilities framework tailored to your organisational structure. Something that could come in handy to review the competency of your candidates based on your team’s structure is a Direct Contributors Competency Framework: 

⬇️ Download this Competency Framework Template 

⬇️ Download this template for Job Families for Engineers

“I personally like to deliver the offer to a successful candidate myself. This allows me to give them a detailed overview of what their first year with the company will look like from an earning point of view. This includes their total possible compensation, stock options, learning budgets and what they need to do to get an increase, bonus or promotion.”

Another tool Adam refers when evaluating candidates and growth paths is a breakdown of Job Families for Engineers which allows him to assess candidate abilities based on what their team’s bar is for themes like client solution focus, data abilities, collaboration, ownership, stakeholder management, self development and growing others.

On the topic of ChatGpt and how Adam sees AI impacting on our ability to effectively assess candidates in the future, Adam’s response was “they are still just tools, if it helps them get to the right answer quicker, so be it”. It might be that a few years from now we start asking candidates to put together a piece of work by using AI.

“AI will never be able to review itself for fitness. Ten years from now it will still be on us to validate that the solution will solve our problem.”