Once you’ve got software engineers applying for your job opening, you’ll need to determine who is good enough to interview, and who is not. Additionally, you’ll want to make sure the time investment made by you and your team has a high return on investment. In this part of our series on hiring software engineers as the CTO of a startup, we’ll talk about Blue Matador’s approach to the screening process and the insights I’ve gained in this part of this CTO experience. This process consists mainly of a code test and a phone screen.
If you’re just starting on this series, here are links to the rest of this series:
- Introduction to Hiring Software Engineers
- How to Write a Job Posting for Software Engineers
- How to Screen Software Engineers (this post)
- How to Interview Software Engineers (coming soon)
- How to Onboard a New Software Engineer (coming soon)
The Code Test
The first step in our screening process is a code test that’s designed to take less than an hour. When I get an application, I send them an email with a PDF attachment that clearly states our requirements, the problems, and the things we’re looking for (performance, code quality, etc). When applicants respond with their programs, a team of myself and one other engineer look at the code independently and then come together to determine if the applicant did well enough to continue.
Why Do the Code Test First?
As the CTO of a startup, you want to make sure that the time you spend on recruiting is a good return on investment. A code test is a great way to make sure that the people you end up advancing are all valid candidates for your position. You’ll get lots of unqualified applicants and there’s simply no time to call all applicants and find the good ones.
One might argue that starting with a code test can deter qualified candidates, but in my experience less than 10% of candidates gave up when presented with a code test. Of those that did, over half of them demonstrated another benefit of the code test: revealing abrasive personalities. One applicant immediately became belligerent, literally saying “3 questions is practically a full day of interviews. You're not that important to me.” This is an incredibly important reason to do a code test. If a candidate is abrasive during the interview process, they will almost certainly hurt team cohesion if hired.
Of course, it’s important to respect the time it takes to do a code test. Candidates are likely also doing code tests for other companies as well. If possible, offer to pay them for the time they spend on the code test.
Defining the Code Test
Having a defined code test and grading criteria is critical. If you have formally defined the problems and intentionally decided on a grading process, you are able to judge each candidate equally. To even the playing field, we also tell applicants to complete the exercise in whatever programming language they feel most comfortable with.
Candidates should be given a chance to shine, not be hobbled by language constructs they are unfamiliar with.
Your code test should take between 30 minutes and an hour. When you send the code test to the applicant, tell them the amount of time it should take and that you know their time is valuable, so they shouldn’t take more than an hour even if they don’t finish.
Next, define two or three problems that help you determine if the candidate would be a good fit for your workload. These problems should be relatively simple, but cover fundamental skills. For example, here’s a high level view of the problems we send out:
- An easy problem that experienced developers find extremely easy, but allows us to start getting a feel for coding style.
- Calling a simple and well-documented public API and using the results. This is important because we do a good amount of API integration at Blue Matador, so we want to make sure candidates can quickly understand APIs.
- An easy statistical algorithm. Our product is built a lot of statistical analysis of metrics to find anomalies in infrastructure health, so candidates need to be able to confidently work with numbers.
You’ll notice that as we developed the problems, we sought to find simplified versions of the work that we do on a daily basis. The typical code test consists of algorithm work, likely in an attempt to test “intelligence”, but creating clever algorithms is not the primary function of most developers, so we defined a test that would reveal the skills we actually needed. Likewise, as you develop the code test, you should use problems that utilize the underlying abilities required by your product.
Make sure to clearly define the problems and look for any ambiguities in the language. The goal is to make it so candidates can easily understand what you’re asking of them. Also include sample input and output. Taking the time to create a good code test respects the amount of time candidates will take completing it.
What to Look for in a Response?
Defining what a good response looks like is just as important as defining the problems themselves. Most candidates will be able to build correct solutions, so creating a list of programming practices that would impress you in the responses allows you to judge candidates equally. For example, with our API question, we decided that we would like to see asynchronous API calls and error handling.
In my experience, candidates who did best in the interview process wrote clean code, considered performance and optimizations, and documented the reasons they did what they did (especially when they explained why they took a shortcut).
The Phone Call
When a candidate successfully passes the code test, I schedule a 15 minute phone call with them. This phone call is mostly to set up a time for an in person interview, but it’s also a chance to reinforce the candidate’s interest in the position. I usually cover the following topics:
- The product vision and the problem we solve (since Blue Matador is an automated monitoring solution, it usually resonates strongly with developers)
- Describe the size of the team, and that the position is a chance to make a meaningful difference in a company
- Talk a little about the types of things the candidate would be working on
- Allow the candidate to ask any questions they might have
Essentially, the phone call is a great chance to get the candidate excited about working for you. Focus on helping them resonate with the mission of your company and having them start imagining what it would be like to be a part of your team.
Next Steps
Once you’ve scheduled the interview, you’ll need to actually carry out the interview. In the next post, we’ll talk about what to include in your interview.
- Introduction to Hiring Software Engineers
- How to Write a Job Posting for Software Engineers
- How to Screen Software Engineers (this post)
- How to Interview Software Engineers (coming soon)
- How to Onboard a New Software Engineer (coming soon)
