Less noise, more data. Get the biggest data report on software developer careers in South Africa.

Dev Report mobile

ChatGPT is Breaking the Technical Assessment — What To Do About It

3 February 2023 , by Karin Bothma

It's likely ChatGPT has already broken your traditional technical assessment. With the advent of AI tools like ChatGPT, you need to rethink your approach to assessing developers' skills and making solid hiring decisions. As an Engineering Hiring Manager at OfferZen, here are my key takeaways on how to do this.

ChatGPT is Breaking the Technical Assessment

How ChatGPT Has Broken the Technical Assessment

ChatGPT is an AI tool that has skyrocketed in popularity. Developers can use it to write and debug code and even get explanations for code changes. While you might see some errors and low-quality answers, for the most part, it’s a powerful tool that I expect will continue to see widespread adoption as it evolves.

As I was experimenting with its capabilities, I wondered what impact ChatGPT has on the technical hiring process. It turns out the implications are profound.

ChatGPT can solve many forms of technical assessment

Some forms of technical assessment rely on providing candidates with code ahead of time to analyse as part of a technical discussion or code review.

Using ChatGPT, candidates can get accurate answers and suggestions for code changes. The only safe territory seems to be questions requiring problem-solving abilities and creativity.

What happened when I fed OfferZen’s technical test through ChatGPT? It provided many of the answers I was looking for using our assessment rubric, especially those aimed at more junior and intermediate candidates. This means that candidates can do the same before the interview and do fairly well without producing these answers themselves.

See this example of ChatGPT providing the answer to a question from a technical test:

ChatGPT answering a technical assessment

Asking a candidate to write code ahead of time poses similar challenges: Many take-home assignments can be solved by ChatGPT, especially if it relies on less complex or self-contained problems.

In these cases, ChatGPT can give good explanations for code changes or provide code suggestions that could pass a technical assessment.

Ways To Adapt Your Technical Interview Process

Here are some ways to mitigate the effects of AI tools like ChatGPT on various forms of technical assessment:

1. Use ChatGPT to refine your test

Before you use a technical test in your interview process, run it through ChatGPT to see what answers it gives you. You can use this to refine your questions or the problems you’re asking candidates to solve — anything that relies on straightforward solutions wouldn’t make for a good test.

Focus on questions that test candidates’ creativity, ability to give thoughtful answers, familiarity with specific technologies, and application of technical concepts beyond what ChatGPT can provide.

The downside is that junior developers might struggle to do well in some of these tests, simply from a lack of experience. A different approach might be needed when assessing junior developers. This brings me to my next point:

2. Use methods that test candidates in a live setting

Interviewing techniques that don’t provide candidates with code ahead of time, like pair programming and whiteboard interviews, will be less affected by ChatGPT.

You could also consider only revealing code to the candidate during the interview. The drawback of this is that candidates do not have time to familiarise themselves with the code, which will raise candidates' anxiety. You will also have to adjust the interview length to give the candidate enough time to read through and understand the code.

If you’re conducting a remote interview, this doesn’t entirely rule out candidates using ChatGPT in the background, but it would be easier to pick up on from interactions with the candidate.

3. Make ChatGPT part of your interview

Lastly, you can also work with AI rather than trying to circumvent it. Getting great answers from ChatGPT requires good domain knowledge and creativity — these are skills we’re looking for in a good candidate. You can allow the use of ChatGPT and make it part of your assessment.

Asking candidates to solve a problem while allowing for the use of ChatGPT can reveal their level of expertise and creative thinking abilities. For example, let candidates share their methods for solving the problem with the help of ChatGPT.

The benefit is that this is more authentic to how developers do their job in the real world or will likely work in future.

Eventually, all developers will use ChatGPT, or some form of AI, in their day-to-day work. When the internet came along, we tried to curb its use when assessing candidates. Today, we accept it as part of the assessment process.

We need to figure out how to work alongside AI: How can you gauge a person's technical ability, even when AI augments it? The solutions we come up with could raise the bar for technical interviews and, in the process, get rid of technical assessments that do not test for real-world ability.


Karin Bothma is a senior software engineering manager at OfferZen, with extensive experience in software development and leading engineering teams. She currently leads an engineering team at OfferZen and is helping other engineering managers drive performance improvements in their teams. When she’s not coaching teams to reach their highest potential, she’s out on the trails with her Jack Russels or playing board games with friends.

Read more

How to Conduct an Effective Tech Assessment.

Recent posts

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.