Can Programmers program?

November 10 2013

10 comments 0
motif

199 out of 200 applicants for every programming job can’t write code.

“A surprisingly large fraction of applicants, even those with masters’ degrees and PhDs in computer science, fail during interviews when asked to carry out basic programming tasks. “

When I first heard this years ago I initially thought it had to be inaccurate. I had seen a large number of developers flunk interview coding tests and write really bad code, including some of this. But how could that many career developers out there pass for great programmers, yet when it comes down to actually producing code couldn’t do it well.

In my previous company I saw developers there using FizzBuzz with candidates during the interview process and I asked myself why are we testing for this? Who could not pass it?

“Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.”

And yet a large percentage of people couldn’t do it. It’s not like they can’t code their way through very complex algorithms and abstract theoretical problems. These are not Google-style computer science interview problems. They can’t program something as simple as FizzBuzz.

Enlightenment through Real Data

We recently started investigating using an online coding test system from a company. It uses automated tests which run automated assessment tests verifying correctness, completeness, scalability and code complexity. This company is able to capture metrics on results and pass rates from a much wider audience of developers across different companies. And this has given us some enlightening information.

Here are just two tests with metrics on the number of developers who took it and pass rates for the test.

 

Test #1 – Count Div

“You have a,b and k integer numbers, count all integers which are between a and b and they are commensurable with k.”

This seems relatively straightforward and here’s a working solution in Java:

It was taken by over 5,000 developers. Over half flunked it. 8% of developers get to a complete working solution. That’s 8 out every hundred.

 

Test #2 – Find the Equilibrium Index

“Find an index in an array such that its prefix sum equals its suffix sum.”

It’s a relatively well known problem which basically says in a sequence A:

A0 = − 7

A1 = 1

A2 = 5

A3 = 2

A4 = − 4

A5 = 3

A6 = 0

3 is an equilibrium index, because:

A0 + A1 + A2 = A4 + A5 + A6

Here’s a solution in JavaScript:

The test was taken by over a quarter of a million of developers. Almost 30% completely flunked it. And only one in ten got to a completely working solution.

What is going on?

There are dozens of reasons why we have this problem, other than the very obvious that a large number of people somehow become programmers who probably shouldn’t.

As Jeff Atwood, the author of the original link said, “this is the equivalent of attempting to hire a truck driver and finding out that 90 percent of the job applicants can’t find the gas pedal or the gear shift.”

There is an unfortunate perspective in this industry that asking a developer to take a test makes them uncomfortable, puts them under pressure and questions their abilities as a engineer. But in reality it’s just checking you can literally do the job.

Like most technology companies we need developers who can program and write code that works. We have a great team of engineers who pair program, code review each others work and enjoy bouncing ideas and problems off each other during the day. The people we bring on board need to be able to do the same and it’s why we test that programmers can program.


We'd love to hear your opinion on this post

10 Responses to “Can Programmers program?”

  1. Isn’t ‘Test #1’ incorrect? Commensurable is defined as “two non-zero real numbers a and b are said to be commensurable if a/b is a rational number.” Rational numbers include fractions, so there could be a remainder if you ‘i % k’. In fact if a, b and k are all non-zero integers then all numbers between a and b are commensurable with k.

    Agree(1)Disagree(0)Comment
    • Hi Sean – seems correct to me. The question asked is “You have a,b and k integer numbers, count all integers which are between a and b and they are commensurable with k.”

      Agree(0)Disagree(0)Comment
  2. As a former computer science teacher who regularly taught the introductory programming course, I can corroborate the 30-60% failure rate.

    However, I find it potentially very disturbing to think that people who have completed an undergraduate computer science degree, or otherwise qualify as “developers”, would exhibit such high failure rates in the two problems shown above.

    I find the claims difficult to assess without knowing:

    (a) What is the company name, and what is its online coding test system?

    (b) How were the subjects – 5,000 (count_div) and quarter million (equlibrium index) “developers” – recruited, and how were their credentials validated? I can imagine an open online test that enables anyone to participate, or only asks for self-reporting of credentials.

    (c) What were the parameters of the test(s), e.g., was there a time constraint, was there any ability for a test taker to seek clarifications regarding the problem statements?

    I would welcome any clarifications you can offer.

    Agree(0)Disagree(0)Comment
    • Hi Joe
      Agree, it’s very disturbing and why I wrote the blog post. The company is one of the top companies out there providing a recruiting platform for hiring/testing programmers. The statistics in the second test I shared are based on a test that anyone can take without signing up or even being part of an interview process with a company, so you would expect lower pass rates. That said, only developers are going to take this test so it is probably still giving a fair assessment on where things are at. However the first test is even simpler, given to developers who are part of a recruiting process, and even more people flunked that one. The time limits vary by test and the ones I showed had 60 minute limits. In the testing we do we give 60 minutes and even stop by a few times during the test to answer any questions a candidate may have. Hope that helps.
      – James

      Agree(0)Disagree(0)Comment
  3. I don’t know where in the world you are pulling your applicants from, but if they are getting tripped up on those ridiculously easy and trite examples, then you (or your consulting firm) have a serious problem. In my experience more than half of interviewees are just average, with maybe 3 out of 10 being incompetent. Maybe 1 in 100 are all out rock star programmers. As a self proclaimed rock star programmer I can tell you there is nothing we find more demeaning than low level questions like “write this for loop.” Here I am with a resume that includes tech lead building distributed web apps that deploy to millions of users and this guy is asking me to write a for loop…

    Agree(0)Disagree(0)Comment
    • Hi jj
      Thanks for responding.
      These seem to be well talked about statistics and we’ve seen correlations from a wider number of interviewees. I’m not sure how I could identify a rock-star programmer in an interview without asking them to write some code. And as much as I’d love to ask someone to code up a distributed system web app as part of the interview process I’d doubt we would have time for anything other than a detailed white-boarding discussion.
      – James

      Agree(0)Disagree(0)Comment
  4. I think it’s a little simplistic to say that these people can’t program, when you’re asking people to do the kinds of problems that they almost never encounter in the real world and so don’t practice thinking that way on a regular basis.

    And I would guess one reason that so many people failed the second one is perhaps they don’t understand “commensurable”? I mean, who uses that word, like, ever?? I had to look it up and I’m fairly literate.

    And that last one: who ever needs to do that? Why? It’s so completely pointless. Can’t we come up with a test that does something useful? At least then, people can place the problem in the context of something they understand. It’s like asking a marathon runner to jump up and down on a pogo stick as a measure of athleticism. Of course the marathon runner can’t do it, they never practice pogo stick jumping. Doesn’t mean they aren’t a good athlete.

    Agree(0)Disagree(0)Comment
    • Hi Elizabeth
      Thanks for your response and feedback. I came from the same camp as you do a few years ago and thought perhaps some of this was pointless. Now after years of interviewing and seeing these statistics I think they are valuable and important. The two tests I pointed out test programming basics like using control loop syntax, using arrays and other fundamentals of any language. Most developers use these every day and they make up the basics of most programs regardless of the language you use. Agree that “commensurable” isn’t the most obvious but most interviewers explain what this means as part of the interview process.
      – James

      Agree(0)Disagree(0)Comment
  5. I have a B.S. in Mathematics and Computer Science and have been programming for many years. I have written many successful programs, but might well fail these two tests because I am unfamiliar with the terms prefix sum, suffix sum, and commensurable.

    I probably would have struggled, read and re-read the questions. I might have guessed correctly what those terms mean, and then solved the problems. But if I failed it would be because your test uses terminology I didn’t know, not because I don’t know how to program.

    Agree(0)Disagree(0)Comment
    • Hi Bob – that is why we sit with our candidates in person and walk them through the test and help answer any questions they might have about the problem. We also check-in with the candidate during the test to answer any questions. I’m not sure if all companies do this but IMO it helps address the situation where someone might not be familiar with some of the terms or the problem domain.
      Thanks for the comment!
      – James

      Agree(0)Disagree(0)Comment