A LLM got 12th in a Leetcode coding contest (probably)
In Weekly Contest 344, a fresh account (gogogoval) solved all 4 problems in 12:13 with commented submissions and test cases that look a lot like LLM code.
This could've been faked, but I doubt it. There are a small group of people in the world (a thousand, maybe) who could solve this contest fast enough to get 12th and add comments (or solve the problems and then feed it to a LLM to add comments), but it would be a very odd thing to do on a fresh account. The code itself also looks like LLM code - it uses techniques like gets on dictionaries (instead of using defaultdicts like most competitive python programmers).
This isn't that impressive. Leetcode contests are pretty easy, and this was an unusually easy contest - all four of the problems are rated easy or medium. Leetcode contests at the top level are about speed, not depth of understanding, so it's not that shocking that a LLM could do very well.
It's still a pretty massive leap. The free version of ChatGPT can solve two of these questions with some prodding, but it's pretty helpless on the other two. In particular, the last problem (the paths through a binary tree) is not trivial - it took me about ten minutes to find and implement a solution last night, and I'm pretty good.
This isn't that big of a deal for competitive coding, but it is a big deal for both Leetcode and the online assessments companies give. I don't know how much prompting or bugfixing it took the human who submitted this code to get the LLM to a correct solution, but either way, it changes the game on how we think about easy problems. We've gone from models that were very dull to a model that could probably pass the first round of a FAANG interview for a new grad. Already, in the week after, a skilled coder used a LLM to solve the first two questions of the Leetcode contest while solving 3 and 4 manually, saving a bunch of time and getting rank 5 in the contest. Easy questions aren't useful discriminators anymore.
This is the first thing I've seen a LLM do that genuinely impressed me. We are steadily approaching a point where either the improvement will slow or these things will have massive economic impact, and I have no idea which will happen.