In 2024 I worked on a project where we called into an API for an LLM AI Service. We gave it a set of data and a list of rules and asked it to provide a JSON-formatted response. We got a JSON-formatted response about three quarters of the time. The other quarter of the time we got a response that looked like: “Ok, here’s your JSON formatted response: {…}”.
After that project (but still in 2024) I installed a VSCode extension that added AI-powered autocomplete. It would add a light-colored preview of what it thought ought to come next and you could hit tab to accept it. This was pretty useful when implementing the editor extensions we were working on, which used a lot of configuration, much of which was very repetitive. On the other hand, it regularly suggested non-functional continuations. One that sticks in my mind is suggesting that .filter() was appropriate. Unfortunately, the data structure was a Set, not an Array, and Sets don’t support .filter.
In 2025 I got access to Cursor, and it was good. The combination of a chat window and access to the project files meant that I could just say “This function needs to be async. Please adjust all callers and grandcallers to be async,” and it would successfully do that. (And, yes, I did use the word “grandcallers”, and it understood what I was asking for.) It had pretty decent ability to reason about the code base. On that particular async refactor, I actually argued with it where it suggested that a synchronous return of cached data would be necessary to make sure all callers were correctly handled, but I knew that all the callers were inside async functions at some level, and we just had to validate that higher up. It had the background knowledge to recognize the risk, but lacked the analytical capacity to determine that it was illusory.
In 2026, I got access to Claude Code, and it was very good. Its analytical capacity is pretty high, and the ability to make tool integration “skills” is extremely useful. My work sessions with Claude look like saying “Read Jira ticket XYZ-1212 and make a plan to implement it”. And then it asks something like “How should we handle the case where the network fetch fails?” To which I reply “Make a toast show to the user”, then it goes and implements the whole thing, and writes the commit messages and the PR too. At this point my input is reduced to having good technical taste and critiquing the approach! (This is still necessary though.)
Cursor and Claude Code don’t make mistakes like trying to call filter on a Set. I don’t know how the fix was made, but it was. My understanding is that the APIs are much better at generating structured output too. The kinds of errors that modern coding assistants make look more like “correcting” tests to match the new implementation, and not like failing to generate the closing brace on page-long JSON. Or like making 10,000 line components that contain the entire page, rather than breaking out reusable parts. They’re reaching the level of a moderately capable programmer, to whom you can hand off the grunt work.
(I have deliberately failed to name the early stage products that disappointed me. Their current versions might be much better! Coding assistant capabilities are growing fast.)