Why I'm Not Using LLM Coding Tools

Was recently asked how much we use AI coding assistants in our work, and when asked why the answer was zero I didn't have a great answer on the spot. So here are my thoughts.

My job is to solve business problems, not type on the keyboard

My job as a software engineer is to creatively solve business problems and deliver usable, reliable tech solutions to these problems.

Yes, eventually that involves writing code. By the time I'm writing code, most of the thought has been done. If I'm going to describe the solution to the problem in enough detail for an LLM to implement, I may as well just implement the solution. I know what I am doing. Typing speed has never been the bottleneck to getting things done.

With that said - we're also pretty good at quickly building and delivering to customers. Most of that success is identifying what actually needs to be built to solve a problem. What identifiable problem is the LLM fixing?

If nobody wrote the code, then nobody understands the system

Code Reviews, where someone reviews the code of another developer before the change is merged, are ok at finding problems before they ship. But reading code is hard. Reviewing human code only works well when there is an assumption that the developers are all acting in good faith, with a shared understanding of the software product and expectations.

None of this applies to code written by an LLM. Reviewers need to be way more careful, analyzing each line of code in a way that you don't need to with a human, in addition to the overall structure. LLMs lie about what they have actually implemented, or don't follow conventions, or write fake tests that produce the correct output but don't test anything. They introduce subtle bugs that are difficult to pick up until runtime. The time "saved" in writing the code is swallowed by the code review and ongoing maintenance.

As you use more LLM code, less and less of the system is understood by anyone. We've benefited from having a development team that's worked on the software for years. Not only is there an understanding of how things work today, but also why they are this way, and where in the code to start when looking for the cause of a problem.

When shit hits the fan, you need people who can understand what is happening, so that problems can be fixed. By delegating your architecture and implementation to an LLM, you are removing experts from the development team. When something goes wrong, nobody will know where to start on fixing it.

The difficult part is changing existing code without breaking things

People love showing how quickly an LLM can knock a proof of concept together. We mostly aren't doing that. We have tens of thousands of users, and they expect the software to keep working.

We are working with a codebase that has been evolving for nearly 10 years. Most of our planning of changes has to do with how existing data is treated, how the new feature can be carved into the existing codebase without breaking anything. Automated testing here helps verify both new and legacy behaviour.

This is not something LLM tools are good at.

Who is responsible for LLM generated code if there is a data breach? Is it me?

We handle Personally Identifiable Information in our software, and therefore take security seriously. We have expectations on the kinds of automated tests need to be written, and an underlying understanding of the security posture of the system.

An LLM does not, and doesn't care. LLM generated code can look good enough on the surface slip through to production, but fail in strange ways, or have unexpected security flaws.

Who is responsible when something goes wrong? Is it me? What about liability with the GDPR or Privacy Act?

Yes, humans can make mistakes too. But the difference is the humans on our team are experienced, and are trying to do the right thing. There is an element of trust that cannot be extended to an LLM.

The speed claims do not hold up to scrutiny

AI coding tools do not make you faster. That seems to be the result of studies that have actually tried to measure performance differences. Developers aren't faster. Not only that, but AI coding tools are bug generators, leading to a higher ongoing maintenance burden.

People are really bad at assessing their own performance with AI. They think they are more productive, but studies keep showing that they are not. Time and again, businesses and governments are just not seeing productivity improvements when adopting AI.

Where’s the Shovelware? Why AI Coding Claims Don’t Add Up

78% of developers claim AI makes them more productive. 14% say it’s a 10x improvement. So where’s the flood of new software? Turns out those productivity claims are bullshit.

Mike JudgeMike Judge

AI makes you stupid

The use of AI by novice programmers reduces their ability to acquire programming skills. LLM coding tools still require a "human in the loop", and the generated code needs to be carefully reviewed. But who is doing the review, and do they have the expertise and experience to spot problems? A novice programmer writing code quickly becomes an intermediate and then senior programmer, as they gain experience. A novice programmer coding with AI stays a novice programmer.

And experienced professionals end up deskilling - getting worse at their jobs. It's happening in other industries as well. Endoscopists became worse at identifying polyps after exposure to AI.

I have no interest in delegating my thinking to a large language model, deskilling myself in the process.

We're already using code generation

One of the best arguments for using LLMs is for generating boilerplate code. Software engineers haaate writing boilerplate! It is truly a waste of time. But we make extensive use of code generation tools already to get rid of the boilerplate; LLMs aren't needed.

Our API is documented (using in-code tags) to OpenAPI 3 Specification. We then use OpenAPI Generator to build our API/network code for our mobile and web apps. This is close to bulletproof, and ridiculously fast. We update the SDK automatically on every PR merge.

We use other tools in the mobile app to generate code for us. Json Serializable automatically generates serialization code so we don't need to. Mockito generates mock services for our automated tests. We've invested in test tooling to make writing tests quicker. I could go on.

We're already using code generation to remove the busy work. Using deterministic tools written specifically for the task has the benefit of being far faster and more reliable.

The tools themselves are security problems

The AI programming tools themselves are a target of hackers. It is possible to perform prompt injection attacks to steal credentials from developer machines, Git repositories, Continuous Integration pipelines, and production systems.

They are also unreliable - they delete your hard drive.

The problem solving is the fun part

At least for me, the hard problem solving is the fun part. I don't understand why people are so keen to ditch the fun part of software development, and keep a worse version of the crap bits (overly tedious spec, code review, worse maintenance) and have no actual benefit to show.

Yes, the ethics are important

Large Language Models were trained on the stolen work of humanity. It should never been allowed to get this far in the first place. I do not want a part in promoting the same system that is degrading the arts, music, literature, entertainment, and yes, software engineering, with slop at the expense of people.

And what about the world we live in? The expansion of AI data centers and the huge amount of energy and water to run them is exacerbating the climate crisis. As we cruise past record heat waves, bushfires, sea level rises, and species extinctions, I worry about the planet my children will inherit.

We could switch off the AI data centers tomorrow, and nothing of value would be lost. The world in fact would be measurably better.