We are used to compilers/linters/formatters of our beloved programming language. In the era of neural net (AI) we want to get the best of the world and make reviews more machinery and more "smart". How do we achieve that?
Machinery Review
Compilers/linters/formatters are cool tools. For decades we have been using them alongside us; our pipelines were built on top of the input of linters, tests and compilers. If all pipelines were green, we trusted them. The static and deterministic analysis of such tools was easy to understand.
Green tests mean the logic is OK: we didn’t violate our product logic regarding WHAT our code does (we may argue about this point, but most of the time that is the most valuable factor to us). Green linters mean we didn’t violate our company rules regarding HOW we write code.
But what if I say that real world companies live with red CI?! That’s basically what happens somewhere, sometimes. They use one tool: the code compiled, which means we can actually ship it somewhere, it is READY.
All of the above is the machinery part of review, based on tools developed by other people. We can have flaky tests, arguable rules for our code, some inline TODO: comments and disabled verifications. But those tools are quite simple, I would say. We configured them, we know how they work, so we are quite familiar with them. A deterministic approach backed by known algorithms.
But the question is: can we actually say that running these tools against the HEAD commit of a developer’s branch is code review? Of course not. This is just some kind of verification that the developer’s code is actually worth reviewing by a human.
I had an enormous count of PRs where I was assigned, but I was giving them back because the code couldn’t even compile on the remote machine, e.g. gitlab-runner. Reviewing such code would be just a waste of my time.
So let’s agree: static analysis and a green build are not code review. Let’s proceed with this idea.
What Is Code Review?
Then what is code review? Until nowadays, the epoch of AI, it was people’s duty to review others’ code, based on some agreement between author and reviewer about how “good/clean” code should look. This definition varies from company to company and even between people inside the same company. I would say this agreement mostly exists within a team, or at maximum within a company’s department.
There are some definitions of “clean/good” code, but basically there are more contradictions than common ground in such terms. So clean-good / bad is a non-deterministic approach. Do you agree on that?
A reviewer today can be very happy and approve nearly anything. Tomorrow he’s upset and does a harsh, deep-dive review. Did the reviewer change his mind, go mad, or? Nah. That’s just a real human being with his emotions and other stuff going on in his head. That’s normal behavior. But it’s far from deterministic behavior, isn’t it?
Big PRs
Another problem you may think of: big PRs. Let’s say a PR > 500 lines long, or even > 1000 lines long. Isn’t this scintillating?
Do you know that according to statistics1 reviewers find far fewer defects in big PRs than in small ones? There’s a meme where the reviewer deep dives into a 50-line PR, asks clarifying questions, and then goes hard on the algorithm chosen.
LGTM!
But when the reviewer sees a 1000-line PR, he tends to just press the “APPROVE” button. Ah, first of all he would try to sabotage or dodge such a PR. Cause what sane being would like to take responsibility for a codebase that is going to be merged upstream? In case other dodging techniques didn’t work, he will just drop 1-2 comments and then press APPROVE. That’s a true story. Because he understands he has no time to deep dive into 1000 lines of code.
That’s, I believe, the reason why people fight neuroslop. It is extremely cheap to write, but very hard to review. Imagine reviewing machine-written code 2k lines long. That’s insane! I wouldn’t have read that, and neither would you (at least we wouldn’t be happy at all spending time on this tedious activity).
PRs As Communication
Another review problem that is hard for people: we used to teach other developers and communicate with each other by reviewing each other’s code. You remember that time?
All the juniors and trainees I have worked with have gone through a harsh code review process. And yes, sometimes I was too harsh and deep diving. I wouldn’t review another senior’s code like that, for sure. But with a junior you are driven to teach them and give some knowledge, and that drives you and makes you more attentive. And vigilant. Cause in no time your junior will tend to write some SELECT * FROM TABLE which is 100m rows long, or even a DELETE without a where clause. So it is your responsibility as the more senior guy to spot such things and help them navigate through this code/engineering world.
But let’s get back to the review problem. We try to teach each other, or communicate, AFTER decisions were made. So it’s a true story when some senior guy decided to implement the feature in SUCH a way, and when you spot that his architecture or decision is incorrect, you are asking him to rm -rf his files. And REIMPLEMENT that. Sounds insulting, no?
Barely. You are asking him to re-implement his work because you don’t like his solution, or you prove his solution is slow/dangerous/unreadable (select the suitable). So you have arguments in your pocket, which are very rational.
But imagine yourself in such a situation. You would use any possible way to dodge this code review and persuade your colleague that your solution is cool and production-ready or you will fix his comments in follow-up PR. Or at least communicate with him that you can do 1-2-3 small fixes to your original code, but for sure you are not going to reimplement.
So the idea of this paragraph is to show that we used PRs as a sort of communication with each other. Among our developer species. This question was raised not by me but by Rachel Laycock2. But now PRs have become something else. We should use other tools to communicate with each other, not just send PRs and say “hey, @developer, here’s my solution. Please drop a review, I’m waiting”.
Code review is hard.. For sure. And now it is even harder than it was before. We have to rethink and revolutionize our approach to the code review process.
PR Review As Safety Net
We consider code review a safety net that can avoid further rollbacks and incidents in production. The main goal is to find the most bugs/design/decisions made wrong during code review.
We want to share responsibility between the author and reviewer A, reviewer B, … reviewer Z. So later, if this code has caused an incident, we know who to call and ask politely to wake up and mitigate. Or beloved devops will roll back and they will deal with the issues in the morning (or on the weekend if they are unlucky).
So the idea is to have one more layer of certainty that your release cycle will go well and no incident is fired. We achieve this with different layers, starting with:
- linters
- formatters
- new tests written
- old tests passing
- green CI
- at least 2-3 approvers 👍
- acceptance testing passed
- and then the release flow, ready to be finally deployed to production
We want more certainty and confidence from these layers; we for sure don’t want to reduce them. Otherwise the time you save in PR review will blow up during a night incident. You basically have no gain from that, you just moved the problem up to the stack head.
Each step requires human effort, except for the CI build (it is handled by the machine). You can automate anything, reducing human effort, but as a result you are just postponing it to the after-release phase. Who needs broken releases where customers complain?
It is easy to convert less human effort into less revenue for your product. But that isn’t the goal. You definitely do not want to end up in the situation I demonstrate in fig. 1 (I would say this is the real situation for any vibe-coded product, but most of them never get to scale, that’s why it is never seen).
Do We Trust Pure AI?
Do we trust pure AI? I don’t think so. Nevertheless, I believe everyone who claims that you can fully rely on AI review is either stupid or not an engineer. Or he’s trying to sell you something you don’t need. In any case you don’t want that, and in any case you will be losing in such a situation. We are not ready for a full AI review, at least now, dated today’s date. Not at all. (yet)
But AI can give you more layers, and that is for sure better than static analysis checks. They can actually benefit your product with more stability and stable release cycles. Those thoughts are exactly what Anthropic mentions in their article3.
Where And How Can We Trust AI?
You can’t trust AI 100%. But consider the layers I have described in fig.1: removing or reducing something there affects the final release and incident rate. Adding new layers won’t. But for sure you have to consider a conscious number of layers. If there are 20 linters and 20 formatters running on a single PR, does it give positive value? Rhetorical question.
But adding wise layers improves it. Why not run an AI agent to review your changes before a human even touches them? So the human has to review less? Yes, of course you can object and tell me that AI review sucks. Give it the same entry two times and you will get different results. And I won’t pretend to be the devil’s advocate of AI review here. Yes, sometimes it can produce weird results. But you are the author, you can read what the AI wrote to you and pick what you as the author consider valuable. If it caught a concurrency issue or an unreadable variable name, would you object that it is a useful note?
After this polishing you can ask your team member for review and improve the code even more. Consider the following situation compared to the awful incidents illustration in fig. 1.
AI Triage
What if I tell you that the solution is easier than you think. Do you believe that an LLM can verify the grammar rules in the text you write? Why can’t you rely on that for your code? So the LLM will find smells and other stuff in your code that may bite, and we can allow the LLM to work on such issues itself.
But for complex solutions there’s triage: who will fix? LLM or human? Or LLM with a human copilot? You can actually route your fixes with an LLM (I wrote about routing with typed labels instead of prose in decision models4). Some don’t need an extra solution from you and the LLM will do just fine.
The other thing is to consider and understand where the critical paths of your application are. What is very important, what is core for your application? Where should every line be reviewed with extra effort? You are the one to define such boundaries. Also you have some business rules to cover, and an LLM won’t do that for you (unless you give it the context).
Conclusion
So basically, my high-level idea is that integrating AI into your company workflow requires effort. We should store the logic and rules valuable for the team/department/company in some factstore, where we can share our knowledge with AI. In the previous epoch (pre-AI) we did this by writing and reading guidelines, sharing them across team members and properly setting up formatters and static analyzers. Now it is time to move forward and think of the next step for such tools.
What about a factstore that can be shared with AI agents for review, building, testing and so on? I think it should look like the following: the guide clearly states which critical pieces should be looked at by a human, and whatever violates the rules goes back to the author for rework. On top of that, another LLM pass looks at the solution from a different angle (security, performance, data), so that it’s a solid piece of the review world.
So that it looks as much as possible like an expert review from the domain area, as if another person reviewed it, knowing their shit, getting it done, not guessing. And not just based on general data by neural net provider trained on it.
Ideally, these guides and AI tools have expert domain knowledge and the solution behind it. Like: don’t touch this shitty-code algorithm, it works, but it’s written really ugly, and nothing should ever break it or refactor it
Yeah, we’ll dig into it somehow, start with something simple, figure it out along the way.
Bonus
For everyone who did read up to this line, a small present: https://www.youtube.com/watch?app=desktop&v=3TNpOD6bov8