Teaching AI to Work in a Codebase Older Than Its Context Window
What I've learned about context, instructions, and guardrails while using AI-assisted development in a large legacy system
Last week my almost 2-year-old son burst into my office and saw something absolutely abhorrent. My fingers gently hovered over the F10 and F11 keys, my terminal was filled with JSON logs, and right where an AI chat window belonged there was instead an open file with lines and lines of code. Dumbfounded, he stared at me, his eyes seeming to say “Daddy, why don’t you just utilize the open-source `/debug-skill` skill with Claude Code on Fable 5? It will utilize a systematic and hypothesis-driven debugging flow to diagnose the issue with 75% greater precision than most senior developers.” He then squatted down, pooped in his diaper, and ran out.
A simple excuse might be that I knew I could debug this issue and therefore could save on token costs or that I wanted to keep my skills sharp. The more practical reason is that this area of the codebase contained several semantic layers that made sense to our team but would require substantial investigation from an AI agent, with no guarantee that it would trace far enough up the call stack to identify the correct fix. Setting up the appropriate guardrails is the challenge faced by those of us tasked with maintaining legacy systems while still hoping to utilize the productivity enabled through agentic workflows. While I’m not professing to be an expert in this field, I have learned a few practical principles along the way that I’d like to share in this article that may help others facing similar challenges.
Human-Driven Codebase Discovery
It’s essential that at least one person on the team has a deep understanding of the codebase. It can be just as effective to divide ownership across team members who each have deep expertise in a particular area, so long as the boundaries between those areas are also understood. This human aspect is required because although the human brain may not provide the same level of precision about a wide variety of topics like the AI brain, our own context window about a system we’ve worked on for years will be much larger. This means we can retain nuances, the trajectory of a call stack, internal naming conventions, and years of implicit architectural decisions that an AI agent has to rediscover. The key then with that expertise is translating it into instructions for the agents.
Depending on your technology stack, system design, and product, your AI instructions can vary. When I was first implementing AI instructions for our legacy system we didn’t even have a `README.md` (technically we did, it just hadn’t been updated since the first commit). I had a pretty high level of expertise across the full codebase, so I took it upon myself to write a full `README.md`, trying to provide the perfect level of specificity. I then asked my GitHub Copilot agent to parse my Markdown and then generate a single `instructions.md`. I thought I was a genius and that this would give all of my agents the perfect amount of context on any task going forward.
I very quickly learned that this was not the case. Not only were my agents eating up tokens faster than I could update my overage budget, but they were also gleaning all the wrong lessons from the context I had provided. As an example, we had been working on a major refactor of one of our core modules in a separate part of the codebase. I had explicitly outlined the updated coding standards utilized in those refactored areas, while also mentioning specific helper functions that would not work in the existing modules. It took that to mean that we shouldn’t use the new coding standards (basic things like docblocks) in the legacy parts of the codebase at all. It was also obsessed with repeatedly mentioning the parts of the codebase I had told it not to touch. I swear, in almost every other sentence the output would be like “but don’t worry, I didn’t alter the `Template::display()` function.” Overall, I knew that I needed to do something to make sure I was extracting more value and the guardrails needed to move.
AI-Driven Codebase Discovery
Phase two required a much more hand-holding approach to generating appropriate agent instructions. The first step was creating a proper map of the codebase. I partitioned out my instructions into related scopes, focusing primarily on typical backlog items we would handle. Then, in my prompt I would outline a typical approach I would use to handle one of these tasks (as a side note, I found it very useful to create my prompts in a Markdown file, this would give me a constant record of the instructions I was using to generate instructions, and I could utilize the AI to alter that file in case instructions changed or didn’t make sense). Outlining that process gave me a clear picture of which parts of the codebase would be involved. The agent could then focus on parsing the more specific parts of the codebase it would need to generate the instructions. I also learned during this second pass how much more useful it was to let “Plan” mode handle the discovery and then bring in an agent to write the instructions file. Overall, the result was a much cleaner, well-partitioned list of instructions that could be used in specific contexts to help us tackle our work hand-in-hand.
A Partner, Not a Direct Report
The final key was recognizing that the AI should be an extension of myself, not simply a tool that I handed work off to. Mind you, I still let it run autonomously, I’m not sitting there hovering over its every output, but every task starts with a back-and-forth to make sure I’ve provided the right context and tools needed for the AI to complete the task the way I would. I start each feature by writing a high-level outline of the product along with steps I would recommend for implementation. I’ll then pass this on to the plan agent along with instructions to report back to me on what questions still need answering and what details need to be filled in. This back-and-forth may last for a few exchanges until we’ve generated a proper spec file with all the edge cases and tests needed to ensure completion. With all of that fleshed out, I’ll turn it loose. Depending on the feature I may stay involved for a little bit to watch it work and utilize the steering tool as needed, but if our feature discovery made me feel confident that it understood the assignment I’m going to extract the value I can out of the tool while I work on other things.
After it finishes, I'll run the necessary manual tests and verify that the system's core functionality still works. One of the instructions I include for all prompts is that the agent heavily document its code and focus the comments on the “why” rather than the “what”. This makes code reviews very simple. Rather than poring over lines and lines of syntax, I can simply review the comments and follow that logic. I’ll make a list of questions as I go through and then pass those back to the AI in “ask” mode, but if it can defend its decisions then I’m not going to lose sleep over it.
Guardrails
I realize I didn’t touch on any specific guardrails during this article, and that’s been deliberate. Different tools provide different documentation on how to implement things like glob-scoped instructions and the different semantics on “SHOULD”, “SHOULD NEVER”, etc. Without diving too far into tool-specific details, an appropriate human pass, an AI-assisted discovery pass, and treating the agent as a partner rather than an underling should help you decide which parts of the codebase you can trust it to work in and which require more supervision. The one specific prompt I would recommend is `Make the smallest complete change that satisfies the requirements`. Including this detail has been a life-saver and keeps the agents from seeing an inch and taking a mile. In a legacy codebase where you may have so many interconnected parts, that line seems to consistently rein things in when it may be tempted to start touching files it doesn’t need.
Not Your Grandfather’s Codebase
One thing I think many engineering managers need to accept in the era of AI-assisted development is that your codebase is going to change. This can be tough to accept if you consider the product to be your baby, but that is going to be the case whether you’re allowing AI tools or hiring a new employee. No codebase stays frozen forever. A system that survives for a decade will inevitably accumulate assumptions, architectural decisions, and patterns that need to be reconsidered over time. I’ve found that focusing your energy on crafting AI tools into a system that works for you and allows you to ship great products is much more effective than using that same energy to fight it. I understand if you want to limit use of these tools (I do both for myself and for my team), but there is absolutely a way for AI-assisted development and an established codebase to coexist productively.
