Free for open source projects and schools. AI Council costs nothing for OSS maintainers and educational institutions. Apply →

Blog / AI development in practice

AI coding agents start every day as new hires. We spent 18 months building them a project memory: here's what worked and what didn't

Dmitry Lunevsky · · 32 min read

In early 2025 we asked Claude to add an API endpoint to one of our projects. It wrote the code quickly and neatly, documentation included. Code review found three problems:

  1. The business logic sat right in the controller. We don't do that here.
  2. It wrote a new authorization function, even though the project already had one.
  3. It named a function getUser instead of get_user.

The next day the same thing happened on a different task. You can fix this with a detailed brief: spell out in the prompt where logic lives, which authorization already exists and how functions are named. But you'd have to spell it all out before every task, the same things over and over.

A new developer spends their first weeks reading more than writing. They work through the code, ask colleagues questions and gradually learn why things are built the way they are. For an AI agent that period never ends. It doesn't remember yesterday's conversation, so every day is its first day on the job. It reads files, greps the project, spends time and tokens on it, and can still miss what it needs, because a decision made six months ago isn't in the code.

We're Kub3, a 35-person software company that has done custom development for more than 15 years. We implement and customize SuiteCRM and Odoo for clients and maintain projects with codebases that are many years old. A typical project has two to five developers plus analysts, testers and a project manager, so knowledge is spread across people, and the agent only learns what someone tells it. We've been working on this problem for 18 months. First we tried other approaches (more on them below), then we built AI Council, a platform that gives AI agents a memory of the project: a code graph, the team's decisions and its rules. We now use it on our own products and on client projects.

The main thing we learned: what the agent lacks isn't intelligence but whatever isn't in the code. It doesn't know what the team decided and why, what its rules are, or which of that is still true. And that knowledge has to reach the agent without being asked for, because the agent rarely goes looking for rules on its own.

Below we cover what the agent is missing, which tools already exist for this, what we built and what it costs. Here's the main result up front: in a head-to-head comparison, the agent with AI Council wrote code that was easier to accept in review, but the model's work cost almost a third more.

It's not just us

First we checked whether this was only our problem. It wasn't. Here are six measured studies published by fall 2026. Each has its caveats, which we put in the last column.

What was measuredResultSource and caveat
Main complaint about AI tools66% of developers say AI answers are "almost right, but not quite", and 45% say debugging that code takes longerStack Overflow Developer Survey 2025, 49k responses, self-reported
Perception vs. measurementExperienced developers believed AI made them about 20% faster; measured, they were 19% slowerMETR, 2025, controlled experiment, 16 people on repositories they knew
Code vs. tasksAfter AI agents were adopted at more than 700 companies, code volume rose 30%, completed tasks stayed statistically flat, and time from PR open to merge grew 49%Harvard, October 2026, working paper, not peer reviewed
Waiting for reviewAn agent's PR waits 17.6 hours for review to start, versus 3.4 hours for a regular oneLinearB Benchmarks 2026, 75th percentile, 8.1M PRs
Review comments per PR10.83 comments on AI-assisted PRs versus 6.45 on human-written onesCodeRabbit, December 2025, 470 PRs; data from an AI review vendor
Code reuseRefactoring fell from 25% to under 10% of changes; copy-pasted code rose from 8.3% to 12.3%GitClear 2025, 211M changed lines

The studies differ, but the picture is the same. AI feels faster than it measures. Code gets written quickly but fits the project poorly, and the cost shows up in review and rework, so the bottleneck has moved from writing code to checking it. Google's DORA 2025 report calls AI an amplifier, with a condition: agents need access to the company's internal documentation and code.

What the agent is missing: three kinds of knowledge, plus time

We went through the review comments that agent-written code received on our projects. Almost all of them came down to the agent not knowing something, and that something falls into three kinds.

Structure. Which modules exist, what calls what, where the function you need already lives. All of this can be derived from the code with a parser, a graph or search. The agent can dig it up itself with grep, but slowly and at a cost.

Decisions. Why payments go through a queue, why nobody touches this module, which option was already tried and dropped. You can't derive this from code: the code holds only the outcome of a decision. The reasons live in people's heads, in closed merge requests and in chats. If the team's processes are mature, they also live in ADRs, meeting notes and systems analysis documents. That doesn't help the agent much: those documents sit apart from the code, and the agent doesn't know it should read them before editing a file.

Team rules. Agreements everyone must follow: layers, naming, things you don't do. A linter checks some of them, but most are beyond it out of the box. For a rule like "all customizations go in directory X, leave everything else alone" you'd have to write the check yourself, and it would only fire after the code is written.

The endpoint story hit two of the three. The agent wrote a second authorization function because it didn't know the structure. The logic in the controller and getUser appeared because it didn't know the rules.

On top of all this sits time. Code changes every day, and what was true yesterday may be false today. Documentation that has fallen behind the code becomes harmful, because the agent believes it. And when five agents work in five branches, a decision made in one branch shouldn't affect the others until it's merged into main.

Diagram: what an AI coding agent is missing, structure, decisions and team rules, plus time

We use this diagram to compare the tools below: which of the three kinds of knowledge each covers and how it handles time. We also compare on something the diagram doesn't show: who brings the knowledge. Is it the agent, on its own request, or the system, unasked? That question turned out to matter more than we expected, and we'll show it in numbers later.

What already exists, and when it's enough

Plenty of people work on memory for agents, and some solutions are free. AI Council is our commercial product, so we'll say separately where you can do without it. The main approaches:

Rule files. CLAUDE.md, AGENTS.md, Cursor rules and Claude Code's auto-memory, which saves notes between sessions on its own. In effect this is the same detailed prompt, written once. You can start in five minutes, and the file lives in git and is versioned with the branch (except auto-memory, which is stored on the developer's machine). The downsides are clear too. The text isn't linked to the code and goes stale without anyone noticing, the agent reads it whole, the relevant mixed with the irrelevant, and the rules in it stay advice, because a file can't make the model follow them.

Symbol navigation. Serena gives the agent IDE-like tools on top of a language server: find a symbol and everything that references it, insert code after a function. It's more precise and cheaper than grep. But Serena knows only code: the team's decisions and rules aren't in it, apart from simple notes. And the agent has to use it on its own initiative. Serena's documentation describes optional hooks that remind the agent about its tools when it slides back to grep, because left alone it forgets them. We'll come back to this.

Semantic search. Zilliz's claude-context indexes code into Milvus or Zilliz Cloud. Search is hybrid, BM25 plus vectors; code is chunked along the AST and the index updates incrementally. By summer 2026 the project had about 12k GitHub stars. It helps when you don't know what the thing you're looking for is called. The limitation is the same as Serena's: there's nothing in it but code.

Code graphs. CodeGraph and TeaRAGs both build an index of the codebase as a graph of symbols and relationships and serve it to the agent over MCP. With the graph the agent quickly finds out what calls what and what its change will affect. Like search, the graph says nothing about decisions or rules.

Memory servers. Mem0, skillmem and agent-memory-mcp give the agent long-term memory as notes and facts. skillmem, incidentally, also ended up using Claude Code hooks. Here you can already record a decision along with its reason. The weak spot is elsewhere: the note lives apart from the code, and when the code changes, the note stays the same.

Commercial context engines. In February 2026 Augment opened its Context Engine as an MCP server for any agent, with local and remote modes. By its own measurement, agent output quality improved by more than 70% with it. Unblocked pulls context from code, documentation, tickets and conversations, so it reaches the team's decisions too. Judging by their documentation, rules aren't a core feature in either product, and neither documents how it handles branches.

AI code review. CodeRabbit, Greptile and Qodo come at it from the other end. Since review has become the bottleneck, they take on part of it: they read the finished PR and leave comments. But by then the code is already written, and anything the agent did differently from how the project does things has to be redone.

ApproachCode structure"Why" decisionsTeam rulesWho initiatesBranches and staleness
CLAUDE.md and auto-memoryNoIf written by handAs advice in textLoaded whole at session startCLAUDE.md is versioned in git, auto-memory isn't; freshness isn't checked
SerenaYes, via LSPSimple notesNoAgent calls itWorking copy
claude-contextChunks and vectorsNoNoAgent calls itIncremental index
CodeGraph, TeaRAGsSymbol graphNoNoAgent calls itCode reindexing
Memory servers (Mem0, skillmem)NoAgent's notesAs notesOn request; skillmem has hooksNot tied to code
Augment, UnblockedSemantic indexUnblocked: from conversations and docsNot a core feature, per the docsAgent calls itLocal or remote index; branch handling not documented
AI code review (CodeRabbit, Greptile)YesPartlyIts own review rulesAfter the PR is openedPer PR
AI CouncilCode graph and static analysisRecords tied to codeMUST and SHOULD rules, delivered wholeThe system delivers on its own, plus MCPKnowledge lives on git branches, freshness checks

If one or two people work on a small project, CLAUDE.md plus Serena or claude-context will do. We've listed the other cases where you don't need AI Council at the end of the article.

Three attempts: two failures and one that worked

Attempt one: let the model describe the project itself. You press a button, the model reads the repository and writes an architecture overview. It's expensive and slow. On top of that, the model doesn't know what matters in the project, so it describes a date-formatting helper in as much detail as the billing engine. And within a week the description no longer matches the code. We dropped this approach.

Attempt two: RAG over the code. Split files into chunks, embed them, search for similar ones by query. Semantic search works, and for a small project it may be enough. But you lose structure: the function is cut off from its class, the class from its module, the module from the rest of the project. And the "why" decisions still aren't in the code.

Attempt three, the one that worked: graph, vectors and knowledge. Static analysis builds the code graph, with no model involved. Vectors handle search by meaning. A separate layer holds what can't be derived from code: decisions, reasons, constraints, rules. The layers are linked; for example, a decision record points to the functions and modules it applies to.

Diagram: AI Council's three layers, code graph, vectors and knowledge, with LLM-free retrieval

For a query, the agent gets a set of evidence: a few dozen fragments from the graph and vector search, merged into one list. By our measurements that's 9 to 18 thousand characters. In tokens it's a few thousand, the first few percent of a 200k context window. That's roughly one source file of three to five hundred lines.

Four decisions that set us apart

You can assemble three layers from open-source parts. We build our own product for the sake of four decisions. The first two are about how knowledge and rules reach the agent; the second two are about how knowledge survives over time.

We deliver two different things: knowledge records and rules.

A knowledge record explains how and why something works. It can be a decision with its reason, an analysis of a defect or ticket, a flow diagram. A record has anchors: names of classes, modules or files that should bring it up when the agent works with them. Until the agent touches that code, it doesn't need the record. Here's how a record looks in the agent's context when it didn't fit in full:

[Knowledge/decision] ADR-124: the gateway returns a server failure as 500 (not 502), the UI
retries 502/503 only for reads … — not shown here (4 KB);
read it with get_knowledge("knowledge/decision/dfda410b…")

A rule (the product calls it a convention) is a team agreement that everyone must follow. A rule has a strength, MUST or SHOULD, and a scope: either the whole project or specific files, classes and functions. The difference is simple. The agent reads a record and takes it into account; a rule it must follow. Here's one from our rulebook, in full, exactly as it reaches the agent:

[MUST] Each service must own its database exclusively. No service may directly access
another service data store (Neo4j, PostgreSQL, Qdrant, Redis). Cross-service data access
must go through gRPC APIs. This ensures independent deployability, schema evolution,
and failure isolation. (convention/database-per-service)

This rule is short. A typical rule runs about a page: what's required, why, how to apply it, and what it can't check by itself.

1. Knowledge arrives on its own, not only on request

This brings us back to the question of who brings the knowledge: the agent or the system. AI Council has MCP tools, such as code search or rules for a file, and the agent calls them when it decides it needs them. And it has hooks. These are commands that Claude Code runs at set moments, say before a file is read or at the end of a response. The model doesn't call them and can't skip them, and a hook can add text to the model's context or reject a tool call.

MCP alone isn't enough, because a tool has to be called, which means it can also not be called. We looked at how this plays out in our production data across 12 thousand sessions: the agent ran a rules check in 16% of them, and before its first edit in 13% of sessions with edits. In seven such cases out of eight, it started on the code without asking for the rules. Hooks were active the whole time; we address the objection "that's why it didn't ask" below, in the numbers section.

So we made hooks the main channel. Claude Code has had them since 2025; Codex added them in spring 2026 as an experimental feature. We weren't alone in this: Serena and skillmem went the same way. MCP stayed. When the agent needs something, it searches through MCP itself, and agents without hooks only get that path.

Through hooks, AI Council reaches the agent even when the agent didn't call it:

EventWhat happens
Start of work and every promptAt the start of work the agent receives the project-level rulebook, each rule in full. Before each following prompt the rules repeat in rotation; the developer caps the volume per prompt in their settings
The agent is about to read or edit a fileThe agent receives rules anchored to that file, its classes and functions, knowledge records about that code, and the file's known smells
The agent has finished respondingA completion checklist: checks recorded for this project and known issues in the changed files

An example from a client project. In a CRM, "unknown" contacts created from incoming emails had to be merged with known ones. The standard merge didn't fit: it would have overwritten the known contact's fields. A developer spent a day figuring out the right way: move the relationships to the known contact and delete the unknown one. He wrote this into the knowledge base and anchored the record to the contact model. The next time the agent opened that model's file, the record landed in its context on its own, and it did the task that way instead of reinventing the merge.

If a file has many records, they queue up and arrive as one block on the next tool call. Anything the agent has already received in this session isn't sent in full a second time.

When a record arrives depends on its anchors, so AI Council checks them. If an anchor matches more than 30 functions or classes, it's considered too broad and dropped. We fell into this ourselves: we anchored a record to graph_service, the name matched 357 entities, and AI Council dropped it, suggesting we name a module, class or file instead of a whole subsystem.

The file-open hook brings only what's anchored to that file. But sometimes the agent reasons about something that has a record in the base without ever opening the relevant file. For that case there's a hint based on the agent's own words. A separate hook counts the identifiers the agent mentions in its reasoning and, for those mentioned five times or more, asks the base whether there's a record about them. If there is and the agent hasn't seen it this session, one line lands in its context: the title and the command to read the record. The weakness is obvious: if the agent calls the same thing by different words, the hook stays silent.

The examples from here on use Keycloak, an open-source authorization server in Java with a large codebase and years of ticket history. We connected it to AI Council as a test bed, so we can show things that client contracts keep private on their projects. The word-based hint fired there, for example, like this: the agent mentioned the OIDCLoginProtocol class seven times and got a line about the record on ticket #51712, about the logout URL in OIDC.

Nobody wrote ticket records like that one by hand for Keycloak. This is usually where knowledge bases stumble: someone has to write them and then keep them in sync with the code, and in our experience nobody finds the time. Here AI Council worked through the GitHub ticket history itself. For each ticket it recorded what was requested and what was done, and anchored the analysis to the code the ticket touched. That produced almost the whole base, more than three thousand records.

It wasn't free. A model analyzed the tickets, so instead of people's time we paid for the model's work. These records don't need re-checking: each is marked as history, with the ticket number and date, and makes no claims about the current code. The freshness check in section 4 takes care of the rest.

What such a base gives you shows in an example you can verify against public tickets. We made up the question ourselves, the kind a developer newly assigned to this class might ask: "Why does GzipResourceEncodingProvider never clear its on-disk cache?"

Before reading any code, the agent already had a SHOULD rule about publishing derived copies, analyses of tickets #52802 and #53238, and a diagram of the request path through the theme cache layers. Some of it the agent requested from the base, some came via the hook when it opened the file. Then it checked against the code and separated the two tickets. The first, #52802, about the cache that never clears, was still open at the time of the check. The race condition from #53238 had been fixed with an atomic file move, and you can see that in the code.

Delivery also has a strict mode we call the KB-first gate: knowledge base first, then code. Until the session has queried the knowledge base at least once, it can't read or edit code, and reading through cat and sed in bash is rejected too. The gate opens after a query through list_knowledge, get_knowledge or brief_me_on. It can be turned off.

2. A rule arrives whole or not at all

You can't send the agent everything in the base at once. The model's context is limited, and the more noise in it, the less room is left for the task and the more each request costs. You have to choose what to send and in what form. With knowledge you can save space: a record that doesn't fit arrives as a title and a read command, as in the ADR-124 example. With rules that trick doesn't work, and it took us a while to understand why.

The first idea was to shorten: trim each rule's text to a couple of lines so the rulebook fits in the context. We dropped it. If instead of "all customizations go in directory X, leave everything else alone" the agent gets the paraphrase "don't touch legacy", it will read that its own way. A shortened rule is a different rule under the same name.

The second idea looked more harmless. Rules that didn't fit, we started sending as a list of names, like this (the format is real; the rules are examples):

  [MUST] Each service must own its database exclusively. … (convention/database-per-service)
    - [MUST] Fix every error you encounter

The first line is a rule with its text, the second is just a name. Same MUST tag, same look; only the indent differs. In August 2026 we asked four sessions to audit themselves, and all four had cited a rule they hadn't read. One read it and found it didn't cover its case; another found it was about something else entirely. Three more sessions spent several hours each re-deriving a principle that was written word for word in a rule that simply never reached them. And the names alone took up two thirds of the space set aside for rules in a single prompt.

Now a rule arrives only whole. The only place to save is selection: which rules to send right now. If some don't fit, the agent gets a single line instead, with no names in it:

!! 15 more project rules exist and were NOT delivered (7 of them MUST). Their names are
deliberately not listed: a name is not a rule, and acting on one is acting on nothing.
If anything outside the rules above could bear on what you are doing, read the rulebook
with list_conventions().

The agent receives the project-level rulebook before its very first action. In practice the first tool call is rejected, the text comes with the rejection, and the agent retries. Our rulebook is about 175 thousand characters. After that, rules come in rotation, a few at a time, and a full cycle stretches over dozens of prompts, so the rule you need right now may not arrive right away. That's why file-level rules take a separate path, through anchors: they arrive when the agent opens the file.

Claude Code session where the AI Council hook sends the agent the project rulebook before its first action

3. Knowledge lives on git branches

A record written on a branch can be wrong for main, and the other way round, because it describes code the other branch doesn't have yet. So each record lives where it was written, and the code graph for each branch gets its own layer on top of the base one.

The agent doesn't need to name the branch, either when writing or when reading. AI Council works it out from the working copy the request came from. It explains why to the agent right in the response to a save:

<branch_derived branch="stable" … source="git in your working tree — you were not asked,
  and cannot be: a branch stated from memory was wrong nine times in ten"/>

The agent sees records from its own line: its branch, the branch it was cut from, and everything already merged into them. Records from sibling branches are invisible to it, so five agents in five branches don't trip over each other's unfinished decisions. Sometimes the same record has already been marked stale in one branch but not yet in another (more on marks in the next section). Then the agent gets it as it looks in the branch closest to it. Closest by history, not by time: its own branch first, then the parent. Time only breaks ties when the distance is equal.

When a branch is merged, its records become visible in the target branch automatically, because visibility is computed from git history and there's no separate "move the knowledge" step. If a branch is deleted without being merged, its records aren't lost; they go to an archive.

Rules, on the other hand, aren't split by branch; they're shared across the whole project. A rule says what code should be like, not what it is right now.

Diagram: knowledge records living on git branches and becoming visible after merge

4. Knowledge is checked for freshness

We haven't found this in other knowledge bases. Each record remembers the commit it was written on. Once an hour AI Council queues a record for re-checking if the code under it changed (every function and class has an AST fingerprint for this), if the record itself changed, if a similar record appeared, or if a branch was merged.

Fingerprints are compared without a model, so the agent gets a "the code under this record has changed" warning immediately, together with the record:

[Knowledge/decision] ADR-201 (self-update.sh, …): hook self-update checks the version
at every session start … — not shown here (4 KB); read it with get_knowledge("…")
    ⚠ This is a decision; the code under it has moved since it was checked (314d7be0):
    1 symbol(s) changed on yours — it is not re-reviewed against code, read the code yourself.

The warning alone turned out not to be enough. We tested this on the test bed, admittedly on just one case and only where the agent answered without looking at the code. Given a stale record with a "code has changed, check it" warning, the agent still answered from the record in 10 runs out of 11, no better than with no warning at all. With a corrected record, 0 out of 10 runs did that. A mark stating what the code does now gave the same result, 0 out of 10. Fingerprint comparison can't write such a mark, so a model re-checks our records.

The knowledge reviewer does this. Despite the name, it's a background mechanism inside AI Council: there's no person behind it and no separate product. It takes a record from the queue, gives the model the record's text along with the current code it refers to, and turns the answer into a mark. The model works read-only: it changes nothing in the base or in the code. We accept a verdict only if it includes a verbatim quote from the source.

Even with a quote the model can be wrong, so the reviewer doesn't touch the record's text. It adds a mark ("stale", "diverges from code since commit X", "superseded", "disputed"), and from then on the agent gets the record with that mark. Here's how a "supersedes" mark looks on a record about a ticket route in our UI:

[Knowledge/decision] ADR-203, amendment 2026-10-20 (TicketsPage, …): a ticket is the path
/tickets/<external-id>, the old ?ticket= redirects …
    Supersedes ADR-203 (…) since 52d56538: Old entry lists the ticket deep-link as
    `?ticket=N` (UI useTicketPages); new entry replaces that with the path route
    /projects/<project-id>/tickets/<external-id>, redirecting old ?ticket= links, while
    explicitly leaving the rest of ADR-203 (e.g. the :Ticket scan) unchanged.

The mark names the commit from which the old record is superseded and lists exactly what changed in it and what is still true.

Marks are applied and take effect on their own; nobody has to confirm or reread them. They all go into a log, where any of them can be reverted. A project can turn on a second opinion: the verdict is then checked again, and if the two answers disagree, no mark is applied and the log gets a line with the reason. We test the reviewer model too: blind, on questions with known answers, and we show the result next to the model choice.

Agents maintain the base as well. On Keycloak, for example, an agent noticed that a ticket record diverged from the code and added a correction to it.

Diagram: how a knowledge record is re-checked when the code under it changes

Review and the model: two more decisions

Review against the team's rules. The agent may receive a rule and still break it, so a finished MR goes through machine review. It runs before human review and checks the diff against the development rules the agent received before editing and against rules written specifically for review. If a rule is broken, the MR gets the Blocked status, with the rule and the location. The status itself doesn't prevent anything: you can still merge such an MR, and teams that need a hard block set it up in CI.

For the status to be trustworthy, the model is constrained in two ways. It doesn't cite line numbers; it quotes the code instead, and AI Council finds the line from the quote. The model can get a line number wrong with no way to check it, while a quote can be matched against the diff. We demand the same of the knowledge reviewer from section 4: a verbatim quote. And for every file the model must say either "checked" or "couldn't, because". A file it says nothing about isn't treated as clean; otherwise a review in which the model read nothing would look like a review with no findings.

The screenshot shows an MR from our own repository: the diff truncates rule text to 180 characters, and review blocks it under the rule from section 2.

AI Council review blocking a merge request that breaks a team rule

The client brings the model. Context retrieval uses no LLM: AI Council invents nothing, it returns the fragments it found with their files, and the agent checks them against the code itself. A model is needed for the project chat, for review, for ticket analysis and for re-checking knowledge. In all these cases it's the client's model, an external CLI agent running under the client's account. Today that's the Claude Code CLI, but through a gateway you can connect other models, including open-source ones on your own servers. We've run Qwen and DeepSeek this way.

The numbers, including the inconvenient ones

Our project: where knowledge and rules come from

We develop AI Council with AI Council, so our most complete data comes from our own repository. It has more than five thousand files and about 50 thousand functions, with about three thousand knowledge records and more than a hundred rules on top of that code.

Record typeShareWhere it comes from
Decisions with reasonsa little over 40%Written when a decision is made: what was chosen, what was rejected and why
Notesa little over 40%Analyses of subsystems, defects and measurements, written as tasks go
Changelogsabout 10%What was done in a task and why
Diagramsabout 5%Flow and structure diagrams in Mermaid

Both people and agents write records. The agent does it at the end of a task, because one of our rules requires it: work on code has to leave a trace in the base. We haven't counted how many records come from each.

Almost every rule has an incident behind it. The rule about delivering rules whole, for example, grew out of the four-session story from section 2. So you don't need to start with a hundred rules. AI Council sets up general rules that suit any codebase in a new project by itself, and a team usually writes 5 to 10 of its own at the start: the ones broken most often. The rest get added as work goes on. An agent can propose a new rule too, but it only takes effect with a human's approval.

Everything AI Council brings to the agent takes up room in its context. Measured over 30 days on one machine, that's 46% of the context on our project and 20 to 30% on others. You can't count that share as pure overhead: the agent would have read part of it in the code anyway. What delivery costs in money is shown by the head-to-head comparison below.

What agents actually do

This is telemetry from our production over the whole observation period, a snapshot from October 20, 2026: 12,241 agent sessions in total. Here a session means a telemetry row, and one conversation sometimes produces two rows, so the shares are what to look at. Actions are counted independently: each row of the table is the share of sessions where the agent did this at least once.

What the agent did in a sessionWhat we countSessionsShare
Gathered project context itselfCalled code search, an architecture or dependency overview, a file summary, read a knowledge record or queried the base6,93857%
Edited codeThe hook recorded at least one file edit4,43636%
Ran a rules check itselfCalled the tool that returns rules for a file, or the search for similar implementations1,91916%
Ran a rules check before the first editSame, but before the first edit; counted against sessions with edits584 of 4,43613%
Wrote knowledge to the base itselfCalled the knowledge-write tool2,73122%
Wrapped upCalled the "think through the consequences of a change" or "summarize and save" tools230.2%

The agent edits code in about a third of sessions; in the rest it answers questions, investigates or checks something. It goes for context on its own in more than half of sessions, so it isn't too lazy to search, but it rarely asks for rules. For us that's the main argument for delivery by the system: what the agent doesn't look for, it won't get without it.

There's an obvious objection. The measurement was taken with hooks running, so maybe the agent didn't call the check because the rules had already come to it. Our telemetry has no period without hooks, since they've been on from day one, so we compared sessions with edits against each other. In those where the file-open hook delivered nothing, the agent called the rules check itself in 46% of sessions, and before the first edit in 20%. In those where the hook did deliver rules, the figures were 55% and 22%. So delivering rules doesn't wean the agent off asking; if anything, it reminds the agent that rules exist.

The shares here are higher than in the table because the group is different: it includes sessions with edits and with hooks, starting August 18, when the delivery event appeared in telemetry. So these numbers should only be compared with each other. And there's one thing we can't separate out: the rulebook at the start of work and the rotation reach every session.

The agent writes knowledge more often than it asks for rules: in 22% of sessions versus 16%. We think it's the reminder, because a hook tells it about writing records on every prompt. Nobody reminds it about the wrap-up tools, and that's where the 0.2% comes from. Over 30 days agents made 12,600 calls to AI Council tools, but 15 of the 61 tools were never called. Some of them we built for nothing, and some we described poorly.

Head-to-head: with and without AI Council

Everything above is observation of agents that already have AI Council. We compared "with" and "without" directly in late October and early November 2026, on Keycloak.

The same agent (Claude Code, same model, same settings, same virtual machine) ran seven tasks in a row twice: once without AI Council and once with it. The tasks: three small ones from real GitHub tickets, one covering nine tickets at once, and three large ones that required implementing a specification from scratch. The work was accepted by hidden tests the agent never saw and by Keycloak's own regression suite. The hidden tests are acceptance tests: they hit the running server as a regular client and check its responses against the ticket or the specification. We ran the three large tasks three times in each variant.

What we comparedWithout AI CouncilWith AI Council
Hidden tests and regression, first runPassed in all seven tasksPassed in all seven tasks
Production code across seven tasks10,410 lines8,584 lines, 18% less
Agent's own tests: how many of 14 deliberately planted bugs they catch (three runs of one task)7, 9 and 1314, 13 and 14
Blind review: whose diff a reviewer would take without knowing which is which (three pairs)0 of 33 of 3
Model workabout 495M tokensabout 675M tokens; 1.3x the cost at API rates
Agent run time6 h 25 min7 h 24 min, 1.15x longer

On hidden tests, AI Council gave no advantage. In the first runs both variants passed everything. Of the four repeat runs, three broke the same requirement (don't change the server's existing behavior), and both repeats with AI Council also missed one to three of the 155 checked specification items. The difference showed up where review usually looks, and all of it favored AI Council: code volume, the agent's own tests and the blind diff comparison. The reviewer explained its choice by tests written the way Keycloak writes them and by the documentation listing the limitations.

The price was model work: the seven tasks took about 675M tokens versus 495M, 36% more. But almost all the increase is cache reads, the cheapest kind of token, because with each request the model rereads the accumulated context. Output tokens, the most expensive kind, were about the same in both variants: 1.7M and 1.6M. So in money, at API rates, the difference is not 36% but 30%. The smaller the task, the more noticeable the premium: the small tasks came out 2.4 to 3 times more expensive, and the other four ranged from 20% cheaper to 1.7 times more expensive.

Where the premium comes from is clear. The material AI Council delivered to the agent accounts for 41% of the cost of the "with" variant, and half of that is ticket analyses. In the large-task runs the agent received 894 such analyses and referred to 11 in its work. In other words, we deliver too much. What helped were the records that matched what the agent was doing at that particular moment; matching the overall topic of the task turned out to be of little use. Delivery precision is what we're working on now.

The measurement is small: one project, one model and just seven tasks. The large tasks were written from a specification, where project history helps the least. We haven't yet measured tasks where you can't make the decision without that history.

We estimated cost at API rates, and it's only an estimate: in practice it depends on how you pay for the model. That's why we give ratios, not amounts. The blind reviewer is the same model that wrote both variants, and the same model wrote the hidden tests for the large tasks, from the specification text and without a reference solution. Neither variant comes out ahead on security; probes found defects in both. And the measurement doesn't separate which part of AI Council makes the difference: the rules, the knowledge records or the order of work itself.

The measurement also doesn't explain why the repeat runs with AI Council missed specification requirements. We only have observations. The base had no records about the new protocol, so AI Council brought the agent Keycloak's conventions but not the specification. In the largest task, the context overflowed and got compacted only in the AI Council variant. The agent without AI Council launched a separate reviewer for its code in eight of nine large-task runs; with AI Council, in one. And the completion checklist that the hook sends at the end of a response didn't reach the agent in this measurement; that was a defect on our side. A detailed write-up of the measurement will come as a separate article.

What we didn't measure at all is AI Council's effect on people. On our own projects we see that a new developer finds it easier to get into a project when the base answers "why is it like this" and they don't have to pull in a senior colleague.

When you don't need AI Council

  • The project is small, the rules fit on a couple of screens of CLAUDE.md, and one or two people work on the code.
  • Tasks run one after another, and agents don't work in parallel branches.
  • Tasks are small and self-contained, and everything needed for the change is visible in the code itself. In our comparison, both variants passed the tests on such tasks, and the cost premium was the largest.
  • You only need fast code search. Serena or claude-context will handle that.
  • You only need checks on finished PRs. AI code review services exist for that.

AI Council starts to make sense when three conditions meet: a team, legacy code and several agents at once. That's when not understanding the project gets expensive. It's the weeks a company pays a new developer while they get up to speed, the hours senior colleagues spend explaining things, task briefs that retell the agent the same things over and over, and reviews that catch what the agent could have known in advance. Against that background, an extra 30% on the cost of model work looks small, but you will pay it: our measurement showed no savings on the model itself.

Summary

Many tools give an agent the structure of the code today, and some of them are free. The hard part is what isn't in the code: the "why" decisions, the team's rules, and time. That knowledge has to reach the agent without being asked for, because on its own the agent checks the rules before its first edit in one session out of eight. It also has to stay correct after the next commit and not leak between branches.

In the head-to-head comparison, AI Council gave no gain on acceptance tests. But the code it produced is easier to accept in review: there's less of it, it's better covered by the agent's own tests and it's closer to the project's conventions. The model's work cost 30% more, and half the cost of delivery went to ticket analyses: of the 894 the agent received in the large tasks, it referred to 11.

The agent still starts every day as a new hire; we haven't changed that. But now its desk holds the team's decisions and rules, not just bare code. What's left is learning to hand it exactly what today's task needs and nothing more.

If you want to test the approach on your own project, start a 14-day trial of AI Council and connect your repository.

Sources

Give your agents a project memory

Connect your repository and see what changes in review. 14-day free trial.

Made on
Tilda