Chapter 1: The Data Platform That Grew a Brain

There is a particular kind of Monday morning that data engineers know well.

You arrive before the rest of the office. You have coffee. You open your laptop, and before you can clear the notifications from the weekend, you are already in it — a pipeline has failed, a schema has drifted, a job that ran clean on Friday is now producing output that does not match anything the downstream system expects. You do not know yet what changed. You know only that something did, and that figuring out what will take the rest of the morning, and that the API consuming this data went quiet some time over the weekend, and that someone in a meeting room down the hall is about to ask why.

I have had many of those mornings.

I have also had the other kind — the quiet kind, where you open your laptop and everything is running, and the night before you were not staring at a terminal but reading to your kids, and the system held together without you. Those mornings feel different. They feel like something has shifted, fundamentally, in the relationship between you and the work.

The difference between those two mornings is not luck, and it is not simply a matter of the underlying platform being more stable than it used to be. Data platforms fail in new and creative ways regardless of how mature they get; that is closer to a law of the field than an embarrassment. The difference is that on the quiet mornings, something had already looked at the failure before I woke up, understood what it meant, decided whether it was serious, and either fixed it, flagged it, or left a clear note about exactly what needed my attention and why. On the loud mornings, nothing had done any of that. I was the only thing standing between a problem and a diagnosis, and I found out about the problem at the worst possible moment: after it had already been running unattended for hours.

This book is about how I got from the first kind of morning to the second.

The project started as most data platform projects do: with a clear objective, a list of requirements, and the quiet assumption that the hard part was the data.

We were building an AWS-based managed data platform. The source was an enterprise ERP system, with business semantics structured through semantic views and passed through an enterprise data layer before landing in AWS. The destination was a search-based API serving store. The consumers were internal API clients querying product, customer, and material data at speed.

The scope was substantial but bounded. Ingest the data. Transform it. Write it to the serving store. Serve it. The usual pipeline shape: sources on one end, consumers on the other, a series of AWS services handling the movement in between.

I had done this before, in various forms. Data moves. You build the pipes. The data lands somewhere useful.

What I had not done before was build it the way I was about to build it.

The first AI tool I used on this project was Claude — Anthropic's large language model, accessed through their API and later through Claude Code, their CLI. I started using it the way most engineers start: for code assistance. Draft this function. Explain this error. What does this IAM policy do?

It was useful. It was genuinely useful. But it was a hammer, and I was using it on every nail regardless of whether the nail matched. Some of those nails were screws.

Then, sometime in the early weeks of the project, I started asking a different kind of question.

Not "write this function" but "how should I think about this problem?" Not "explain this error" but "what would break if I changed this?" Not "draft this policy" but "what are the security implications of what I am about to do, and what am I probably missing?"

And something shifted.

A well-prompted language model does not just answer your question. It reflects your problem in a different frame. It surfaces the assumption you were not examining. It asks the follow-up you did not think to ask yourself. It is not artificial intelligence in the science-fiction sense — it does not understand your problem the way a colleague does — but it is something genuinely useful, which is a tireless thinking partner that has read a great deal and never gets tired of your questions.

I started leaning into it.

The natural next step, once you have a capable AI assistant, is to try to make it autonomous.

Not autonomous in the "replace the engineer" sense — that framing has always seemed to me to misunderstand both what engineers do and what language models do well. But autonomous in the practical sense: can this system do reliable, bounded work while I am not watching?

Can it run the overnight analysis? Can it write the report while I am in meetings? Can it check the data quality rules at midnight and surface a finding by the time I arrive in the morning?

The answer, I discovered, is yes — but only if you have built the right infrastructure around it.

And building that infrastructure turned out to be the real project.

The data platform is, from one angle, an AWS data engineering project. Cloud landing buckets. Data transformation jobs. Workflow orchestration. A search-based serving layer. Monitoring and observability. Standard enterprise architecture, competently assembled.

From another angle, it is an operating system for an agentic workforce.

The agents I am referring to are not physical robots, nor are they the autonomous agents of speculative fiction. They are AI assistants, each built to perform a specific class of task, each governed by explicit rules about what they may and may not do, each connected to a shared memory system that lets them learn from each other's work.

As I write this, that roster has grown to several hundred agents, and it keeps growing — any specific number I gave you here would likely already be stale by the time you read it, which is itself a small lesson this book will return to.

To be clear about what that number means: it is not several hundred copies of the same thing. It is several hundred specialized entities, each with a distinct purpose, each with defined inputs and outputs, each with explicit governance constraints. A security auditor who never writes code. A diagram specialist who never touches infrastructure. A session coordinator who opens every morning and closes every evening. An agent whose entire job is to calculate whether the overnight run is worth the tokens it will cost.

The first agent I built was a general-purpose assistant. It was capable of anything, which meant it was optimized for nothing. Most requests today route to a specialist rather than falling back to that original generalist — a ratio I watch, not a fixed number I can quote with a straight face, since it moves week to week as the roster and the work both change.

That is not a coincidence. That is the architecture.

I remember the specific conversation that first showed me the shape of this problem, because it happened outside the industry entirely. Over dinner, a friend with no data background and no AI background asked what I had been disappearing into most weekends for the better part of a year. I said I had built two hundred and fifty specialized AI agents. She laughed — not unkindly — and said it sounded like the part of a heist movie where the crew introduces itself one member at a time. Then she asked the question that actually mattered, the one I did not have a clean answer to that night: what does two hundred and fifty of anything do that twenty could not? I have an answer now. It took the rest of this chapter's argument, and a fair amount of failure, to earn it. This is the reconstructed shape of that conversation, not a transcript — but the question itself is exactly the one I kept getting, in one form or another, for the next year.

The number two hundred and fifty tends to provoke one of two reactions, and both of them miss what actually matters.

The first is skepticism: it sounds like a marketing number, the kind of thing that gets claimed and never actually used. I would have had the same reaction a year earlier — if someone had told me I would eventually need a written policy about which agent is allowed to name other agents, I would have assumed they were describing a support group, not a codebase. I would have been fully confident in that assumption, too, which is a pattern you will see me repeat throughout this book, usually right before something I built myself corrects me. The number is real, and also less impressive than it sounds, because it did not arrive through grand design. It arrived one gap at a time: a task kept landing on the wrong tool, or no tool at all, until that became annoying enough that I built something narrow to handle exactly that gap and nothing more. Multiply that by a year and a half of continuous work and you get two hundred and fifty. Less a testament to foresight than to persistence.

The second reaction is the opposite: if two hundred and fifty is good, isn't five hundred better? This is the instinct I actually had to fight, and fighting it mattered more than building any individual agent. A roster that grows without discipline becomes exactly the sprawl it was meant to prevent — overlapping specialists, none clearly the right choice for a given task. I have retired agents. I have merged two specialists into one because their boundaries had drifted close enough that routing between them was a coin flip. Growth here is not automatically good; growth without pruning is just a different flavor of the mess I was escaping.

What actually matters is not the raw count. It is the ratio of general-purpose fallback to specialist routing, and whether that ratio holds as the roster grows. A small, steady fallback share, at this scale, tells me the specialization is holding its shape. A fallback share climbing toward a real fraction of total requests would tell me the registry had drifted out of sync with the actual work. I watch that ratio the way an operations team watches an error rate — the single number that tells me, faster than any manual audit, whether the architecture is still doing its job.

The actual data platform work has stakeholders who could not care less how many specialized agents exist. They want the source semantics honored correctly, the search index fast and right under real load, the API contract to hold. The operating-system scaffolding matters to them only if it helps deliver those things.

What convinced me it was worth building was watching the same category of mistake happen more than once, in different disguises, because nothing in the system remembered the first time. A specialist invoked under a false byline. A capability authorized on paper and never actually built because nobody converted the authorization into a task. A registry entry describing a narrower scope than the agent had quietly grown into. Each is small on its own. Left unaddressed, they compound, the same way undetected schema drift compounds until someone downstream gets an answer that is quietly wrong.

I did not set out to build an operating system. I set out to stop re-explaining myself every morning, and to stop discovering, months later, that a specialist I thought I had built had never actually done any real work. The operating system is what accumulated between those problems and their solutions.

The rules became explicit: which agent could touch which files; which decisions required my sign-off; what counted as evidence that work was done. They became the project's governing instructions, read at the start of each session. No production changes without human review. No secrets in generated files. No destructive action without an explicit approval phrase typed by me, not inferred from context. A memory system kept a lesson about, say, a Windows-specific bug available to the next session that touched the same kind of code.

A rules file. A registry. A memory system. None of these pieces sounds remarkable alone. Each exists because something failed without it, and each gives the next session a record to check instead of relying on my memory. That is how this book is organized: around the failures that made the rules necessary.

I call the result an agentic operating system: a structured, governed framework for deploying AI agents to do real work, within boundaries set by a human who remains the final authority. The data platform is still in progress. Its source systems, transformation jobs, access policies, and serving layer are the work this framework has to support. This book describes how I built it, incrementally, and what it took to trust it with bounded work while I slept.

You do not need a data platform to recognize the problem. If you use an AI tool for anything you do more than once — drafting the same kind of message, answering the same kind of question, producing the same kind of document — you have probably met some version of these three frustrations. Quality that varies without explanation. Context that vanishes between sessions. A solved problem you cannot find again when it returns in a new disguise.

You do not need two hundred and fifty agents to feel that cost. You need one recurring task and enough repetitions to notice the pattern. My system scaled to match a data platform. The underlying problem would have been worth naming at any size.

The story starts, as most engineering stories do, with a problem I did not fully understand until I had been trying to solve it for three months.

The second Monday morning was three months into the project. I had started using Claude Code as my primary development environment. I had accumulated a handful of useful agents. I had begun, almost accidentally, to notice a pattern in how the work was going — which tasks I was doing myself, which I was delegating to agents, which felt stuck despite delegation.

It did not arrive as a single bad morning. If it had, I think I would have fixed it faster. Instead it arrived the way most real engineering problems arrive — as a low, persistent hum you get used to before you notice it is there at all.

The first thread of it was quality. The same kind of task — a security review, a piece of documentation, a data validation check — would come back sharp and useful one week and flat and generic the next. I did not have a good explanation for the difference. Sometimes I chalked it up to how I had phrased the ask. Sometimes to which model was behind the tool that day. It took longer than I am comfortable admitting to notice the actual pattern: the good outcomes were the ones where, without really deciding to, I had happened to give the work to something narrow and specific. The flat ones were the ones I had handed to whatever was generically available. I was not routing. I was just reaching for whatever was closest, the way you reach for whichever wrench is on top of the pile instead of the one that actually fits the bolt.

The second thread was context. Every time I sat down to work with an AI tool on something related to the data platform, I found myself re-explaining the same handful of things: the constraints of the source system, the shape of the downstream consumers, the specific things I had already learned not to do. None of it persisted. Every session started from zero. I was, in effect, re-onboarding a new colleague every single morning, except the colleague was capable and the onboarding never stopped.

The third thread was the quiet, expensive one. I would solve something — a genuinely hard problem, the kind that takes real effort to work through cleanly — and then, months later, face something close enough to it that I should have been able to reuse the answer. I couldn't. There was no record. The solution existed only in whichever chat window I had solved it in, already scrolled out of reach, already forgotten. I was not building on what I had already figured out. I was rediscovering it, at cost, again and again.

None of these three things, on its own, would have been enough to make me stop and build something new. Inconsistent quality is a tuning problem. Context loss is an annoyance. Forgotten solutions are a filing problem. But the three of them together, happening at the same time, on the same project, week after week, added up to something I could no longer describe as friction. It was a shape. And once I saw the shape, I could not unsee it: I did not have a set of tools. I had a set of unrelated, ungoverned, forgetful interactions, each one starting over, each one only as good as whatever I happened to reach for that day.

I pulled out a blank document and wrote down what I thought was happening.

The list was longer than I expected. And the shape of it told me something I had not yet named.

I needed a registry.