An assistant that takes actions in the product

The company's user-facing AI assistant. I was one of its main contributors and did much of the research behind it.

It doesn't just answer questions. It takes actions in the product, leveraging many of the product's features on the user's behalf.

Every request goes down two paths at once. One infers intent. The other assumes the request is a question and starts answering it, because answering is the path that takes real time.

If it is a question, the work is already underway and nothing was spent waiting on the classifier. If intent comes back as navigation, the question path is cancelled and the user goes straight to the page. Neither case makes the user wait.

Being wrong about the guess is close to free. Intent resolves in under half a second, and the question path opens with guardrails and disambiguation, which are cheap — so a cancelled path has spent a few tokens and no more.

Behind the question path is a multi-agent system with agentic loops, using tools we built against our proprietary database, scraped sites, and web search.

Some questions are genuinely hard, requiring many tools and multiple iterations of agent loops because some of the data depended on the data before. That's why we designed the system to show the work happening: the system reports what the agents are doing while they do it. A hard question reads as work in progress instead of a page that might have frozen.

It holds state across conversations and within them. An entity the user disambiguated three turns ago stays disambiguated. The context window is managed as that history grows and the tools pull more into it — a long conversation with real tool use overruns the window otherwise.

Live in production since March 2025

The case for testing

We were adding components to the assistant's agent graphs, adding tools, and making the tools do more, and the assistant's behavior was moving under us while we did it. Nothing was catching that. I argued for regression tests: an inexpensive, controlled cost that wouldn't slow development. Pay that up front to catch regressions and keep them out, or pay far more later stumbling into them, with no consistent way to confirm a fix hadn't caused another.

I designed the suite and proved the design with a proof of concept. The team implemented my design, and today it runs on every merge to main. It measures alignment of real LLM calls against a golden set.

The golden set includes a benchmark response or an expected, static guardrail response, and a list of expected tool calls. Much of the data is private, so an answer assembled without the expected tools is very unlikely to be right.

Different tools covered different parts of the database, and the boundaries between them weren't clean. Information could overlap, could look like it overlapped and not, or could take several tools to assemble. Grading the calls made the system prove it knew which one to reach for.

It also grades that only the expected tools were called — that the agent took its path without casting around for something it didn't need. Churn costs seconds and money. Web search is only a fallback because we want to cite our own data, so an agent that churns instead of finding the right tool can end up there unnecessarily, using expensive tokens. The two are measured separately: whether the answer is right, and whether it was reached without waste.

What they turned up was less regression than inconsistency. The answers varied in ways you couldn't see reading them one at a time — the same kind of question handled one way, then another. The judges weren't the problem: measured against the golden set, they graded consistently. The variance was in what they were grading.

Both fixes went at that directly — an enhancement to a component early in the graph, and few-shot examples written for each tool. The tools are hard to prompt: their schema varies with the asking user's permissions, so what the agent is choosing between isn't the same from one user to the next. The tests showed how much more often the agent got it right when it was given a couple of examples.

The proof of concept that worked, and the design it disproved

Regression tests cover one product. The larger job is evaluation that other work can build on, rather than a harness bolted to a single assistant. I proposed that framework a while ago, and we're building it now.

The proof of concept I built evaluated over 600,000 real data points, and what it caught wasn't a model problem. A service and the people reviewing its output were normalizing differently, so reviewers were hand-editing records that were already correct — just formatted in a way they were instructed to correct.

The first fix went to a field the POC showed had been edited 22,000 times in two months, in ways that could be automated. Several other normalization processes changed after it. That reviewing time now goes to the subjective outputs that are hard to program instead of to formatting.

The same proof of concept was a test of my own design, and it didn't all survive. I had argued for a fully abstract middle layer, but when I gathered the team's real use cases to build against, every one of them was coupled to the data being evaluated: a generic layer would have solved theoretical use cases and carried none of the real, identified work. So I cut the layer I'd argued for, and what we're building is the rest.

The migration that would have shipped quietly broken

A new feature needed a class of data we didn't hold — data millions of users would consume. I was handed the proposal and a crawling service to try to extract it. What I decided was how to judge it: the relevance of what came back, the cost, and the runtime. The POC cleared all three, and I shipped the pipeline.

Later the team identified a different, cheaper crawling service. A spike was done, and I inherited the implementation. I found the spike had missed two things:

• The old service crawled recursively from a starting page and we depended on that. The new one didn't have that capability, so I implemented it by hand.
• The new service dropped each crawl into its own S3 bucket. But those crawls are deliberately complementary to avoid storage redundancy — finding the data means scanning several previous crawls, not only the most recent one.

Either gap would have let the pipeline run without failing and produce visibly worse data. Nothing would have alarmed anyone; quality would just have been bad. I built the bridges to close both before we used the service for anything real, then made it fast enough to be viable, with necessary multiprocessing on the compute-bound paths.

Then I produced something no one had asked for: a concrete quality comparison between the two services on our actual use cases, before switching.

Taking initiative

I joined a team building maps for autonomous vehicles and was tasked with onboarding to a monolith that was known to have issues and in dire need of throughput improvements, with no set plans to identify or address either.

It had been written largely by engineers new to PyTorch, many of them junior. The work ran on a private supercomputing cluster of NVIDIA nodes, and I quickly found multiple GPU memory failures every time the monolith ran. It poorly managed GPU usage and was constantly over allocating memory, failing, and then looking for another GPU. There were also mistakes underneath that — PyTorch layers that should have been frozen and dropout left active during inference. Those were errors rather than decisions, so I fixed them.

Restructuring was a different matter, so I proposed it before touching anything: bring in Dask and turn the whole thing into a DAG so independent stages could run in parallel. I got the go-ahead and did it.

GPU memory stabilized. Throughput and speed both improved dramatically.