Architecture · PiniShv

Open WebUI Isn't a ChatGPT Clone. It's AI Infrastructure.

Wed, 18 Mar 2026 10:00:00 +0200

Here’s a question nobody’s asking: who owns the layer between your engineers and the AI models they use every day?

Right now, for most teams, the answer is OpenAI. Or Anthropic. Or Google. Your engineers open ChatGPT, or Claude, or Gemini, and they work inside someone else’s product. Someone else’s UI. Someone else’s data policies. Someone else’s feature roadmap.

That’s fine when AI is a nice-to-have. It stops being fine when AI becomes how your team actually works.

Open WebUI is the project that makes this question real. Not because it’s a better chatbot. Because it turns the AI interface layer into infrastructure you can own, deploy, and control. And once you understand what that means, the conversation about AI tooling changes completely.

What Open WebUI actually is
#

Strip away the GitHub stars (128k+ and counting) and the marketing language about “bringing intelligence home.” What you’re looking at is a self-hosted control plane for AI models.

It runs in a container. Docker, Kubernetes, Podman, Helm, whatever your infra looks like. First account becomes admin. Later signups need approval. For a solo setup you can disable login entirely. One container, local storage, browser UI. You’re up and running.

But the interesting design decision is that it’s protocol-first, not vendor-first. Open WebUI uses OpenAI Chat Completions as the shared language across providers. It has compatibility layers for Anthropic. It supports Ollama for local models. It can route to any OpenAI-compatible backend. That makes it less like “an Ollama UI” and more like an operations layer sitting above whatever models you choose to run.

This is the same architectural pattern we’ve seen play out in infrastructure before. Think about how Terraform became the control plane above cloud providers, or how Kubernetes became the orchestration layer above compute. Open WebUI is making that same move for the AI interface layer.

What it can actually do (beyond chat)
#

Most people discover Open WebUI because they want a local ChatGPT. Then they realize the feature surface is much wider than they expected.

RAG and knowledge work. Multiple vector databases, document uploads, URL ingestion, web search across 15+ providers, and full-page URL fetching. This isn’t a toy retrieval setup. It’s a real knowledge pipeline.

Agent capabilities. Open WebUI distinguishes between Tools, Functions, and Pipelines. It supports MCP natively. It can attach external actions like search, scraping, image generation, and voice. It can expose MCP through OpenAPI-compatible flows. This is an agent platform, not just a chat box.

Code execution. Python through Pyodide or Jupyter, Mermaid rendering, interactive artifacts. At the extreme end there’s Open Terminal, which gives the model a real OS-level environment in a container. That’s powerful and terrifying in equal measure.

Team workflows. Folders, projects, chat history, shared conversations, channels for multi-user collaboration, RBAC, SCIM provisioning, OpenTelemetry. The admin surface is deeper than most people expect from an open-source project.

Media and voice. Image generation and editing, speech-to-text and text-to-speech with local, browser, and remote options.

The feature list is impressive. But feature lists are easy. The real question is what happens when you actually run it.

The reality of running it in production
#

For a hobbyist or solo developer, Open WebUI is deceptively simple. Container up, connect a model, start chatting.

For production, the defaults are just defaults. Out of the box you get SQLite, embedded ChromaDB, and one Uvicorn worker. That’s fine for one person. The moment you want multi-worker or multi-node deployment, the project tells you to move to PostgreSQL with PGVector, Redis for caching, and shared storage. Easy to start. Not magically “no-ops” once it matters.

If you use RAG heavily, the reality gets sharper. The project’s own scaling guide warns that the default PDF extractor and default embedding path are common causes of memory leaks and RAM blowups at scale. They explicitly recommend externalizing them in production.

I’m not saying this to dismiss the project. I’m saying it because this is exactly the kind of detail that separates “I read the feature list” from “I actually deployed it.” If you’re considering Open WebUI for your team, go in with eyes open. This is infrastructure. Infrastructure requires ops.

Who’s behind it and why that matters
#

Open WebUI is led by founder Tim J. Baek and backed by Open WebUI, Inc. The team page credits community contributors, but the organization is explicit that it’s not looking for outside governance advice. This is founder-led open source, not a neutral foundation-governed commons.

Why does that matter? Because the business model is visible in the decisions.

Since version 0.6.6, the project added a branding-protection clause for larger deployments. Code up to v0.6.5 remains under the original BSD-3 terms. Enterprise offerings include theming, SLAs, LTS, and direct support. This is the standard playbook: open core with enterprise upsell.

The community has opinions about this. Some people on Hacker News get sharp about the licensing change and the fact that a project called “Open” WebUI has branding restrictions. Others say they don’t care because they’re not planning to fork it anyway.

My take: this is a normal and healthy tension. Building sustainable open-source software costs money. Branding protection is one of the less invasive ways to fund it. But if you’re betting your team’s AI infrastructure on this project, you should understand the governance model you’re buying into.

The security conversation nobody wants to have
#

Here’s the uncomfortable part.

Open WebUI’s Tools, Functions, Filters, Pipes, and Pipelines execute arbitrary Python on your server. The docs say “only install from trusted sources.” That’s honest, but it also means the extension system is a real attack surface.

This isn’t theoretical. A code-injection issue in Direct Connections was patched in 0.6.35. An SSRF issue in retrieval processing was patched in 0.6.37. Both are the kind of vulnerabilities that come with running user-extensible systems.

For your team, this means treating Open WebUI the same way you’d treat any infrastructure component: pin versions, review extensions, monitor for CVEs, control who can install what. The freedom to extend the platform comes with the responsibility to secure it.

Why teams and orgs actually adopt this
#

Features are nice. But nobody migrates their AI tooling because of a feature checklist. They do it because something about the current setup is broken. I spent time researching the best tools for an internal ChatGPT alternative, talking to other engineering leaders who did the same. Here’s what actually drives the decision.

Cost visibility and control. When your team uses ChatGPT or Claude directly, every person needs a subscription. Or worse, everyone shares credentials. Or worst of all, engineers use their personal accounts and company data flows through consumer products with consumer privacy terms. With Open WebUI in front of your API keys, you get one set of credentials, usage tracking per user, and the ability to route different workloads to different models based on cost. Need a quick answer? Route to a cheap local model. Need deep reasoning? Route to Claude or GPT. Same interface, conscious cost allocation.

Data stays where you decide. For a lot of orgs this is the whole conversation. Regulated industries, government contracts, security-conscious startups. The moment your engineers paste proprietary code into ChatGPT, you have a data governance problem. Self-hosting the interface layer means the data flows through your infrastructure, your logging, your retention policies. You can run sensitive workloads on local models that never leave your network, and routine tasks on cloud APIs. Same UI for both.

No vendor lock-in on the workflow layer. This is the one that hits engineering leaders hardest. Today your team builds workflows, prompt libraries, knowledge bases, and habits around ChatGPT. Tomorrow OpenAI changes the pricing, kills a feature, or deprecates a model. Everything you built around their interface is tied to their decisions. When the interface is yours, the models are pluggable. You can switch from GPT to Claude to Gemini to a local model without retraining your team or rebuilding your workflows.

Unified AI experience across the org. Instead of some engineers using ChatGPT, some using Claude, some using local models, and nobody sharing anything, everyone works through one interface. Shared conversations, shared knowledge bases, shared tools. New team member joins, gets access to the same AI setup as everyone else. That might sound like a small thing until you’ve managed an engineering org where every person has their own disconnected AI workflow and none of that institutional knowledge is captured anywhere.

A real sandbox for innovation. Want to test a new model? Add it as a backend. Want to build a custom agent for your team? Use the extension system. Want to integrate your internal knowledge base? Plug in RAG. Want to give your AI access to your tools via MCP? It’s supported. You don’t need to wait for OpenAI or Anthropic to ship a feature. If you can build it, you can plug it in. For teams that move fast, that’s the difference between waiting for a vendor’s roadmap and building what you need right now.

None of this is free. You trade managed simplicity for operational responsibility. But for teams that are serious about AI being part of how they work, not just a tool they occasionally open, owning the interface layer starts making a lot of sense.

How it compares to the incumbents
#

The comparison isn’t really about features. It’s about what you’re optimizing for.

Versus ChatGPT. ChatGPT has Projects, Deep Research, Apps, Company Knowledge, and mature business controls. SSO, retention policies, permissions, training defaults. It’s zero-ops SaaS. Open WebUI’s advantage is that you own the stack. Data stays local. You mix local and remote models. You’re not locked to one vendor’s interface. If zero-ops matters most, ChatGPT wins. If ownership matters most, Open WebUI wins.

Versus Claude. Claude has Artifacts, Projects, Skills, Research, and Google Workspace integration. Anthropic also created MCP. Open WebUI can route to Claude’s models, but Anthropic’s own docs note that their OpenAI-compatible endpoint is mainly for testing, and the native API is recommended for the full feature set including PDF processing, citations, extended thinking, and prompt caching. Protocol compatibility is powerful, but it flattens vendor-specific superpowers.

Versus Gemini. Gemini is strongest when your work already lives in Google’s ecosystem. Deep Research can pull from Search, Gmail, Drive, and NotebookLM. Open WebUI is the better fit if you want one interface above Google models, Anthropic models, OpenAI models, local models, and whatever comes next.

The pattern is consistent: the SaaS products win on managed experience and vendor-native depth. Open WebUI wins on control and independence. Neither is wrong. They’re different bets.

How it compares to open-source alternatives
#

The open-source landscape is more nuanced.

LibreChat is probably the closest direct competitor. Agents, MCP, artifacts, code interpreter, broad provider support. It reads like the closest open-source answer to the mainstream chat products. Open WebUI feels more infrastructure-oriented, more invested in deployment patterns, admin controls, and the local/offline story.

AnythingLLM leads with “chat with your docs.” Built-in agents, multi-user support, vector databases, document pipelines, no-code agent builder. If your center of gravity is private documents and internal knowledge workflows, AnythingLLM has a clear story. Open WebUI is broader if you want one extensible front end for many kinds of AI workflows.

Onyx is enterprise-search-heavy. Connectors, synced knowledge sources, deep research, MCP, enterprise knowledge grounding. Compelling when “AI over company knowledge” is the main requirement. Open WebUI is a general AI workspace. Onyx is sharper as an enterprise retrieval layer.

Jan is desktop-first and personal. 100% offline, runs on your laptop, turns it into an AI workstation. Great for single-user local AI. Open WebUI becomes more compelling the moment you want browser access, shared workspaces, or team deployment.

What this actually means for engineering leaders
#

Here’s the strategic point that matters more than any feature comparison.

For the last two years, the AI interface layer has been bundled with the model provider. You use ChatGPT because you want GPT. You use Claude because you want Anthropic’s models. The interface and the intelligence came as a package deal.

Open WebUI (and projects like it) are unbundling that. The model is one layer. The interface is another. And once those layers separate, the dynamics change.

Your team can switch models without switching workflows. You can run sensitive workloads on local models and routine work on cloud APIs, through the same interface. You can add RAG, agents, and custom tools without waiting for OpenAI to ship them. You can audit, log, and control every interaction.

The price of that freedom is real. You own deployment. You own patching. You own extension security. You own operational tuning. You inherit everything that SaaS normally hides behind a login page.

That’s not a reason to avoid it. It’s a reason to approach it the way you’d approach any infrastructure decision: with clear requirements, honest assessment of your ops capacity, and a plan for what happens when things break at 3am.

Who should care about this
#

If you’re a solo developer who wants a better local AI setup, Open WebUI is probably the best option out there right now. Install it, connect your models, enjoy.

If you’re an engineering leader evaluating AI tooling for your team, Open WebUI is worth understanding even if you don’t deploy it. It represents where the AI tooling ecosystem is heading: model-agnostic interfaces, self-hosted control planes, protocol-first architectures. The question isn’t whether this pattern wins. It’s how fast.

If you’re already running AI agents in production (like I am), Open WebUI is interesting as the potential front end for your entire AI operations layer. One interface for your agents, your knowledge base, your model routing, your team’s AI workflows. That’s a compelling vision. Whether the project can deliver on it at enterprise scale is still an open question.

Either way, the conversation has shifted. It’s no longer just about which model is best. It’s about who controls the layer where your team meets the model. Open WebUI is one of the first projects to take that question seriously.

And that’s worth paying attention to.

Running self-hosted AI infrastructure? Thinking about owning the interface layer? I’d love to hear what you’re using. Find me on X or Telegram.

Your AI Agents Are Flying Blind. Here's How to Fix That.

Sun, 15 Mar 2026 10:00:00 +0200

Your AI agent just rewrote the authentication flow. The code is clean. Tests pass. The PR looks great.

One problem: it broke the SSO integration with three enterprise customers because it didn’t know the auth service has a contract with the identity provider that requires a specific token format. That contract lives in a Slack thread from 2023 and one engineer’s head.

The agent didn’t make a mistake. It made a perfectly reasonable decision with the information it had. The information it had was almost nothing.

This is happening across your codebase right now. Not just with authentication. With everything. Business rules, API contracts, deployment constraints, database conventions, service boundaries. Your agents write code that compiles, passes tests, and violates assumptions that live nowhere except in people’s heads and scattered documents nobody maintains.

I’ve written about why context is the fundamental problem in AI. I’ve written about putting AI agents on the org chart and managing them like team members. But none of that matters if the agents start every session blind.

If you’re running agents in production, this is the problem you need to solve next.

Two teams, same agents, wildly different results
#

Let me describe what I’m seeing.

Team A has agents embedded in their development workflow. An agent picks up a ticket to add a new validation rule to the user registration flow. Before writing a line of code, it queries a knowledge base and gets back: the existing validation rules, the reason the email format check is stricter than RFC 5322 (because of a legacy migration), the API contract with the notification service, and the team’s convention for error handling. The agent writes code that fits. The PR gets approved on the first review.

Team B has the exact same agents, same models, same IDE. Their agent picks up a similar ticket. It reads the code in the repo, sees patterns, generates a solution. The solution uses a different error handling pattern than the rest of the codebase. It changes the validation response format, which breaks the mobile client. It adds a database column without following the team’s migration conventions. The PR gets three rounds of review comments and a refactor.

Same AI. Same capability. Completely different outcomes.

The difference isn’t the model. It’s that Team A solved the knowledge problem and Team B didn’t.

Where knowledge actually lives (and why that’s broken)
#

In most engineering organizations, critical knowledge is scattered across:

People’s heads. The worst possible storage medium.
Slack threads. Searchable in theory, buried in practice.
Confluence pages. Written once, updated never.
Code comments. Spotty at best, misleading at worst.
Tribal knowledge. “Ask Daniel, he built that service.”

None of this is accessible to AI agents. None of it is structured for retrieval. None of it stays current.

And here’s the compounding problem: as AI agents do more work, the knowledge gap matters more, not less. When humans wrote all the code, at least the person writing it carried the context. When agents write the code, the context has to come from somewhere else. Or it doesn’t come at all.

Think about it this way: a senior developer who’s been on your team for three years carries hundreds of micro-decisions in their head. Why the payment service retries exactly three times. Why the user permissions check happens at the API gateway, not the service layer. Why that database query uses a specific index hint. Now imagine replacing that developer with an agent that knows none of this. That’s what you’re doing every time an agent starts a session.

The wrong way to fix this
#

The instinct is to throw more code at the agent. Bigger context windows. More files in the prompt. RAG over the entire codebase.

I’ve seen teams try this. Here’s what happens:

They dump the entire repo into the context. The agent drowns in irrelevant code and can’t find the signal, and every token costs money, so you’re paying premium rates to confuse your own agents. They build RAG over Confluence. The retrieval returns pages from 2021 that contradict how things actually work. They write massive README files. Nobody maintains them. Within three months they’re more misleading than helpful.

And the costs compound. More tokens in the context means higher API bills on every single request. Bad context leads to wrong code, which leads to longer review cycles, which leads to rework, which means more agent sessions with the same bad context. It’s compound interest working against you. Every layer of waste multiplies the next.

The problem isn’t volume of information. It’s the right information, maintained, structured, and delivered at the moment the agent needs it. Get this wrong and you’re not just getting bad code. You’re paying more for it with every iteration.

What actually works: a developer knowledge hub
#

After months of thinking about this problem and looking at how every available solution falls short, I believe the answer is a system with three components that work together.

Layer 1

Source of Truth

📁

Knowledge Repo (Git)
Developers author markdown: product rules, system docs, architecture specs, skills

▼ CI/CD syncs on every merge ▼

Layer 2

Index & Push

🔍

Vector Store + Embeddings
Chunk, embed, index → semantic search

📄

AGENTS.md + Skills per repo
Generated context + reusable workflows

▼ Serves queries at dev time ▼

Layer 3

Universal Bridge

🔌

MCP Server
One server → every IDE & agent can query knowledge

▼

Consumers

All tools

💻

Cursor Claude Code Copilot Codex Kiro + any future MCP-compatible tool

Git for authoring
#

Not Confluence. Not Notion. Not some SaaS product with its own editing UI.

A Git repository. Markdown files. Pull requests for review. CI/CD for automation. The same workflow developers already use for code.

Why Git? Because the adoption problem kills every knowledge initiative that requires developers to learn a different tool. PRs already have review workflows. Blame shows who wrote what. History shows when things changed. CODEOWNERS controls who can approve what. Your developers already know all of this. Zero adoption friction.

The repo holds four types of knowledge:

Product knowledge. Business rules, domain logic, edge cases, validation requirements. Why the user registration flow requires that specific email format. Why the discount calculation has a different rounding rule for enterprise customers. This changes every sprint.

System knowledge. Build commands, repo structure, coding conventions, database patterns, module boundaries. Why you always run migrations before the test suite. Why the cache invalidation uses event sourcing instead of TTL. This changes when code changes.

Architecture knowledge. API contracts, data flows, service boundaries, system invariants. Why the payment service is the only service allowed to write to the transactions table. Why the notification queue has exactly-once delivery semantics. This changes rarely but matters enormously.

Operational skills. Code review checklists, debugging guides, feature scaffolding patterns, cross-repo change workflows. How to add a new API endpoint. How to set up a feature flag. How to run a database migration across services. How the CI/CD pipeline works, which checks run on PR, which run on merge, what gates production. How linting and formatting are enforced and what to do when a check fails. How to roll back a deployment. How to triage a failing build. These are reusable agent workflows that encode how your team actually works. Not just the code, but the entire delivery process around it.

One thing you’ll notice is missing from this list: the code itself. That’s intentional. AI IDEs and coding agents like Cursor, Copilot, and Claude Code already do a solid job indexing your codebase. They understand file structure, imports, function signatures. You don’t need to duplicate that work. What they can’t index is everything around the code. The why, the rules, the decisions. That’s what the knowledge hub is for. That said, the system is designed to be agile. If you want to add code indexing, documentation from other sources, or any other category of data, the architecture supports it. Same Git authoring, same search layer, same MCP delivery.

Semantic search for retrieval
#

Raw markdown is great for humans. Useless for agents that need to find the right three paragraphs out of thousands for a specific task.

This layer chunks the markdown by section, embeds it into vectors, and indexes it for semantic retrieval. When an agent asks “what are the validation rules for the registration flow?” it gets the relevant sections, with citations back to the source documents.

AWS Bedrock Knowledge Bases does this out of the box. So does Pinecone, Weaviate, or any vector store with a decent chunking strategy. The specific tool doesn’t matter. What matters is that knowledge becomes semantically searchable, not just keyword-matchable.

CI/CD syncs markdown to the search index on every merge. Knowledge stays current automatically. No manual re-indexing. No stale embeddings.

MCP for delivery
#

Here’s where it comes together.

Your developers use Cursor, Claude Code, Copilot, Codex, Kiro. Probably several of them. Each one is an island. Each one starts every session without context.

Model Context Protocol (MCP) is the open standard that connects all of them. I wrote a deep dive on MCP earlier. If you haven’t read it, start there.

One MCP server wraps your knowledge base and exposes it to every IDE and agent through a standard interface. Build one server. Every tool connects natively. New tools that support MCP work automatically. Zero per-tool maintenance.

The server exposes three tools: search_knowledge for semantic search across all knowledge, get_document to fetch a specific doc by path, and list_knowledge_bases to discover available sources. Simple interface, massive impact.

Without MCP: You build a separate integration for each IDE. Maintain six connectors. Each tool gets knowledge differently. Every new tool means new work.

With MCP: You build one server. Everything connects. When the next AI coding tool launches next month, it just works.

The loop that makes it compound
#

Here’s where this gets really powerful. The system doesn’t just serve knowledge. It grows.

🔍

Read

Agent queries KB
via MCP

→

💻

Work

Develops with
full context

→

📝

Write Back

Opens PR to
knowledge repo

→

✅

Merge

Dev reviews
CI re-indexes

↩

🔄

Updated

Next session
starts smarter

Fully automated. No manual curation. Knowledge grows as the team develops.

The workflow in detail:

Agent reads. Before starting work, queries the knowledge base via MCP. Gets business rules, conventions, architecture constraints relevant to the task.
Agent works. Develops with full context. The code actually follows the patterns and rules.
Agent writes back. A built-in skill instructs the agent to capture what it learned during development and open a PR to the knowledge repo.
Developer reviews. Standard PR review. Approves or refines the knowledge doc.
CI syncs. Merged knowledge is automatically indexed. Next agent session starts smarter.

Knowledge capture becomes part of development, not a separate chore. The developer just reviews. No separate authoring step.

There’s a sixth step that takes this even further. When new knowledge merges, a CI step can run an LLM over the diff and ask: “What else in the entire knowledge base might be affected by this change?” Remember, this is a centralized system across all your repos. A change to how one service handles authentication could affect product knowledge for three other services, architecture docs for the API gateway, and operational skills for the deployment pipeline. The system uses embeddings to find related documents across every domain, checks for contradictions or staleness, and opens follow-up issues flagging what might need updating. Ripple effect detection across your entire engineering knowledge. You update the validation rules for user registration, and the system flags that the API contract doc, the mobile client integration guide, and the error handling conventions might all need a second look. It’s cheap to run and catches the kind of cross-cutting knowledge drift that humans miss because nobody has visibility into every document across every team.

Every feature built makes the next feature easier. Every agent session makes the next session smarter. The knowledge compounds.

The AGENTS.md safety net
#

Not every agent session has MCP access. Sometimes developers work offline. Sometimes a new tool doesn’t support MCP yet. Sometimes the knowledge server is down.

For these cases, CI generates a lightweight AGENTS.md in each repo. It’s a table of contents for the agent: what this repo does, how to build and test it, architecture boundaries, conventions and constraints, and where to find the full knowledge base.

Think of it as the offline fallback. Agents get essential context even without network access. Push model (always in-repo) complementing the pull model (on-demand via MCP).

Why nothing on the market solves this
#

I looked at many solutions out there. Each solves a piece, and the approach I’m describing borrows the best parts from all of them.

Meta-repos (centralized Git docs). Git-native authoring, but no semantic search. Agents can’t find what they need.

Wiki + RAG (Confluence/Notion with retrieval). Searchable, but not Git-native. Developers won’t update it. Knowledge rots within months.

Code wikis (auto-generated from code). Clever, but usually tied to one AI tool. Not universal.

Cloud RAG services (Bedrock KB, Vertex). Managed search, but no authoring story. Where does the content come from?

Agent memory (Copilot memory, Letta). Per-tool, per-session. Not centralized. Not shared across the team.

You need all five capabilities in one system. That’s what this approach delivers.

How to start (without boiling the ocean)
#

Day 1: Create the knowledge repo. Sit with your two or three most senior engineers, the ones who carry the most context in their heads. Ask them: “What do you find yourself explaining over and over?” That’s your first knowledge document.

Day 2-3: Set up semantic search. Connect your markdown to a vector store. Get retrieval working. This is not a multi-week project. The tooling exists. Use it.

Day 4-5: Deploy the MCP server. Configure it in your team’s primary IDE. Have a developer pair with an agent on a real task and compare the output to what they’d get without the knowledge base. That’s your first signal.

Week 2: Add the write-back loop. Build the skill that instructs agents to capture knowledge after completing work. Train your developers on how to review knowledge PRs, not just code PRs. This is where it starts compounding.

The technology side of this is days of work. The harder part is getting your team to treat knowledge as a first-class deliverable, not an afterthought. That’s a leadership problem, not a tooling problem. But once developers see their agents producing better code because someone took 20 minutes to document business rules, the culture shift happens on its own.

We’re in the AI era. If the infrastructure takes you months, you’re overengineering it. Get something working in days, iterate from there. The humans will make it great.

The key insight: start with the knowledge that hurts most when it’s missing. That’s usually the domain logic, the business rules that experienced developers carry in their heads and that agents get wrong in ways that look correct until they hit production.

The uncomfortable question
#

If your AI agents are generating code without context, how much of that code is actually correct?

Not “does it compile” correct. Not “does it pass the tests you wrote” correct. Actually correct. Follows the business rules, respects the architecture, uses the conventions, handles the edge cases that burned you last quarter.

If you can’t answer that confidently, your agents aren’t helping as much as you think. They’re generating plausible-looking code that somebody has to review against all the unwritten knowledge that exists only in people’s heads. And you’re paying for every token of that wrong output, then paying again for the review, again for the rework, and again when the agent generates the same mistake tomorrow because nothing changed.

That’s not an AI problem. That’s a knowledge management problem. And it’s solvable.

The organizations that figure this out first will have AI agents that don’t just write code. They write the right code. Every time. From session one.

That’s the difference between AI as a novelty and AI as a genuine multiplier. And it’s what separates teams that are actually shipping with agents from teams that are just generating code and hoping for the best.

Building knowledge systems for AI agents? Thinking about MCP? I’d love to hear how you’re approaching it. Find me on X or Telegram.

Building AI Systems That Don't Break Under Attack

Sun, 12 Oct 2025 00:00:00 +0000

This is Part 2 of the “Securing Intelligence” series on AI security.

In Part 1, we looked at how prompt injection has evolved from party tricks to production threats. We covered indirect injection, cross-context attacks, and the uncomfortable reality that every defense can be circumvented. That’s the problem space.

Now comes the harder question: if perfect security is impossible, what does responsible AI deployment actually look like?

I’ve spent 15+ years in software engineering, development, and technical leadership, with recent years deeply focused on AI—both building production systems and guiding 100+ engineers on how to work with it. I’ve seen what separates organizations that sleep soundly from those waiting for their incident. It’s not about having perfect defenses. It’s about having defenses that work together, that fail gracefully, and that make attacks expensive enough that most attackers move on to easier targets.

The Foundation: Structured Prompts and Separation of Concerns
#

The first line of defense is architectural. If you’re mixing system instructions and user input in the same unstructured blob of text, you’ve already lost.

Structured prompts treat instructions and data as separate entities with clear boundaries. Think of it like the difference between eval(user_input) and proper API calls with typed parameters. One is begging to be exploited; the other has clear attack surfaces.

Here’s what this looks like in practice:

SYSTEM_CONTEXT (immutable):
You are a customer support assistant for Acme Corp.
You can access customer records and order history.
You cannot process refunds without manager approval.
TRUSTED_DATA (verified sources):
Customer #12345: Premium account, joined 2020
Order #789: $299.99, shipped 2025-10-10
USER_INPUT (untrusted):
[User's actual query goes here]

The key is that your application logic treats these as distinct components. Your system prompt isn’t just text at the top of your context window that can be overridden by clever user input; it’s enforced at the API level, in your orchestration layer, before it ever hits the LLM.

OpenAI’s structured outputs API and Anthropic’s system messages both support this pattern natively. Use them. Don’t try to enforce separation purely through prompt engineering. That’s like trying to prevent SQL injection by asking users nicely not to type semicolons.

AI Firewalls: The First Real Defense Layer
#

Traditional firewalls inspect network traffic for malicious patterns. AI firewalls do the same for prompts and outputs. They’re not perfect, but they’re necessary.

An AI firewall sits between your users and your LLM, analyzing inputs and outputs for injection attempts, data leakage, and policy violations. Think of it as your WAF (Web Application Firewall) equivalent for AI systems.

What good AI firewalls detect:

Known injection patterns (both direct and indirect)
Attempts to extract system prompts or bypass guardrails
Suspicious output patterns that suggest compromised responses
PII or sensitive data leakage in outputs
Unusual token patterns that don’t match legitimate queries

Companies like Lakera, Robust Intelligence, and Promptarmor are building commercial solutions. Open-source options like LLM Guard and NeMo Guardrails give you more control but require more expertise.

The catch: AI firewalls add latency (typically 50-200ms per request) and cost (you’re running additional inference). They also have false positives. Your customer support bot might flag legitimate technical questions as injection attempts.

This is where trade-offs start mattering. For high-risk applications (financial transactions, healthcare, code generation), the overhead is worth it. For low-risk use cases (general knowledge chatbots), maybe not.

Dual LLM Architecture: The Evaluator Pattern
#

Here’s a pattern that’s gaining traction: use one LLM to evaluate the safety of requests before they reach your main system.

The flow looks like this:

User submits input
Evaluator LLM analyzes: “Is this a legitimate query or an injection attempt?”
If safe, proceed to main LLM
Main LLM generates response
Evaluator LLM checks output: “Does this response follow policies?”
If clean, return to user

Why this works better than simple filtering: LLMs are actually quite good at detecting adversarial inputs when that’s their only job. By dedicating a model specifically to security evaluation, you get better accuracy than trying to bolt security onto your main workflow.

Why this isn’t a silver bullet: The evaluator LLM can be attacked too. Researchers have shown that with enough effort, you can craft prompts that fool the evaluator while still injecting malicious instructions into the main system. It’s defense in depth, not a complete solution.

Real-world implementation: Use a smaller, faster model for evaluation (GPT-4o-mini, Claude Haiku) and your primary model for generation. This keeps latency reasonable while adding a meaningful security layer.

Zero-Trust Principles for LLM Applications
#

The most important architectural shift is applying zero-trust principles to AI systems. Every output is untrusted until proven safe. Every action requires explicit authorization.

Implement least-privilege access aggressively. Your chatbot doesn’t need write access to your production database. Your code completion tool doesn’t need network access. Your document summarizer doesn’t need the ability to send emails.

When you do grant permissions, scope them narrowly:

Read-only access to specific tables, not entire databases
Ability to create draft emails, not send them automatically
Access to public documentation, not internal source code

Require human approval for high-stakes actions. If your AI system wants to process a refund over $500, issue a database migration, or modify production configuration, it should create a request for human review, not execute directly.

This is actually where AI systems have an advantage over traditional applications. Users expect a conversation. “I’ve drafted this refund for $750. Would you like me to submit it for approval?” feels natural. Use that to your advantage.

Output Sanitization and Monitoring
#

You can’t catch everything at the input layer, so you need robust output controls.

Content filtering should check for:

Leaked system prompts or internal instructions
PII or credentials that shouldn’t be in responses
Malicious content (phishing links, social engineering)
Off-policy responses (your customer support bot shouldn’t be giving medical advice)

Anomaly detection is where things get interesting. Build baselines for normal behavior:

Typical response length and complexity
Expected data access patterns
Common phrasing and tone
Frequency of certain operations

When you see deviations (responses that are suddenly much longer, accessing unusual data combinations, or using phrases that don’t match your trained patterns), flag them for review.

The implementation challenge: Building good anomaly detection requires instrumentation from day one. You need to log everything: prompts, responses, data accessed, operations attempted, confidence scores. Most teams don’t think about this until after an incident.

Start logging now. Future you will thank present you.

The Tool Use Problem
#

Here’s where it gets really interesting. Modern AI systems don’t just answer questions; they use tools. They query databases, call APIs, execute code, interact with other systems.

Each tool is an attack vector. If an attacker can inject instructions that cause your AI to use tools maliciously, they’ve achieved something close to remote code execution.

The defense: Implement tool use policies at the orchestration layer, not in the prompt.

Instead of telling your LLM “you can use the database tool to look up customer records,” implement it in code:

def can_use_tool(tool_name, parameters, context):
 if tool_name == "database_query":
 # Enforce read-only
 if "INSERT" in parameters.query.upper():
 return False
 # Enforce scope
 if context.user_role != "support" and "customer_data" in parameters.table:
 return False
 return True

Your orchestration layer validates every tool call before execution. The LLM can request actions, but your code decides what’s allowed.

The Real Talk: Trade-offs Nobody Mentions
#

Every security control has costs. Let’s be honest about them:

Latency: AI firewalls, dual LLM evaluation, output filtering all add 50-200ms. Stack them together and you’re adding seconds to response times. For real-time applications, this might be unacceptable.

False positives: Aggressive filtering catches legitimate queries. Your technical users will be frustrated when their debugging questions get flagged as injection attempts. Your security team and product team will argue about where to set thresholds.

Cost: Every evaluation layer is additional inference. If you’re processing millions of requests, the costs add up fast. A dual LLM architecture with output filtering can easily 3x your inference costs.

Complexity: More security layers mean more failure modes. What happens when your AI firewall goes down? Do you fail open (risky) or fail closed (customer impact)? These aren’t theoretical questions; you need answers before production.

The practical approach: Start with structured prompts and least-privilege access. These are low-cost, high-value changes. Add AI firewalls for high-risk operations. Implement dual LLM evaluation where the stakes justify the cost. Build monitoring and anomaly detection from day one.

Don’t try to implement everything at once. You’ll slow down your team and create a system so complex that security controls become the thing that breaks.

What’s Working in Production
#

After investing countless hours researching and experimenting with AI security, both theoretically and hands-on in production environments, here’s the architecture that actually works:

Layer 1: Input validation - Structured prompts, basic pattern matching, rate limiting

Layer 2: Execution control - Least-privilege tool access, operation allowlists, human approval workflows

Layer 3: Output verification - Content filtering, PII detection, policy compliance checks

Layer 4: Monitoring - Logging, anomaly detection, audit trails, incident response playbooks

Notice what’s missing: attempts to make the LLM itself secure. That’s not how this works. The LLM is a powerful but fundamentally untrustworthy component. Your architecture assumes it can be compromised and builds controls around it.

It’s the same philosophy we use for traditional applications: don’t trust user input, validate at boundaries, enforce least privilege, assume breach.

What Engineering Leaders Should Focus On
#

If you’re responsible for AI security, here’s your practical checklist:

This week: Audit your current AI systems. What data can they access? What actions can they take? Where are you mixing trusted and untrusted data?

This month: Implement structured prompts and least-privilege access. These are table stakes and should be non-negotiable.

This quarter: Add monitoring and anomaly detection. You need visibility before you can respond to incidents.

This year: Build tool use policies, implement human approval workflows for high-stakes operations, and establish incident response procedures.

Don’t wait for perfect solutions. The organizations getting this right aren’t the ones with the fanciest technology; they’re the ones who started early and iterated based on real-world experience.

What’s Coming Next
#

Defensive architectures are maturing fast. We’re seeing:

Better frameworks that enforce security by default
Standardized APIs for AI firewalls and evaluation
Industry benchmarks for measuring AI security effectiveness
Compliance frameworks that mandate specific controls

But here’s what nobody’s talking about: all of these defenses assume you control your infrastructure. What happens when the vulnerability isn’t in your code, but in the pre-trained model you downloaded? The prompt template you copied from GitHub? The RAG knowledge base you inherited from the previous team?

In the next part of this series, we’ll explore the AI supply chain: the attack vector that most teams don’t even know exists. Because the biggest security risk might not be in what you build, but in what you’re building on top of.

AI's Dual Edge: When to Disrupt and When to Compound

Wed, 08 Oct 2025 00:00:00 +0000

The CEO just announced an “AI transformation” in the all-hands.

Your board wants to know your AI strategy. Product is pitching AI features for every roadmap. And you’re the one who has to turn vague executive enthusiasm into actual engineering work that creates value.

Here’s the decision you’re actually making: AI has two fundamentally different plays, and they require different resource allocation, different timelines, and different organizational commitment.

You can disrupt: fundamentally rewrite the economics of something, change what’s possible. Or you can augment: make existing systems measurably better without rebuilding them.

Disruption sounds impressive in board decks. Augmentation sounds boring. But picking wrong costs you six months of engineering time, burns team morale, and kills your credibility when you have nothing to show for it.

The question isn’t “should we do AI?” It’s “which play can we actually execute with the team, timeline, and organizational support we have right now?”

Most engineering leaders default to disruption because it’s what executives want to hear. The reality is that augmentation is usually the better play: faster to value, lower risk, and it builds organizational muscle for bigger bets later.

Disruption: When You’re Changing the Game (And What It Actually Costs)
#

Disruption isn’t about being radical for board slides. It’s about fundamentally changing what’s economically viable: making something possible that wasn’t before, or making something so much cheaper that it changes behavior.

What real disruption looks like:

Tesla’s FSD learns from every mile driven by every car. They ship updates weekly because the fleet is the training ground. Hardware companies used to iterate in 3-year cycles. Their AI stack iterates in 3-week cycles. That’s not an improvement. It’s a different game.

Retail demand forecasting used to mean: forecast six months out, order inventory, pray you got it right, discount what you got wrong. Short-horizon AI forecasting turns that into a control system. Inventory, labor, and pricing adjust daily based on what’s actually happening. Companies doing this aren’t just reducing stockouts. They’re changing their cost of capital and margin structure.

Drug discovery used to mean brute-forcing millions of combinations. AI narrows the search space dramatically, eliminating 95% of dead ends before anyone wastes time and money on them.

Here’s what nobody tells engineering leaders about disruption:

It’s expensive, slow, and organizationally risky. You need:

12–18 month runway. Not “we’ll pilot it for a quarter.” Real disruption takes multiple iterations to get right. Your exec team needs to understand this is a long bet.

Dedicated team capacity. You can’t do this with 20% of someone’s time or as a side project. You need engineers who can focus without getting pulled into production fires every week.

Robust instrumentation from day one. You need to measure what’s actually happening in production, not what you hope is happening. Shadow mode, A/B testing infrastructure, automated rollback.

Executive air cover. When this takes longer than expected (it will), or when early results are mixed (they will be), someone senior needs to protect the team from getting cancelled.

Risk tolerance. Data will be brittle. Regulators might have opinions. Users might not trust it initially. These aren’t edge cases. They’re the entire problem space.

The question you need to answer honestly: Do we actually have these things, or are we pretending we do because the CEO is excited about AI?

If you don’t have this organizational support, you’re not doing disruption. You’re setting your team up for a science project that gets cancelled in Q3 when it hasn’t shipped yet.

Augmentation: Where Most Engineering Leaders Should Start
#

This is the play most teams should run first. Not because it’s less ambitious, but because it compounds faster, fails cheaper, and builds organizational credibility for bigger bets.

Augmentation means: take what you’re already doing, make it measurably better with AI, repeat. You’re not rebuilding the system. You’re making existing operations perform at a higher level.

What this looks like in practice:

Your warehouse operations team guesses where to put high-velocity items. You add AI slotting optimization. Labor costs drop 15%, on-time delivery improves 10%. Same warehouse, same people, better math. Engineering investment: 2-3 engineers for 8 weeks.

Your support team bounces tickets between departments until someone knows the answer. You add AI triage that routes correctly the first time. First-contact resolution goes up. Handle time goes down. Same team handles more volume with less frustration. Engineering investment: 1 team lead + 2 engineers for a quarter.

Your factory maintenance team discovers equipment failures when production stops. You add predictive maintenance that gives 48 hours warning. Unplanned downtime craters. You schedule repairs during planned maintenance windows. OEE improves without adding headcount. Engineering investment: 1 senior engineer + 1 ML engineer for 12 weeks.

Your fraud detection flags 1,000 transactions for manual review (970 are false positives). You improve risk scoring with AI. Manual review team focuses on actual problems. You catch more fraud with less work. Engineering investment: 2 engineers for 6 weeks.

None of this is revolutionary. All of it creates measurable value.

The business case is straightforward: improve a process by 10–15%, replicate across 20 facilities or 50 teams, create millions in value without changing your fundamental business model.

The leadership advantage: if it doesn’t work, you turn it off. Your rollback plan is “go back to how we did it last month.” You haven’t burned 18 months rewriting core systems. Your team learned something. Your organizational credibility is intact. You’re ready to try the next thing.

The Engineering Leader’s Playbook
#

Here’s what separates AI projects that succeed from ones that become expensive lessons. This isn’t theory. It’s what you need to set up and enforce to ship value.

Force Clarity on Metrics Before You Allocate Headcount
#

Before you assign engineers, demand this: What are we measuring, and what’s the current baseline?

Pick three use cases maximum. Each gets exactly two metrics:

Demand forecasting → Mean Absolute Percentage Error (MAPE) ↓, stockouts ↓
Fulfillment → cost per order ↓, on-time delivery ↑
Support → first-contact resolution ↑, handle time ↓

If your team can’t define success this precisely, don’t start. You’ll burn engineering capacity building something technically impressive that nobody can prove is working.

This is also how you protect your team from scope creep. When product comes back with “let’s add AI to five more things,” you point at the three use cases you committed to. Nail those first. Prove they work. Then, and only then, expand.

Teams that try to do ten AI initiatives simultaneously ship zero things that create value. Your job is to say no until the first three are in production and measured.

Set Default Architecture Standards (And Enforce Them)
#

Your team will want to overcomplicate this. They read papers, get excited about fine-tuning and agentic systems, and skip the boring foundations that actually ship.

Set this as the default path for 90% of use cases:

Start with RAG. Retrieval-Augmented Generation gets good results fast. The model pulls relevant context, then generates answers based on that context. Tell your team: make retrieval great and evals solid before touching anything fancier.

Fine-tune only when proven necessary. RAG solves most problems. Only let teams fine-tune when they’ve proven RAG can’t work and identified specific, consistent gaps. Fine-tuning is expensive, brittle, and requires maintaining training pipelines. Make them write a decision doc explaining why simpler approaches won’t work.

Agents require approval. Tool use and autonomous behavior are powerful, but they need rock-solid evals, guardrails, and failure handling. Don’t let teams build agents until they’ve proven they can ship and maintain production RAG systems.

Why this matters as a leader: Teams that skip straight to fine-tuning and agents because it sounds impressive waste six months debugging before admitting they should have started simpler. Meanwhile, teams that follow the standard path are in production after 8 weeks, collecting user feedback, and iterating based on real usage.

Your job is to protect your team from their own over-enthusiasm. Set the standard. Make exceptions require written justification.

Make Evals Non-Negotiable Infrastructure
#

Here’s what you need to enforce: No AI system goes to production without automated evaluation. Period.

Without evals, your team is flying blind. They don’t know if prompt changes improve things or break them. They don’t know if performance is degrading. They’re operating on vibes and anecdotes, and that’s how you end up with production incidents at 3am.

Mandate these measurements for every AI system:

Task success rate. Can it actually do the job? Your team defines what “success” means for each use case and measures it automatically. No handwaving.

Harmful/false output rate. How often does it hallucinate? How often does it generate something actively wrong or dangerous? This number needs to go in your operational dashboard.

Latency budget. Set it based on user expectations, not engineering wishful thinking. A perfect answer that takes 30 seconds is useless if users expect 2 seconds.

Drift detection. Is performance degrading over time as data or user behavior changes? Automated alerts when things slide.

Adversarial testing. Prompt injection, jailbreaks, data exfiltration attempts. These aren’t one-time tests. Make them part of CI/CD.

Enforce a deployment process that assumes failure:

Shadow mode → compare AI output to current system without user exposure
Canary → 5–10% of traffic
Staged rollout → gradual expansion with metric monitoring
Automated rollback → one command to revert

If your team can’t roll back in minutes, don’t let them ship. “Hope nothing breaks” isn’t an operational strategy.

Your role: Make evals part of definition-of-done. No PR merged, no deployment approved, until automated evaluation exists and passes.

Budget for Data Quality Like You Budget for Security
#

The engineering leaders winning with AI aren’t the ones with the fanciest models. They’re the ones who allocated engineering time to data infrastructure.

Your AI is only as good as your data. If your critical tables are stale or wrong, your AI will be confidently incorrect. Unlike traditional software where bad data causes visible errors, AI with bad data generates plausible-sounding nonsense. Users trust it, act on it, and then you discover the problem three weeks later when decisions were based on garbage.

What you need to mandate and fund:

Automated freshness and accuracy checks. If inventory data should update hourly and hasn’t updated in six hours, automated alerts fire before your AI starts making predictions based on stale state. This requires ongoing engineering time.

Feature stores and lineage. When AI goes wrong (it will), your team needs to trace it back. Where did this feature come from? How was it computed? When was it last updated? Without lineage, debugging takes days instead of hours. Budget for building this.

Privacy boundaries as architecture. PII redaction, consent management, access controls. These need to be architectural decisions from day one, not patches you add when legal asks questions or customers complain.

The mistake most leaders make: treating data quality as a one-time project. “We’ll clean it up in Q1, then focus on AI in Q2.”

That’s not how this works. Data quality is continuous infrastructure work like security or performance monitoring. If you don’t budget ongoing engineering time for it, your AI systems degrade slowly until they’re generating nonsense and nobody knows why.

Allocate 20-30% of your AI engineering capacity to data infrastructure. Yes, that feels like a lot. No, you can’t skip it and succeed.

Instrument Cost Tracking from Day One
#

Set up cost instrumentation before your team ships anything. You need to see problems before they show up on the bill.

Track unit cost per task, not cost per token. Tokens are implementation details. What matters to your P&L: how much does it cost to process one customer inquiry? Generate one forecast? Triage one ticket? Make your team instrument this.

Set budget caps per service with automated alerts. If your support bot suddenly makes 10x more API calls because someone changed a prompt, you want alerts firing immediately, not a surprise $50K bill at month-end.

Default to “good enough” models with justification required for upgrades. Most tasks don’t need GPT-5. They need consistent, fast, correct answers at reasonable cost. Smaller models deliver that for 10% of the cost. Make your team write a doc explaining why they need expensive models before approving it.

Why this matters: AI costs scale with usage in ways traditional infrastructure doesn’t. A prompt change can 10x your API costs overnight. Without instrumentation, you discover this when finance asks why cloud spend jumped 300% last month.

Set Security Policies Early
#

Organizations that treat AI security like web security from 2005 learn through expensive incidents. Don’t be one of them.

Mandate isolation for untrusted tools. If your AI can call APIs or access systems, require sandboxing and signed function calls. Don’t let teams assume models will only do what they want. Make them plan for unexpected behavior.

Require output filtering for sensitive data. If AI works with PII, PHI, or confidential information, mandate automated checks that verify sensitive data doesn’t leak through responses. Trust but verify.

Include models in post-incident reviews. When things break, your team needs to trace through code, data, and model behavior. “The AI did something weird” isn’t a root cause. Make them explain why it behaved that way.

Assume hostile users from day one. Users will try to jailbreak your system. They’ll attempt prompt injection. They’ll try to extract training data. Make adversarial testing part of your standard release process, not something you add after an incident.

What to Demand from Your Executive Leadership
#

If you’re an engineering leader trying to get organizational support for doing AI right, here’s what you need from the C-suite. Don’t assume they understand this—educate them.

They need to ask for metrics, not demos. Train your CEO to say “show me the before/after chart” instead of “show me the demo.” Demos prove nothing. Metrics prove value.

They need to enforce constraints. When the CEO says “add AI to everything,” your job is to push back: “We’re committing to three use cases. We’ll nail those, prove they work, then expand.” Get executive support for saying no to scope creep.

They need to protect measurement windows. AI projects need time to collect data and iterate. When the board wants to see progress every week, your CEO needs to explain that AI isn’t like shipping features. It requires measurement cycles. Get them to buy you that time.

They need to understand build vs. buy. Most AI infrastructure is undifferentiated. Default to buying foundation models and tooling. Build only where you control the workflow and the data improves by being used. Make sure your CFO understands why you’re spending $50K/month on API calls instead of building custom models.

They need to tie incentives to adoption and impact, not shipped features. Shipping AI features is easy. Making them create measurable value is hard. Make sure compensation and promotions reward outcomes, not output.

If you can’t get this from executive leadership: Your job is harder but not impossible. Set these expectations yourself through data. Track baselines religiously. Publish metrics that show real impact. Kill things that don’t work publicly. Build your credibility through measured results, then use that credibility to demand better organizational support.

A 90-Day Plan for Engineering Leaders
#

Here’s a realistic timeline that assumes you have normal organizational constraints: technical debt, competing priorities, and a team that’s already fully loaded. Adjust based on your capacity.

Week 0–2: Define Success and Get Organizational Alignment
#

Pick 3 use cases maximum. Document the success metrics and measure current baseline. Get exec buy-in on these metrics (they become your definition of success). If you can’t measure it today, you can’t prove AI improved it later.

Assign one engineer to stand up a basic evaluation harness. Start simple: a script that runs AI on test cases and validates outputs.

Have your data engineering team add quality checks to tables that feed these use cases. You need automated alerts when input data goes stale or wrong.

Organizational work: Get your CEO/CFO to agree that these three use cases are the commitment for the quarter. Push back on new requests until you deliver these.

Week 3–6: Ship v1 in Shadow Mode
#

Allocate 2-3 engineers to build v1. Put it behind a feature flag. Run in shadow mode (processes real traffic, users don’t see output). Compare AI decisions to what your current system does.

Have one engineer instrument cost tracking per task. Set budget caps with automated alerts.

Run red-team exercises. Assign someone to try breaking it. Fix the top five issues.

Organizational work: Weekly metrics review with exec team. Show shadow mode results. Manage expectations: this is data collection, not feature launches.

Week 7–10: Canary to Real Users (Finally)
#

Route 5–10% of traffic to the AI system. Monitor metrics obsessively. Is it actually better than baseline from week 1?

Run table-top incident exercises with your ops team. Practice rollback procedures. Make sure everyone knows how to revert quickly if needed.

Make a hard decision: Look at your three use cases. Kill the weakest one. Reallocate that team capacity to double down on the strongest performer.

Organizational work: Present early results to exec team. Explain why you killed one project. Frame it as disciplined resource allocation, not failure.

Week 11–13: Scale What Works, Stop What Doesn’t
#

Increase traffic to 25–50%. Publish before/after charts showing real business impact: cost reduction, quality improvement, time savings. Whatever metrics you committed to in week 0.

If you have appetite for risk and spare capacity, move one agentic capability (tool use, function calling) into a low-risk workflow with human approval required for every action.

Refresh your backlog. Add one new use case only if you’ve proven the others work and have team capacity. Don’t accumulate half-finished AI projects that drain morale.

Organizational work: Deliver a quarterly retrospective to leadership. What worked, what didn’t, what you learned. Set expectations for next quarter based on demonstrated capacity, not aspirations.

Anti-Patterns Engineering Leaders Need to Kill
#

If you see these in your organization, stop the work and fix them. These are the warning signs of projects that will fail expensively.

“Our AI is 90% accurate!” Ask: 90% of what? Measured how? Against what baseline? Compared to what existing system? If your team can’t answer precisely, they’re not measuring. They’re guessing. Don’t let them continue without proper evaluation.

Prompts managed in Notion, Slack, or tribal knowledge. If prompts aren’t in version control with regression tests, they will drift. Someone will make a “small change” that breaks production, and your team won’t know what changed or how to roll back. Mandate version control for prompts like you mandate it for code.

“We’ll clean the data after we ship the feature.” This never happens. Your team will ship with dirty data, get weird results, spend weeks debugging, and trace it back to data quality issues they identified in week 1 but deprioritized. Make data quality a prerequisite, not a nice-to-have.

Building agents before mastering basic RAG. If your team can’t reliably retrieve the right document and generate a good answer with basic RAG, don’t let them add autonomy and tool use. It doesn’t make failures better. It makes them more expensive and harder to debug.

Quarterly demos with unchanged metrics. If your teams demo AI features every quarter but unit costs, cycle times, and error rates haven’t moved, they’re building demos, not products. Metrics are reality. Demos are theater. Shut down projects that can’t show measurable business impact.

What Success Looks Like for Engineering Leaders
#

The gap between “we’re doing AI” and “we’re getting measurable value from AI” isn’t technology or budget. It’s leadership discipline.

The organizations winning aren’t the ones with the biggest AI teams or the fanciest models. They’re the ones whose engineering leaders:

Force outcome clarity before allocating resources. They know exactly what they’re optimizing for before assigning engineers. No vague mandates, no “we’ll figure it out as we go.”

Build boring infrastructure first. Data quality checks, evaluation harnesses, cost tracking, rollback mechanisms. The unglamorous work that doesn’t make good board slides but determines whether you succeed in production.

Measure and publish honestly. Before/after charts with real baselines. When something doesn’t work, they say so publicly. When something works, they have numbers to prove it.

Kill things decisively. They’re as comfortable shutting down failed experiments as launching new ones. They frame it as disciplined resource allocation, not failure.

Protect their teams from organizational chaos. They push back on scope creep. They demand measurement windows. They buffer their engineers from executive enthusiasm that would otherwise destroy focus.

This isn’t science fiction or research. It’s practical systems thinking applied to a new capability.

Warehouses that stop guessing where to put inventory. Support teams that route correctly the first time. Maintenance teams that fix things before they break. All of it measurable. All of it replicable. All of it built by engineering leaders who understood the difference between disruption and augmentation, picked the right play for their organization, and executed with discipline.

Your CEO wants AI transformation. Your board wants competitive advantage. Your job is to deliver measurable business impact while protecting your team’s capacity for the work that actually matters.

Pick your play. Set your constraints. Allocate deliberately. Measure obsessively. Kill ruthlessly. Scale what works.

That’s how you turn executive enthusiasm for AI into lasting organizational value.

The Context Problem: Why AI Can't Remember You Across Apps (And Why That's Not an Accident)

Mon, 29 Sep 2025 00:00:00 +0000

You just spent 20 minutes teaching Claude your codebase. The mental model is perfect. Claude gets the architecture, knows your constraints, understands the goal.

Then you remember ChatGPT is better at Python refactoring. You switch over.

“Let me explain my project again…”

Stop. Before you paste that context for the hundredth time, let’s talk about what’s really happening here. Not the surface-level “AIs don’t share memory” explanation. The real engineering. The deliberate decisions. The philosophy of what context even means.

Because once you understand how AI memory actually works, you’ll see why this problem exists, and why it might never be “solved” the way you think.

The 10-minute mental model
#

Let’s build your understanding from first principles. Here’s what “context” actually means in AI systems:

1. Context is attention, literally
#

When you talk to an AI, your words become tokens (numerical representations). These tokens flow through attention mechanisms that decide what’s relevant. Context isn’t “stored” like files on disk. It’s a temporary computational state, like RAM, not a hard drive.

Every token costs compute. A 200K context window means the model is actively attending to 200,000 tokens worth of patterns every single time it generates a response. That’s why context is expensive. It’s not storage cost, it’s processing cost.

2. Memory is retrieval, not recording
#

When ChatGPT “remembers” you prefer React, it’s not writing to a database. It’s creating embeddings (mathematical fingerprints of concepts) and storing those in a vector space. Next conversation, it searches that space for relevant patterns and injects them into the context.

Think of it like this: The AI doesn’t remember conversations. It remembers the shape of conversations and reconstructs relevant bits on demand.

3. Sessions are stateless by design
#

Here’s the kicker: Large language models are fundamentally stateless. They’re functions: text in, text out. No persistence. Every “memory” feature is scaffolding built around this stateless core.

Why? Because stateless is scalable. One model can serve millions of users simultaneously. Add state, and suddenly you need persistent storage, session management, consistency guarantees. The infrastructure complexity explodes.

Why context doesn’t transfer (and never will)
#

Here’s where it gets interesting. The context problem isn’t technical. It’s architectural, economic, and philosophical:

The embedding incompatibility problem
#

Each AI uses different embedding models. Claude’s vector representation of “Python” differs from ChatGPT’s differs from Grok’s. Even if they shared raw text, the semantic understanding wouldn’t translate. It’s like trying to share thoughts between brains with different neural structures.

The context window economics
#

A 200K context window at current prices costs about $2-4 per conversation in compute. Multiply by millions of users. Now imagine maintaining that context across sessions, across platforms. The economics don’t work unless someone’s paying (either users directly or through lock-in).

The competitive moat reality
#

Let’s be honest: If Claude context seamlessly transferred to ChatGPT, why would you pay for both? Context lock-in is the subscription retention strategy. Every AI provider knows this. Interoperability is antithetical to their business model.

The philosophical divide
#

Here’s the deep question: What even is context? Is it the raw text? The extracted meanings? The interaction patterns? Each AI platform has a different answer, and those answers are incompatible by design. They’re not just building different features. They’re building different theories of mind.

How the big three actually implement memory
#

Now that you understand the primitives, let’s see how each platform builds “memory” on top of stateless models:

Claude: Structured context hierarchies
#

Anthropic took the “explicit is better than implicit” approach:

Projects as context containers. A Project isn’t just a folder. It’s a persistent context namespace. Documents get chunked, embedded, and indexed. When you chat, Claude runs semantic search across project contents and injects relevant chunks into the prompt. It’s RAG (Retrieval Augmented Generation) with a nice UI.

Artifacts as working memory. These aren’t just displayed code. They’re part of the active context. Claude maintains a pointer to artifact state and includes it in subsequent prompts. Close the browser, lose the pointer.

Constitutional memory. Claude uses constitutional AI principles even for memory. It won’t remember things it shouldn’t (passwords, PII) even if you ask. The memory system has built-in ethical constraints.

The philosophy: Claude treats context like a research assistant would. Organized, hierarchical, and bounded. It’s memory as a filing system, not a stream of consciousness.

ChatGPT: Implicit extraction and injection
#

OpenAI went for “it just works”:

Automatic memory extraction. After each conversation, ChatGPT runs a secondary pass to extract “memorable” facts. These get stored as embeddings with metadata (timestamp, confidence, topic). No user action required.

Probabilistic injection. New conversations trigger similarity searches across your memory bank. High-scoring memories get prepended to your prompt invisibly. You never see this happening. It’s seamless.

Cross-session state. ChatGPT maintains a persistent user profile that evolves. It’s not just remembering facts; it’s building a model of you. Your writing style, reasoning patterns, preferences. All get encoded.

The philosophy: Memory should be invisible and automatic. The AI adapts to you, not the other way around. It’s memory as personality modeling.

Grok: Stream processing and real-time context
#

xAI took the “everything is a stream” approach:

Event-sourced memory. Grok treats conversations as event streams. Each message is an event that updates the state. Memory is the accumulated state changes over time, allowing for precise replay and branching.

Real-time context injection. The X integration isn’t just API calls. It’s streaming context. Grok maintains a sliding window of relevant real-time data that gets mixed with conversational context. It’s the only one doing true stream processing.

Pattern learning over storage. Grok emphasizes learning interaction patterns over storing facts. It’s less “remembers you like Python” and more “adapts to your communication style.”

The philosophy: Context is fluid and temporal. What matters isn’t what was said, but how it relates to what’s happening now. It’s memory as stream processing.

The architectural escape routes
#

Despite the challenges, here are four ways the context problem could be solved. Each with profound implications:

Architecture 1: The Semantic Intermediary
#

Instead of sharing context directly, share semantic representations:

User Context Space
├─ Universal embeddings (model-agnostic)
├─ Semantic graph (relationships)
├─ Intent vectors (what you're trying to do)
└─ Interaction patterns (how you communicate)

How it works: A middle layer that translates between AI-specific representations. Like Unicode for meaning. A universal encoding that each AI can interpret.

Why it’s hard: Requires agreement on semantic primitives. It’s like asking English, Mandarin, and Arabic speakers to agree on universal grammar.

What it would enable: True AI interoperability. Switch models mid-conversation. Use multiple AIs simultaneously on the same problem.

Architecture 2: Federated Context Protocol
#

Borrowed from federated learning:

Context Federation
├─ Local context store (your device)
├─ Encrypted sync protocol
├─ Differential privacy layer
└─ Model-specific adapters

How it works: Your context lives on your device. AIs request relevant portions through a privacy-preserving protocol. You control what’s shared, when, and with whom.

Why it’s powerful: Solves privacy, ownership, and portability simultaneously. Your context becomes a personal asset, not platform property.

The catch: Requires fundamental changes to how AI services work. They’d have to give up data control.

Architecture 3: Context as Computation
#

The radical approach. Don’t store context, compute it:

Generative Context System
├─ Base facts (minimal storage)
├─ Generative rules (how to reconstruct)
├─ Verification hashes
└─ Incremental updates

How it works: Store only essential facts and rules for regenerating context. Like seed-based procedural generation in games. Each AI reconstructs the full context from seeds.

Why it’s elegant: Tiny storage footprint. Perfect consistency. Context can evolve without storing every state.

The challenge: Requires deterministic generation across different models. We’re nowhere close to this.

Architecture 4: The Model Context Protocol (MCP)
#

The standard that actually exists:

Anthropic created MCP to standardize AI-to-data connections. Cursor just shipped deeplinks for MCP. Click a link, install a context server. But here’s the thing:

What MCP actually does: Standardizes how AIs connect to data sources (databases, APIs, documents). It’s plumbing, not memory.

What MCP doesn’t do: Share context between different AI platforms. It’s a connection protocol, not an interchange format.

The reality: MCP is useful but orthogonal to the context problem. It’s like having standardized power outlets but different voltages.

What actually works today (ranked by effectiveness)
#

Forget the future. Here’s how to minimize context pain right now:

Level 1: The Context Discipline
#

Build a system, stick to it:

# CONTEXT.md
## Mental Model
[How I think about this problem]

## Decisions Made
[What we've already figured out]

## Current Focus
[What we're working on now]

## Constraints
[What we can't change]

Update after every session. Start every conversation by pasting this. It’s manual but it works.

Level 2: Context Bridges
#

Tools exist that sync context across AIs:

Browser extensions that capture and replay context
Note-taking tools that become context hubs
Automation platforms that chain AI calls with context

They’re imperfect but better than copy-paste.

Level 3: Single-Tool Mastery
#

The nuclear option: Pick one AI and commit. Learn its memory system deeply. Use its features fully. Let compound context work for you.

Choose based on your primary need:

Deep work: Claude with Projects
Continuous assistance: ChatGPT with Memory
Real-time research: Grok with streaming context

Level 4: Context as Code
#

For developers, the ultimate solution:

class ContextManager:
 def __init__(self):
 self.embeddings = VectorStore()
 self.sessions = {}
 self.memory = PersistentDict()

 def capture(self, ai, conversation):
 # Extract and store semantic patterns
 patterns = self.extract_patterns(conversation)
 self.embeddings.add(patterns)

 def inject(self, ai, prompt):
 # Retrieve and prepend relevant context
 context = self.embeddings.search(prompt)
 return f"{context}\n\n{prompt}"

Build your own context layer. Control everything. It’s work, but you’ll never lose context again.

The next 12 months: Watch these signals
#

The MCP test: If Cursor’s MCP deeplinks gain adoption, context sharing becomes inevitable. If they don’t, we’re stuck with silos.

The memory tax: When someone figures out how to monetize context portability, everything changes. Watch for “context as a service” startups.

The regulation forcing function: GDPR-style rules for AI memory are coming. Portable context might become legally required.

The open source wildcard: One good open source context protocol could force everyone’s hand. The community is building alternatives.

The uncomfortable truth about memory
#

Here’s what this all reveals: We’re trying to solve a human problem with a technical solution.

The context problem exists because we’re using AIs wrong. We treat them like persistent assistants when they’re actually stateless functions. We expect them to remember like humans when they’re designed to compute like calculators.

Maybe the answer isn’t better memory. Maybe it’s better prompting. Better task decomposition. Better understanding of when context helps and when it hurts.

Because here’s the thing: Perfect memory might make AI worse, not better.

Fresh context forces clearer thinking. Explaining again reveals new angles. Starting over prevents assumption lock-in. The context “problem” might actually be a feature.

Your move
#

The context problem isn’t going away. But you don’t have to be its victim:

Build a context discipline today. Simple markdown files beat no system.
Experiment with bridges. Try the tools, see what works.
Question the premise. Do you really need perfect memory? Or better workflows?
Think philosophically. What is context? What is memory? What are you really trying to preserve?

The magic isn’t in perfect memory. It’s in understanding what memory means for intelligence.

And maybe, just maybe, the fact that Claude and ChatGPT can’t share notes isn’t a bug.

It’s a glimpse of how alien artificial intelligence really is.

When someone asks why we don’t have AGI yet, tell them we can’t even agree on what memory means. Then watch them try to define it.

Architecture · PiniShv

Open WebUI Isn't a ChatGPT Clone. It's AI Infrastructure.

What Open WebUI actually is #

What it can actually do (beyond chat) #

The reality of running it in production #

Who’s behind it and why that matters #

The security conversation nobody wants to have #

Why teams and orgs actually adopt this #

How it compares to the incumbents #

How it compares to open-source alternatives #

What this actually means for engineering leaders #

Who should care about this #

Your AI Agents Are Flying Blind. Here's How to Fix That.

Two teams, same agents, wildly different results #

Where knowledge actually lives (and why that’s broken) #

The wrong way to fix this #

What actually works: a developer knowledge hub #

Git for authoring #

Semantic search for retrieval #

MCP for delivery #

The loop that makes it compound #

The AGENTS.md safety net #

Why nothing on the market solves this #

How to start (without boiling the ocean) #

The uncomfortable question #

Building AI Systems That Don't Break Under Attack

The Foundation: Structured Prompts and Separation of Concerns #

AI Firewalls: The First Real Defense Layer #

Dual LLM Architecture: The Evaluator Pattern #

Zero-Trust Principles for LLM Applications #

Output Sanitization and Monitoring #

The Tool Use Problem #

The Real Talk: Trade-offs Nobody Mentions #

What’s Working in Production #

What Engineering Leaders Should Focus On #

What’s Coming Next #

AI's Dual Edge: When to Disrupt and When to Compound

Disruption: When You’re Changing the Game (And What It Actually Costs) #

Augmentation: Where Most Engineering Leaders Should Start #

The Engineering Leader’s Playbook #

Force Clarity on Metrics Before You Allocate Headcount #

Set Default Architecture Standards (And Enforce Them) #

Make Evals Non-Negotiable Infrastructure #

Budget for Data Quality Like You Budget for Security #

Instrument Cost Tracking from Day One #

Set Security Policies Early #

What to Demand from Your Executive Leadership #

A 90-Day Plan for Engineering Leaders #

Week 0–2: Define Success and Get Organizational Alignment #

Week 3–6: Ship v1 in Shadow Mode #

Week 7–10: Canary to Real Users (Finally) #

Week 11–13: Scale What Works, Stop What Doesn’t #

Anti-Patterns Engineering Leaders Need to Kill #

What Success Looks Like for Engineering Leaders #