A year or two ago, most enterprise AI architecture conversations started and ended with one question:
"Which model should we use?"
Today, that is one of the least interesting questions. Models are improving quickly and becoming easier to swap. The real architecture challenge is everything around the model: the data it can see, the tools it can use, the controls around it, and how it fits into existing systems.
The model is a component. AI architecture is the system around it.
A Reference Stack
When I help organizations think about AI architecture, I use a simple layered view.
Experiences
Where people actually meet AI: assistants in the CRM, copilots in productivity tools, chat interfaces for customers, or agents working in the background. The best AI experiences live inside existing workflows rather than in a separate tool nobody opens.
Orchestration and agents
The logic that turns a model into something useful: prompts, workflows, multi-step reasoning, deciding which tools to call and in what order.
Models
Foundation models from providers, open models, or specialized models. A good architecture avoids hard-wiring a single model everywhere, so you can choose the right model per use case and change later.
Context and knowledge
Models know a lot about the world but nothing about your business. Retrieval, search and vector databases bring in relevant enterprise knowledge at the moment it is needed. In my experience, the quality of this layer determines the quality of the answers far more than the choice of model.
Tools and integration
This is where AI moves from answering to acting. APIs, events and standards like the Model Context Protocol let AI read from and act on enterprise systems. Everything we learned about integration architecture applies here, with higher stakes.
Data foundation
Clean, governed, well-described data. Without it, everything above it is built on sand.
Cross-cutting concerns
Security and identity, evaluation and testing, observability, cost management and governance run through every layer.
Principles I Recommend
- Start from business problems, not from technology demos
- Keep models swappable behind clear interfaces
- Invest in context and data before chasing the newest model
- Expose capabilities through governed APIs, not direct database access
- Evaluate continuously, because AI quality can drift
- Put humans in the loop for consequential decisions
- Track cost per use case, since usage-based pricing adds up quickly
Common Mistakes
- Dozens of disconnected pilots with no shared platform
- Building everything custom when platform capabilities already exist
- Ignoring security until an agent can see data it should not
- Measuring demos instead of business outcomes
Final Thought
AI architecture is not a separate discipline from enterprise architecture.
It is enterprise architecture applied to a new kind of component: one that can reason, and increasingly, act.