AI Engineering Architecture and User Feedback (Chapter 10 - AI Engineering - By Chip Huyen)

Download Book

<<< Previous Chapter Next Book >>>

Building Real AI Products: Architecture, Guardrails, and the Power of User Feedback

Why successful AI systems are more about engineering and feedback loops than models

---

Introduction: From “Cool Demo” to Real Product

Training or calling a large language model is the easy part of modern AI.

The hard part begins when you try to turn that model into a real product:

• One that real users rely on

• One that must be safe, fast, affordable, and reliable

• One that improves over time instead of silently degrading

This chapter is about that hard part.

So far, most AI discussions focus on prompts, RAG, finetuning, or agents in isolation. But real-world systems are not isolated techniques—they are architectures. And architectures are shaped not just by technical constraints, but by users.

This blog post walks through a progressive AI engineering architecture, starting from the simplest possible setup and gradually adding:

• Context

• Guardrails

• Routers and gateways

• Caching

• Agentic logic

• Monitoring and observability

• Orchestration

• User feedback loops

At the end, you’ll see why user feedback is not just UX—it’s your most valuable dataset.

---

1. The Simplest Possible AI Architecture (And Why It Breaks)

The One-Box Architecture

Every AI product starts the same way:

1. User sends a query

2. Query goes to a model

3. Model generates a response

4. Response is returned to the user

That’s it.

This architecture is deceptively powerful. For prototypes, demos, and internal tools, it works surprisingly well. Many early ChatGPT-like apps stopped right here.

But this setup has major limitations:

• The model knows only what’s in its training data

• No protection against unsafe or malicious prompts

• No cost control

• No monitoring

• No learning from users

This architecture is useful only until users start depending on it .

---

Why Real Products Need More Than a Model Call

Once users rely on your application:

• Wrong answers become costly

• Latency becomes noticeable

• Hallucinations become dangerous

• Abuse becomes inevitable

To fix these, teams don’t “replace the model.”
They add layers around it.

---

2. Step One: Enhancing Context (Giving the Model the Right Information)

Context Is Feature Engineering for LLMs

Large language models don’t magically know your:

• Company policies

• Internal documents

• User history

• Real-time data

To make models useful, you must construct context.

This can be done via:

• Text retrieval (documents, PDFs, chat history)

• Structured data retrieval (SQL, tables)

• Image retrieval

• Tool use (search APIs, weather, news, calculators)

This process—commonly called RAG—is analogous to feature engineering in traditional ML.

The model isn’t getting smarter; it’s getting better inputs .

---

Trade-offs in Context Construction

Not all context systems are equal.

Model APIs (OpenAI, Gemini, Claude):

• Easier to use

• Limited document uploads

• Limited retrieval control

Custom RAG systems:

• More flexible

• More engineering effort

• Require tuning (chunk size, embeddings, rerankers)

Similarly, tool support varies:

• Some models support parallel tools

• Some support long-running tools

• Some don’t support tools at all

As soon as context is added, your architecture already looks much more complex—and much more powerful.

---

3. Step Two: Guardrails (Protecting Users and Yourself)

Why Guardrails Are Non-Negotiable

AI systems fail in ways traditional software never did.

They can:

• Leak private data

• Generate toxic content

• Execute unintended actions

• Be tricked by clever prompts

Guardrails exist to reduce risk, not eliminate it (elimination is impossible).

There are two broad types:

• Input guardrails

• Output guardrails

---

Input Guardrails: Protecting What Goes In

Input guardrails prevent:

• Sensitive data leakage

• Prompt injection attacks

• System compromise

Common risks:

• Employees pasting secrets into prompts

• Tools accidentally retrieving private data

• Developers embedding internal policies into prompts

A common defense is PII detection and masking:

• Detect phone numbers, IDs, addresses, faces

• Replace them with placeholders

• Send masked prompt to external API

• Unmask response locally using a reverse dictionary

This allows functionality without leaking raw data .

---

Output Guardrails: Protecting What Comes Out

Models can fail in many ways.

Quality failures:

• Invalid JSON

• Hallucinated facts

• Low-quality responses

Security failures:

• Toxic language

• Private data leaks

• Brand-damaging claims

• Unsafe tool invocation

Some failures are easy to detect (empty output).
Others require AI-based scorers.

---

Handling Failures: Retries, Fallbacks, Humans

Because models are probabilistic:

• Retrying can fix many issues

• Parallel retries reduce latency

• Humans can be used as a last resort

Some teams route conversations to humans:

• When sentiment turns negative

• After too many turns

• When safety confidence drops

Guardrails always involve trade-offs:

• More guardrails → more latency

• Streaming responses → weaker guardrails

There is no perfect solution—only careful balancing.

---

4. Step Three: Routers and Gateways (Managing Complexity at Scale)

Why One Model Is Rarely Enough

As products grow, different queries require different handling:

• FAQs vs troubleshooting

• Billing vs technical support

• Simple vs complex tasks

Using one expensive model for everything is wasteful.

This is where routing comes in.

---

Routers: Choosing the Right Path

A router is usually an intent classifier.

Examples:

• “Reset my password” → FAQ

• “Billing error” → human agent

• “Why is my app crashing?” → troubleshooting model

Routers help:

• Reduce cost

• Improve quality

• Avoid out-of-scope conversations

Routers can also:

• Ask clarifying questions

• Decide which memory to use

• Choose which tool to call next

Routers must be:

• Fast

• Cheap

• Reliable

That’s why many teams use small models or custom classifiers .

---

Model Gateways: One Interface to Rule Them All

A model gateway is a unified access layer to:

• OpenAI

• Gemini

• Claude

• Self-hosted models

Benefits:

• Centralized authentication

• Cost control

• Rate limiting

• Fallback strategies

• Easier maintenance

Instead of changing every app when an API changes, you update the gateway once.

Gateways also become natural places for:

• Logging

• Analytics

• Guardrails

• Caching

---

5. Step Four: Caching (Reducing Latency and Cost)

Why Caching Matters in AI

AI calls are:

• Slow

• Expensive

• Often repetitive

Caching avoids recomputing answers.

There are two main types:

• Exact caching

• Semantic caching

---

Exact Caching: Safe and Simple

Exact caching reuses results only when inputs match exactly.

Examples:

• Product summaries

• FAQ answers

• Embedding lookups

Key considerations:

• Eviction policy (LRU, LFU, FIFO)

• Storage layer (memory vs Redis vs DB)

• Cache duration

Caching must be careful:

• User-specific data should not be cached globally

• Time-sensitive queries should not be cached

Mistakes here can cause data leaks .

---

Semantic Caching: Powerful but Risky

Semantic caching reuses answers for similar queries.

Process:

1. Embed query

2. Search cached embeddings

3. If similarity > threshold, reuse result

Pros:

• Higher cache hit rate

Cons:

• Incorrect answers

• Complex tuning

• Extra vector search cost

Semantic caching only works well when:

• Embeddings are high quality

• Similarity thresholds are well tuned

• Cache hit rate is high

Otherwise, it often causes more harm than good.

---

6. Step Five: Agent Patterns and Write Actions

Moving Beyond Linear Pipelines

Simple pipelines are sequential:

• Query → Retrieve → Generate → Return

Agentic systems introduce:

• Loops

• Conditional branching

• Parallel execution

Example:

• Generate answer

• Detect insufficiency

• Retrieve more data

• Generate again

This dramatically increases capability.

---

Write Actions: Power with Risk

Write actions allow models to:

• Send emails

• Place orders

• Update records

• Trigger workflows

They make systems vastly more useful—but vastly more dangerous.

Write actions must be:

• Strictly guarded

• Audited

• Often human-approved

Once write actions are added, observability becomes mandatory, not optional.

---

7. Monitoring and Observability: Seeing Inside the Black Box

Monitoring vs Observability

Monitoring:

• Tracks metrics

• Tells you something is wrong

Observability:

• Lets you infer why it’s wrong

• Without deploying new code

Good observability reduces:

• Mean time to detection (MTTD)

• Mean time to resolution (MTTR)

• Change failure rate (CFR)

---

Metrics That Actually Matter

Metrics should serve a purpose.

Examples:

• Format error rate

• Hallucination signals

• Guardrail trigger rate

• Token usage

• Latency (TTFT, TPOT)

• Cost per request

Metrics should correlate with business metrics:

• DAU

• Retention

• Session duration

If they don’t, you may be optimizing the wrong thing .

---

Logs and Traces: Debugging Reality

Logs:

• Record events

• Help answer “what happened?”

Traces:

• Reconstruct an entire request’s journey

• Show timing, costs, failures

In AI systems, logs should capture:

• Prompts

• Model parameters

• Outputs

• Tool calls

• Intermediate results

Developers should regularly read production logs—their understanding of “good” and “bad” outputs evolves over time.

---

Drift Detection: Change Is Inevitable

Things that drift:

• System prompts

• User behavior

• Model versions (especially via APIs)

Drift often goes unnoticed unless explicitly tracked.

Silent drift is one of the biggest risks in AI production.

---

8. User Feedback: Your Most Valuable Dataset

Why User Feedback Is Strategic

User feedback is:

• Proprietary

• Real-world

• Continuously generated

It fuels:

• Evaluation

• Personalization

• Model improvement

• Competitive advantage

This is the data flywheel.

---

Explicit vs Implicit Feedback

Explicit feedback:

• Thumbs up/down

• Ratings

• Surveys

Pros:

• Clear signal
  Cons:

• Sparse

• Biased

Implicit feedback:

• Edits

• Rephrases

• Regeneration

• Abandonment

• Conversation length

• Sentiment

Pros:

• Abundant
  Cons:

• Noisy

• Hard to interpret

Both are necessary.

---

Conversational Feedback Is Gold

Users naturally correct AI:

• “No, I meant…”

• “That’s wrong”

• “Be shorter”

• “Check again”

These corrections are:

• Preference data

• Evaluation signals

• Training examples

Edits are especially powerful:

• Original output = losing response

• Edited output = winning response

That’s free RLHF data.

---

Designing Feedback Without Annoying Users

Good feedback systems:

• Fit naturally into workflows

• Require minimal effort

• Can be ignored

Great examples:

• Midjourney’s image selection

• GitHub Copilot’s inline suggestions

• Google Photos’ uncertainty prompts

Bad feedback systems:

• Interrupt users

• Ask too often

• Demand explanation

---

Conclusion: Architecture and Feedback Are the Real AI Moats

Modern AI success is not about:

• The biggest model

• The cleverest prompt

It’s about:

• Thoughtful architecture

• Layered defenses

• Observability

• Feedback loops

• Continuous iteration

Models will commoditize.
APIs will change.
What remains defensible is how well you learn from users and adapt.

That is the real craft of AI engineering.

Tags: Artificial Intelligence,Generative AI,Agentic AI,Technology,Book Summary,