LLM Application Requirements: A Production Checklist

Most LLM apps I've seen fail in production don't fail because the model is bad. They fail because nobody wrote down what "working" meant before launch. The demo passed, the stakeholder nodded, and three weeks later a retrieval bug, a runaway token bill or a prompt injection turned the thing into an incident. This is the checklist I use now, built from RAG pipelines and multi-agent systems that run unattended, including the ones behind BizFlowAI.

Functional requirements: define the contract, not the vibe

A functional requirement for an LLM app is a testable statement about inputs, outputs and failure behavior, not a description of what the model "should understand." If you can't write it as a test, it isn't a requirement yet.

For every capability I write four things down:

1. Input contract. What shapes, languages and lengths can arrive, and what happens when something outside that arrives.
2. Output contract. Structured output with a schema, not free text that a regex has to rescue later.
3. Failure behavior. What the system does when the model refuses, times out, returns malformed JSON or says "I don't know."
4. Acceptance set. A fixed evaluation set (even 50 to 100 real examples) that defines "done."

In my content pipelines, every stage (research, draft, optimize, publish) has an explicit output schema and a validator. A stage that returns something invalid doesn't pass its garbage downstream. It retries once with the validation error in the prompt, then fails loudly into a queue a human can inspect. That single rule removed most of the "weird output three steps later" bugs, because the weirdness gets caught at the boundary where it was created.

The most underrated functional requirement is the "I don't know" path. For RAG, I require that the system can answer "the sources don't cover this" and I test that explicitly with questions I know are out of corpus. If your acceptance set has no unanswerable questions, you are measuring how well the model bluffs.

Data requirements: your RAG is only as good as your corpus hygiene

The data requirements for a RAG system are provenance, freshness, chunk integrity and access control. Embeddings and the vector index come after those. Teams tune embedding models for weeks while the corpus contains duplicated, stale and unlabeled documents.

What I require before indexing anything:

• Provenance on every chunk. Source document, version or timestamp, and a stable ID. If a user disputes an answer, I need to trace it to the exact passage.
• A re-index strategy. Documents change. Decide up front whether you do full rebuilds, incremental upserts keyed on content hash, or both. I key on a content hash so unchanged documents cost nothing.
• Chunking tested against real queries, not a default of "500 tokens with overlap." I look at what retrieval actually returns for 30 real questions and adjust until the answer-bearing passage shows up intact.
• Per-tenant or per-role filtering enforced at query time, in the retrieval layer, not in the prompt. Telling the model "don't reveal other customers' data" is not access control.

On retrieval itself, my default for anything beyond a toy is hybrid search: pgvector for semantic similarity plus keyword/full-text in Postgres, fused with reciprocal rank fusion (RRF). Pure vector search misses exact identifiers, SKUs, error codes and names, which are exactly the queries users care about. Keeping both in one Postgres instance also means one thing to back up, secure and monitor.

There's a security angle here too. OWASP's 2025 Top 10 for LLM Applications includes LLM08:2025 Vector and Embedding Weaknesses, which covers risks specific to RAG-style systems (OWASP Top 10 for LLM Applications 2025). Anything you ingest is effectively part of your prompt. Treat ingestion as an untrusted input path, not an internal one.

Infrastructure and cost requirements: design for the bill, not just the demo

Infrastructure requirements for an LLM app come down to three things: where state lives, what scales to zero, and what the per-request cost envelope is. I decide all three before writing prompts.

My default architecture for unattended systems is serverless and event-driven: EventBridge schedules, Lambda workers, queues between stages, Postgres (Supabase or RDS) as the system of record. Idle cost is near zero, and a failed stage is a message in a dead-letter queue instead of a dead server. The same pattern is behind the AWS and Zendesk integration that delivered first-ever SLA compliance: boring, observable, retryable steps.

Cost is a requirement, so give it a budget the way you'd give latency one. Three levers matter most, and Anthropic's pricing page documents all of them:

Notice the Haiku 5.5 detail: its price is tiered by prompt size, $0.10 input / $0.50 output per MTok up to 100,000 tokens and $0.50 / $2.50 above that. A RAG system that stuffs context carelessly can quietly cross that line and multiply its own cost. Set a hard cap on retrieved context per request and alert when the average creeps up.

My practical rules: put stable content (system prompt, tool definitions, reference documents) at the front of the prompt so it caches; route classification and extraction to the cheapest tier that passes your eval set; push anything that doesn't need a human waiting into batch. And check the pricing page before you commit to a model, since these numbers change.

Latency requirements: pick the budget per user-facing step

Latency should be specified per interaction type, because a chat answer, a background enrichment job and a tool-calling agent loop have completely different tolerances. One global "it should be fast" is useless.

I split workloads into three buckets:

• Interactive (a human is watching): stream tokens, show retrieval progress, keep the critical path to one model call plus one retrieval. Time to first token matters more than total time.
• Near-real-time (a webhook or ticket event): seconds are fine, but it must be idempotent and retry-safe.
• Background (scheduled pipelines): no latency requirement, so use batch pricing and maximize quality instead.

The number I watch most in agent systems is step count. Latency compounds across the loop: every tool call is a model round trip plus the tool itself. I cap iterations, give each tool a timeout, and make the agent return a partial result with an explicit "incomplete" flag rather than spin. Specific p95 targets depend entirely on your product and users, so measure your own baseline under realistic load and write the budget from that, not from someone else's blog.

Two design moves buy more latency than model swaps: parallelize independent retrievals and tool calls, and cache aggressively at the application layer for questions that repeat.

Safety and compliance requirements: assume the model output is untrusted

The safety requirement that matters most in practice is to treat all model output as untrusted input to the next component. That is straight from OWASP: LLM05:2025 Improper Output Handling is defined as insufficient validation, sanitization and handling of LLM outputs before they reach downstream components (OWASP).

For agents, the concept I use most is LLM06:2025 Excessive Agency, which splits into three root causes: excessive functionality, excessive permissions and excessive autonomy (Aembit's OWASP breakdown). That maps cleanly onto three review questions for every tool an agent can call:

1. Does the agent need this capability at all? (functionality)
2. Does the credential behind it have the minimum scope? (permissions)
3. Does this action need a human approval or a dry-run gate? (autonomy)

In my multi-agent systems, the publishing stage is the dangerous one, so it is the one with the tightest permissions and a validation gate in front of it. Read-only tools get wide freedom. Anything that writes, sends or spends gets narrow scope and a check.

On the compliance side, if you serve EU users the timeline moved. Regulation (EU) 2026/1744, the Digital Omnibus on AI, entered into force on 27 July 2026 (Cloud Security Alliance note). Stand-alone high-risk systems (Annex III) now comply from 2 December 2027 rather than 2 August 2026, and AI embedded in products under EU product-safety law (Annex I) from 2 August 2028. It is deferred, not cancelled. Meanwhile most Article 50 transparency duties, such as chatbot disclosure, have applied since 2 August 2026, and generative systems already on the market before that date have until 2 December 2026 to add machine-readable marking (technology.org summary). I'm not a lawyer, and these come from secondary summaries, so confirm against the Official Journal before you build a compliance plan on them.

For a voluntary but useful baseline, NIST's Generative AI Profile (AI 600-1, released 26 July 2024) names 12 risk categories, including confabulation, data privacy and information security (NIST). I use it as a checklist for design reviews, not a certification target.

Observability requirements: if you can't replay it, you can't fix it

Every production LLM request should be reconstructable: the exact prompt, the retrieved chunks with their IDs, the model and parameters, the tool calls, the output and the cost. Without that, "the bot said something wrong" is an unsolvable ticket.

What I log per request:

• A trace ID that follows the request across retrieval, model calls and tools
• Retrieved chunk IDs and scores (so I can tell a retrieval failure from a generation failure)
• Token counts and cost per call, aggregated per pipeline stage
• Validation failures and retry counts
• A pointer to the stored prompt and response, with PII handling decided in advance

OpenTelemetry has GenAI semantic conventions for this, in the open-telemetry/semantic-conventions-genai repository. Two cautions: everything there is still at Development stability, so attribute names can change between releases (overview), and gen_ai.provider.name replaced the deprecated gen_ai.system, so older dashboards may need migrating. A RAG retrieval span uses gen_ai.operation.name = retrieval with span kind CLIENT (spec). I'd adopt the conventions but wrap them in a thin internal layer so a rename is a one-file change.

Observability also closes the loop. My self-learning content system works because it measures real search performance and feeds it back into targeting. Operational telemetry should do the same for quality: sample real traffic into your eval set, so your acceptance tests drift toward reality instead of staying frozen at launch day.

What I'd do

If you're shipping an LLM feature next month, here is the order I'd work in:

1. Write the acceptance set first, including unanswerable and adversarial cases. Fifty good examples beat none.
2. Put a schema and validator on every model boundary. Fail loudly, retry once, queue the rest.
3. Set a per-request cost and latency budget before choosing the model, then tier models to fit it.
4. Enforce access control in retrieval and permissions in tools, never in the prompt.
5. Trace everything from day one. Retrofitting observability after an incident is miserable.
6. Treat compliance dates as a moving target and re-check primary sources each quarter.

None of this is glamorous. It is also the difference between an LLM demo and a system you can leave running overnight.

If you're designing something like this and want a second pair of eyes, you can reach me at lazar-milicevic.com/#contact, or keep reading on the blog for more production notes on agents and RAG.