LLM Apps in Python: A Production Stack Without Framework Bloat

The first LLM feature I shipped that actually paid for itself was about 200 lines of Python and one SDK. The one that cost me a weekend of debugging was a framework-based pipeline where I could not tell which layer had swallowed a retry. Since then I have built agents, RAG pipelines and scheduled workers that run unattended, and my default stack has gotten smaller every year. This post is the stack I use now, where I still reach for a framework, and what I do about retries, cost and deployment.

Start with the raw SDK, and know what you are skipping

Use the provider SDK directly until you can name the specific problem a framework would solve for you. Most LLM apps are an HTTP call with structured input, a loop over tool calls, and some state. That is a small amount of code, and owning it means every failure is yours to read.

Here is the whole core of an agent, which is the part frameworks like to hide:

import anthropic

client = anthropic.Anthropic(max_retries=2, timeout=60.0)

def run_agent(messages, tools, tool_impls, max_turns=8):
    for _ in range(max_turns):
        resp = client.messages.create(
            model=MODEL, max_tokens=2048, tools=tools, messages=messages
        )
        messages.append({"role": "assistant", "content": resp.content})
        if resp.stop_reason != "tool_use":
            return resp
        results = []
        for block in resp.content:
            if block.type == "tool_use":
                out = tool_impls[block.name](**block.input)
                results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": str(out),
                })
        messages.append({"role": "user", "content": results})
    raise RuntimeError("agent exceeded max_turns")

The max_turns cap is not decoration. An agent without a turn budget is an unbounded spend loop waiting for the day a tool returns something confusing.

Where I do reach for a framework: when I need a durable graph with checkpoints and human-in-the-loop pauses, or when a team of five people needs a shared vocabulary for orchestration. What I avoid is adopting one for the first prototype. The abstraction you pay for on day one is the one you fight on day ninety, when you need to change how a single request is retried or how one tool's output is truncated.

Structure prompts and tool calls like an interface

Treat the prompt as a versioned artifact with a typed contract, not a string you edit in place. Most "the prompt worked in testing" failures are really contract failures: the input shape drifted, a field went missing, or the output was parsed with a regex.

My rules:

1. Keep the system prompt static and put volatile data last. This matters for caching (more below), and it also makes diffs readable.
2. Describe tools like you would document a public API. The tool description is the only documentation the model gets. Vague descriptions produce wrong tool choices, which no amount of prompt tuning fixes.
3. Validate every tool input and output with Pydantic. Return structured, bounded results. A tool that dumps a 40,000-token blob into the context is a cost bug and an accuracy bug at the same time.
4. Return errors to the model as tool results, with a message it can act on, instead of raising. The model recovers from "date must be ISO 8601, got '3rd of May'" better than most of my exception handlers do.
5. Store the prompt version with every run. When quality drops on a Tuesday, you want to join the bad runs against the prompt that produced them.

from pydantic import BaseModel, Field

class SearchArgs(BaseModel):
    query: str = Field(min_length=3, max_length=200)
    limit: int = Field(default=5, ge=1, le=10)

def search_docs(**raw):
    try:
        args = SearchArgs(**raw)
    except Exception as e:
        return f"INVALID_INPUT: {e}"
    return run_search(args.query, args.limit)

Add RAG as a pipeline, not a feature

RAG fails in retrieval far more often than in generation, so build and measure retrieval on its own. If the right chunk is not in the top results, no model will save you. I keep retrieval as a plain function with a clear signature, separate from the LLM call, so I can test it without spending a token.

My production default is Postgres with pgvector, which keeps vectors next to the relational data they describe, plus a hybrid search that fuses keyword and vector results with Reciprocal Rank Fusion (RRF). Pure vector search is bad at exact identifiers, product codes and rare names. Pure keyword search is bad at paraphrase. Fusing them is cheap and fixes both.

The constant in the formula is a smoothing parameter. The original RRF paper (Cormack et al., 2009) introduced the method, and a commonly used default for that constant is 60, which is what I start from and then tune against my own test questions.

def rrf(rank_lists, k=60):
    scores = {}
    for ranks in rank_lists:
        for pos, doc_id in enumerate(ranks):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + pos + 1)
    return sorted(scores, key=scores.get, reverse=True)

The gotchas I have actually hit:

• Chunk by meaning, not by character count. Splitting mid-table or mid-procedure produces chunks that retrieve well and answer badly.
• Keep a small labeled set of real questions and check retrieval recall against it every time you change chunking or the embedding model. Ten honest questions beat a hundred synthetic ones.
• Cap and cite. Put a hard limit on retrieved tokens and have the model reference chunk IDs, so a wrong answer can be traced to a wrong chunk in minutes.
• Let the model say it does not know. Instruct it explicitly, and test that it does. A confident answer from irrelevant chunks is the worst failure mode RAG has.

Retries, timeouts and cost: the unglamorous 80%

The default SDK settings are fine for a script and dangerous for a worker. According to Anthropic's docs, the Python SDK retries connection errors, 408, 409, 429 and 5xx responses twice by default with short exponential backoff, configurable through max_retries. The same docs list a default request timeout of 10 minutes, and a timed-out request raises APITimeoutError and is retried. Put those together and one stuck call can quietly consume 30 minutes of wall-clock time. On a serverless function with a hard ceiling, that is an outage. The SDK reference is at platform.claude.com/docs/en/cli-sdks-libraries/sdks/python.

So I always set an explicit timeout and decide retries per call:

resp = client.with_options(max_retries=1, timeout=45.0).messages.create(...)

Then I add an application-level layer on top, because SDK retries only handle transient transport errors, not logical failures:

• Idempotency keys on anything with side effects, so a retried tool call does not send the email twice.
• A circuit breaker or budget per run: max turns, max tokens, max tool calls. Fail loudly and write the failure down.
• Dead-letter the poison inputs. If the same input fails repeatedly, park it and alert. Do not let it loop forever.

For cost, the biggest single lever is prompt caching. Per Anthropic's pricing docs, cache writes cost a premium over the base input price, while cache reads cost only a fraction of it. Their docs describe when each cache duration pays off. The current multipliers and break-even points are on the pricing page and in the prompt caching docs. That arithmetic is why prompt layout matters: long, stable content (system prompt, tool definitions, reference documents) goes first, volatile content goes last, so the prefix is reusable.

Model choice is the second lever. Prices per million tokens differ by roughly an order of magnitude across tiers, and you can compare the current ones on the pricing page linked above. I route by task: a small model for classification, extraction and routing, a larger one only for the steps that need real reasoning. In an agent that makes eight calls per run, moving six of them to the cheap tier changes the monthly bill more than any prompt trick.

One more habit: log token counts and cost per run, per step. You cannot optimize what only shows up as a monthly invoice.

Deploy to serverless AWS, with the limits in your head

Lambda plus API Gateway scales to zero and suits bursty LLM workloads, but several hard limits shape the design. The AWS documentation lists a maximum invocation duration, payload size caps for synchronous and asynchronous invocations, a memory range with CPU scaling alongside it, and default concurrency and throttle quotas. I do not quote the numbers from memory because they change and differ by region. Check the current values in the Lambda limits documentation (and the equivalent page for your region) before you design around them. One consequence is worth stating plainly: API Gateway's default throttle and Lambda's default concurrency limit are different numbers, so a burst can exceed Lambda capacity unless you raise the concurrency quota.

What this means in practice:

1. Do not run long agents synchronously behind an API. Accept the request, enqueue it, return a job ID, and let the worker run asynchronously. This is the same pattern I used on a serverless AWS and Zendesk integration, and it is why the system could finally hold an SLA: the user-facing path never waited on a model.
2. If you need token streaming to a browser, use streaming. API Gateway REST APIs support response streaming for Lambda proxy integrations when the transfer mode is set to STREAM, which relaxes the usual response size and timeout limits. The details and idle timeout behavior are in the response transfer mode docs.
3. Set the SDK timeout well under the Lambda timeout, so your code handles the failure instead of the platform killing it mid-write.
4. Use Powertools for AWS Lambda (Python) for structured logging, metrics and idempotency. I would rather adopt a thin, well-scoped library like that than a full orchestration framework. Check the supported runtimes list for the Python version you target.
5. Keep state out of the function. Postgres or DynamoDB for run state, so a retried or resumed invocation can pick up where it stopped. For workflows that must survive pauses, AWS also offers durable functions on selected Python runtimes (see the durable functions supported runtimes), which is worth evaluating before you build your own checkpointing.

EventBridge Scheduler plus Lambda is also how I run unattended workers: a cron-like trigger, a short-lived function, state in the database. Nothing idles, so the system costs almost nothing when there is nothing to do.

What I'd do

If I were starting a new LLM application tomorrow, I would do this:

1. Raw SDK, a 30-line agent loop, hard caps on turns, tokens and tool calls from the first commit.
2. Pydantic on every tool boundary, with errors returned to the model as readable messages.
3. Postgres with pgvector and RRF for retrieval, plus ten real questions as a retrieval test set before touching the generator.
4. Explicit timeouts and retry counts on every call, and idempotency keys on every side effect.
5. Prompt layout designed for caching (static first, volatile last), and model routing by task difficulty.
6. Async by default on AWS: queue in, worker out, streaming only where a human is watching.
7. Per-run logging of prompt version, tokens, cost and failure reason, so quality and spend are queryable.

A framework earns a place only after one of these pieces hurts. At that point you will know exactly what you need it to do, which is the best moment to choose one.

If you are weighing a build like this and want a second pair of eyes on the architecture, you can reach me at lazar-milicevic.com/#contact, or keep reading the other engineering notes on the blog.