Engineering
Four things that keep an agent usable
You cannot remove a model's randomness, and you cannot rely on its good behaviour. We constrain it with middleware, prompt discipline and retrieval: cost stays flat, the evidence stays fixed, every sentence carries a source, and the first character arrives in a second. That is the difference between a professional build and a wrapper demo.
Cost · Long conversations stay affordable
Answer cost = input tokens x turns. We cut both.
- Old tool output is trimmed automatically The long tool output produced by each retrieval or search turn is replaced with a one-line placeholder as soon as the user asks the next question, while the current turn keeps its full call chain. Tool results are the biggest token consumer in a long conversation.
- Automatic summarisation above a threshold Once a conversation passes a token or message-count threshold, earlier history is distilled into a structured summary — goal, key conclusions, open items — and only recent messages are kept. The memory survives; the payload does not grow.
- A soft limit on answer length The system prompt requires the model to stop once it has given enough context: concise without being curt, and never padding for length.
Consistency · The same question, the same basis for the answer
An answer need not drift with the model, provided it is anchored to something else.
- Answers are anchored to the knowledge base Every answer is retrieved from the client's own corpus first (RAG), so what is returned rests on indexed source text. While the corpus holds, the basis of the answer holds — regardless of model version, time of day or phrasing.
- Persona and tone are constrained Voice and values are written into the constraints, so the hundredth question meets the same character as the first.
- Session state is persisted Conversation history is written to the database, so a dropped connection or a different device does not break the thread.
Accuracy · Every sentence carries a source
This is the hardest commitment: citation is enforced by the system.
- Retrieval and generation have separate jobs Retrieval uses a dedicated embedding model plus query rewriting to map corpus and question into the same semantic space, so a different wording of the same thing is still recalled. Generation then fuses and organises the recalled passages. Retrieval finds; the model writes.
- Provenance travels with the passage Every recalled passage carries its source filename, page number, confidence score and document path.
- Inline citation is enforced by prompt The system requires inline [1][2] citations and forbids a list of sources at the end — the citation has to sit on the sentence it supports.
- A closed traceability loop A reader who sees [2] can open source document 2 in the knowledge base and check the original. When an answer is wrong, we can point to which document caused it.
Speed · First character in a second, no dead wait
Perceived speed follows first-token latency, not total duration.
- WebSocket streaming The answer appears character by character; the first one arrives within a second.
- Sub-second semantic retrieval Vector search runs inside the database and returns the top ten passages in under a second, with no extra hop to an external search service.
- The workflow is visible The interface shows \u201csearching the knowledge base\u201d and \u201cwriting the answer\u201d as they happen. People can see the system working rather than staring at a black box — and the visible retrieval step is itself the clearest demonstration that answers are grounded.
- Model choice is configuration Provider and model are configuration values; switching to a faster, cheaper model takes no code change.
What these two screens show
Constraints start at the front door
Persona and values are written into the system constraints, and the suggested openers steer the first question into what the agent can answer.
The footer permanently carries the AI-generated notice and a crisis-support pointer — the boundary sits where the user can see it.
The retrieval step is shown, and the citation lands on the sentence
The interface displays this query’s retrieval parameters (query and top_k = 5) and the source documents recalled; [1][2] in the answer point back to them.
An answer can be checked, and a wrong one traced to the document that caused it. “Processed in 8.6s” is the time for the complete answer, with every step visible along the way.
Client branding is redacted in these screenshots. The interface shown is the Chinese production build.
In one table
| Capability | What we built | What it means for the client |
|---|---|---|
| Cost | Two-layer context compression (tool-output trimming plus history summarisation) with a conciseness constraint | Long conversations keep token cost under control |
| Consistency | Answers anchored to the client's own knowledge base, with persona constraints and persisted session state | The same question always rests on the same basis, in the same voice |
| Accuracy | Retrieval and generation separated, with enforced inline [1][2] citations carrying source file and confidence | Every sentence has a source; a wrong answer can be traced to its document |
| Speed | WebSocket character streaming, in-database vector search, switchable models | First character in a second, no dead wait on long answers |
These mechanisms run in production in a question-answering agent built for a cultural education institution, and the same base has been reused for university administration and post-visit care. Implementation detail is deliberately out of scope on this page.
Contact
Start with one conversation about the case
If you have a specific case — one production line, one faculty, one department, one clinical pathway — we'll do a feasibility conversation at no charge and tell you whether it's worth building.