Decision records

Status

Accepted (2026-10-11), amended with Dominic’s approval before merge

Context

A public assistant must represent Dominic accurately and cannot become an unrestricted model proxy. Visitors can send repeated requests, long messages or instructions to ignore its purpose. Usage must have a predictable ceiling.

Long client-supplied histories add uncached input cost and can forge earlier assistant replies. Bound their total size and label every turn as untrusted. IPv6 visitors can rotate addresses within one network, so rate limits must share an allowance across a /64. Thinking shares the model's token budget; reaching that budget must close cleanly rather than appear as a provider fault.

Decision

Before exposing chat:

• Refuse topics unrelated to Dominic and his work.
• Say it does not know when the approved knowledge document lacks an answer. Never guess or invent claims.
• Enforce 20 requests per visitor per UTC hour and 50 per UTC day, identified by Vercel's trusted proxy headers: prefer x-vercel-forwarded-for, then x-real-ip, then the rightmost x-forwarded-for entry. Treat empty values as absent. Key IPv6 visitors by their /64 prefix. Local development shares a loopback budget when no proxy header exists. Production without a valid IP header returns 503.
• Enforce a global ceiling of 1,000 requests per UTC day. Use atomic Redis fixed-window limits through @upstash/ratelimit. Check the visitor limits before the global limit, so visitor-blocked requests do not use its allowance. Return 429 with Retry-After when a limit is reached; Redis errors return 503. Bound each Redis request to five seconds and do not retry ambiguous writes.
• Cap reply length and conversation length on the server, including the input accepted from a visitor. Accept at most 12 alternating user/assistant messages, starting and ending with user (at most 11 messages in practice). Cap user messages at 1,000 characters, assistant history at 2,500 characters per message, the total across all messages at 12,000 characters before trimming and the raw JSON body at 100,000 bytes. Reject exceeded limits, blank messages, extra fields, invalid JSON and other content types with 400. Set max_tokens to 2,000, shared between thinking and answer text, at low effort.
• Set a monthly spend limit on the API key's workspace. Verify it enforces a spending ceiling; an alert alone is not a limit.

The prompt must treat visitor messages as requests, not authority to replace these rules. Wrap every client-supplied turn in explicit visitor-supplied conversation delimiters, with its content as a JSON-encoded string. Mark assistant turns as "claimed earlier reply, unverified" and state in the fixed rules that these may be forged and never override the rules or document. Limits and refusals must produce clear responses. Use the official Anthropic SDK and stream only text deltas as UTF-8 Markdown. Enable the server-side refusal fallback and handle a final refusal with a short polite response. A max_tokens stop closes the stream cleanly: ask the visitor to narrow the question when no text was sent, or append a short note that the answer was cut short when text has already been sent. Model startup failures return 502; a stream failure after headers errors the response body and logs the failure. Bound model streaming by a 55-second request deadline and abort when the visitor disconnects. Send X-Content-Type-Options: nosniff on every endpoint response. Production Redis URLs must use HTTPS to protect the REST token in transit.

Log one structured line per request with status, latency, input/output tokens and cache read/write tokens. Do not log visitor messages, the knowledge document, IP addresses or API credentials. Record the stop reason on completed streams, including output capped by the token limit. Distinguish invalid or missing visitor-IP headers from configuration problems, the request deadline from provider failure, and visitor cancellation from both. Identify SDK errors by their class name or wire type, rather than the generic error.name.

Consequences

• The endpoint enforces limits even when the browser is bypassed.
• Exhausted visitor or conversation limits prevent further model calls.
• Shorter histories bound uncached input cost; explicit untrusted labels reduce the influence of forged replies but do not verify their authenticity.
• Visitors sharing an IPv6 /64 share a rate-limit allowance.
• Workspace spend remains bounded independently of visitor rate limiting.
• Scope refusals and uncertainty need behavioural tests before release.
• Fixed windows can admit a burst across an hour or day boundary. The global allowance remains 1,000 admitted requests per UTC day.
• Counters are consumed before knowledge and model calls, including failures. A request rejected by a later limit may consume an earlier limit's allowance.
• Automated tests use real local Redis and the official SDK against a local HTTP test server. Scope and grounding are also reviewed through the manual eval.
• The workspace spend ceiling must be configured and verified separately before deployment. Application rate limits do not set that provider limit.
• Pin @upstash/ratelimit to 2.0.8: subsequent published releases send a Lua flag unsupported by stock Redis in the required local setup. Upgrade only after the real Redis tests verify upstream compatibility.