Streaming and grounded chat

Streaming

Set "stream": true on POST /api/llm/v1/chat/completions. The response is text/event-stream with X-Accel-Buffering: no, so intermediaries do not buffer it.

data: {"object":"rag_metadata","active":false,"status":"inactive","snippet_count":0,"rag_snippets":null}

data: {"choices":[{"delta":{"content":"One"},"finish_reason":null}]}

data: {"choices":[{"delta":{"content":"."},"finish_reason":null}]}

data: {"choices":[{"delta":{"content":""},"finish_reason":"stop"}]}

data: [DONE]

Three things to build around:

The first frame is metadata, not content. It is an object with "object": "rag_metadata" and tells you whether the answer will be grounded in your documents, and how many snippets were retrieved. A parser that assumes every frame has choices will crash on it.

The stream terminates with the literal data: [DONE], which is not JSON. Check for it before parsing.

Lines beginning : are heartbeat comments. They keep the connection alive during long generations and carry no data. Ignore them.

Delta frames are OpenAI-shaped, so most existing SSE clients work with minor changes. The final content frame carries "finish_reason": "stop" with empty content.

POST /api/llm/v1/chat/stream has been removed and returns 410 Gone. Use chat/completions with "stream": true, or the threads endpoint below.

Grounding answers in your own documents

Retrieval is not a separate endpoint. Add any of three fields to a normal chat request and the platform retrieves relevant passages from your data before answering:

Field	Meaning
workspaceIds	Ground in everything in these workspaces — the server resolves their collections itself
collectionIds	Ground in specific vector collections
ragScopes	Finer-grained scoping; preferred when you need more than a flat list

curl https://riddlerai.io:4443/api/llm/v1/chat/completions \
  -H "X-API-Key: rdk_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{ "role": "user", "content": "What does our onboarding policy say about laptops?" }],
    "workspaceIds": ["3f9a1c2e-..."]
  }'

Naming a workspace is enough — you do not need to know which collections it contains.

The response tells you whether grounding actually happened: rag_active is a boolean, rag_status explains why when it is false, and rag_snippets carries the passages used. Check rag_active rather than assuming. If a workspace has no indexed content, you will get a fluent answer from model weights alone, which is exactly the failure mode grounding was meant to prevent.

Persistent conversations

POST /api/llm/v1/chat is a separate surface that persists a conversation and returns a conversationId you pass back on subsequent turns. It accepts the same grounding fields.

{
  "message": "What does the knowledge base say?",
  "model": "auto",
  "workspaceIds": ["3f9a1c2e-..."]
}

Use this when you want the platform to keep the history. Use chat/completions when your own application owns the transcript.

Threads

For assistant-style workflows there is a threads API at /api/llm/v1/threads, shaped after the OpenAI Assistants API: create a thread, append messages, then start a run. Streaming runs are at POST /api/llm/v1/threads/{threadId}/runs/stream.

The threads API uses snake_case — assistant_id, workspace_ids, max_tokens — while chat/completions uses camelCase. This catches people out. Check which surface you are calling before naming your fields.

Which surface to use

You want	Use
A single answer, you keep the history	chat/completions
Token-by-token output	chat/completions with stream: true
Answers grounded in your documents	Any of them — add workspaceIds
The platform to keep the history	chat
Assistant-style threads and runs	threads