Streaming
Stream tokens as server-sent events on every chat endpoint, get usage at the end of a stream, and handle errors mid-stream.
Set "stream": true and the answer arrives as server-sent events (text/event-stream) while the model writes it. The gateway passes the events on as the model provider sends them, in the format of the endpoint you called, with model set to the model's public ID. Streaming works on /v1/chat/completions, /v1/messages and /v1/responses.
Chat completions
Each event is a data: line with a chat.completion.chunk. The text is in choices[0].delta.content. The stream ends with data: [DONE].
data: {"id":"chatcmpl-8f1c","object":"chat.completion.chunk","created":1795075200,"model":"openai/gpt-6-astra","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]} data: {"id":"chatcmpl-8f1c","object":"chat.completion.chunk","created":1795075200,"model":"openai/gpt-6-astra","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]} data: {"id":"chatcmpl-8f1c","object":"chat.completion.chunk","created":1795075200,"model":"openai/gpt-6-astra","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} data: {"id":"chatcmpl-8f1c","object":"chat.completion.chunk","created":1795075200,"model":"openai/gpt-6-astra","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":10,"total_tokens":19}} data: [DONE]Usage at the end of a stream
To get the token counts, set stream_options.include_usage to true. The last chunk before [DONE] then has an empty choices array and a usage object, as above. Without it, the gateway leaves usage out of the stream. You are billed the same either way.
stream = client.chat.completions.create( model="openai/gpt-6-astra", messages=[{"role": "user", "content": "Hello"}], stream=True, stream_options={"include_usage": True},)for chunk in stream: if chunk.choices and chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="") if chunk.usage: print("\ntokens:", chunk.usage.total_tokens)Messages (Anthropic format)
On /v1/messages, every event has a name: message_start (with the input tokens), content_block_start, content_block_delta (the text), content_block_stop, message_delta (the output tokens and the stop reason) and message_stop. Usage is always included.
event: message_startdata: {"type":"message_start","message":{"id":"msg_01Xf","type":"message","role":"assistant","model":"anthropic/claude-sonnet-5.5","content":[],"stop_reason":null,"usage":{"input_tokens":9,"output_tokens":1}}} event: content_block_deltadata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}} event: message_deltadata: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":12}} event: message_stopdata: {"type":"message_stop"}Responses
On /v1/responses, events are named after their type, such as response.created, response.output_text.delta and response.completed. The response.completed event carries the usage in response.usage.
Keep-alive lines
While a model is thinking, comment lines (: keep-alive) can arrive between events. SSE clients and the SDKs skip them.
Errors during a stream
Errors before the stream starts come back as a normal JSON error with an HTTP status, such as 402 or 429. Once the stream has started, the status is already 200, so an error arrives as an event:
- Chat completions: a
data:line with anerrorobject. The code isstream_interruptedwhen the connection to the model was lost. /v1/messages: anevent: errorwith an Anthropic error object./v1/responses: anevent: error, or aresponse.failedevent.
data: {"error":{"message":"The model stopped responding. Try again.","type":"server_error","param":null,"code":"stream_interrupted"}}The gateway ends a stream when no data arrives from the model for two minutes. There is no fallback to another route once data has reached you, so retry the request yourself.
How a stream is billed
We bill on the usage the model reports at the end of the stream. If you close the connection early, the gateway stops the request and bills the input and the output written so far. If a stream breaks off without usage, we estimate the tokens from the text, at about four characters per token. A stream that fails before any output costs nothing, except on /v1/messages: there the model reports the input tokens at the very start, in message_start, and from that moment the input is billed. See Pricing & usage.