Streaming

Pass stream=True and create() returns an iterator of OpenAI-compatible chunks instead of a finished completion. Trace relays provider chunks as they arrive and keeps only a bounded redaction tail so an identifier split across chunks is not exposed. The final safety review runs after generation and can end a stream with an error frame after output has already started.

from trace_sdk import Trace

client = Trace()

for chunk in client.chat.completions.create(
    messages=[{"role": "user", "content": "Count from one to five."}],
    stream=True,
):
    print(chunk.text, end="", flush=True)
print()

chunk.text is Trace's shortcut: the text in this chunk, or "" when there is none. The mirrored path is chunk.choices[0].delta.content, which can be None and, on the final frame, has no choice to index at all.

The shape of a stream#

Frames arrive in this order:

1. An opening frame whose delta sets role and usually has empty content.
2. Many content frames.
3. A frame with an empty delta and a finish_reason.
4. A final frame with no choices at all and a usage object.

A loop that indexes chunk.choices[0] unconditionally crashes on that last frame. Either use chunk.text, or guard:

for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")

Getting the cost of a stream#

The last frame carries usage, including credits_spent:

text = ""
spent = None

for chunk in client.chat.completions.create(
    messages=[{"role": "user", "content": "Explain gravity in two sentences."}],
    stream=True,
):
    text += chunk.text
    if chunk.usage:
        spent = chunk.usage.credits_spent

print(text)
print(f"{spent} credits")

An approved stream ends with this usage frame. If the stream fails, it raises before yielding a usage frame, but Trace still settles the primary generation against the account. You can read the settled entry from the credits usage endpoint.

Finish reason#

for chunk in stream:
    if chunk.choices and chunk.choices[0].finish_reason:
        print("\nfinished because:", chunk.choices[0].finish_reason)

"length" means max_tokens cut the reply off mid-sentence.

Errors during a stream#

Two failure points, and they raise at different moments:

Before the stream opens — a bad model, an empty balance, a rate limit. create() itself raises, before you get an iterator:

from trace_sdk import InsufficientCreditsError

try:
    for chunk in client.chat.completions.create(messages=[...], stream=True):
        print(chunk.text, end="")
except InsufficientCreditsError as error:
    print("Balance:", error.balance)

After the stream starts — the provider can drop the connection, or the final safety check can block the reply or remain unavailable. The iterator may already have yielded text when that happens, then raises with the error frame. The credits reserved for the primary generation are still settled.

Because the exception can come out of the for loop as well as out of the call, wrap the loop, as above. A stream that fails its final review can include text collected before the review completed, so callers should discard the partial result when they catch the error.

Streaming and tools#

A model can stream tool calls, arriving as fragments across several deltas — chunk.choices[0].delta.tool_calls, with the arguments string built up piece by piece. If you are learning tool calling, leave stream out first: reassembling partial JSON is a separate problem from understanding the tool loop, and a non-streaming call hands you whole, parsed calls. See Tool calling.

The wire format#

Under the covers this is server-sent events: lines of data: {...}, terminated by data: [DONE]. The SDK parses that for you and does not yield the sentinel. If you are inspecting the gateway with curl, that is what you will see.