What you'll learn
Quick Answer
An LLM API takes text, splits it into tokens - chunks of roughly four characters in English - and predicts the next token over and over. The context window is the maximum number of tokens for the prompt and the response combined, not each separately. Temperature controls how random the next-token pick is: near 0 is almost deterministic, higher is more varied. Streaming sends tokens as they are generated so the user sees output sooner, though total time and cost are unchanged.
Tokens, not words
Models do not see characters or words. A tokenizer splits text into tokens - common sub-word chunks from a fixed vocabulary of tens of thousands of entries. A frequent word is one token; "tokenization" might be one or two; a rare long word is several. Whitespace and punctuation are usually part of a token too.
A rough guide for English is about four characters, or 0.75 words, per token, so 100 tokens is roughly 75 words - but treat that as an estimate, not a rule. Other languages often tokenize far less efficiently, using several times more tokens for the same meaning, and so does source code with its brackets and indentation. You are billed per token, input and output, so the count is real money.
Gotcha: a string that looks short can be long in tokens. A UUID, a base64 blob, or densely nested JSON can be two to three times more tokens than plain prose of the same character count. Measure with the provider's tokenizer library or token-count endpoint before you assume a payload fits.
Next-token prediction
At each step the model produces a score for every token in its vocabulary - tens of thousands of numbers - which are turned into probabilities for how likely each token is to come next. The API picks one according to your sampling settings, appends it to the input, and runs the whole thing again. Generating 500 tokens means 500 passes through the model.
This loop is why output arrives at a roughly steady pace, why longer responses cost more and take longer, and why the model cannot "think ahead" reliably. It has no plan, only a running guess, so it can open a JSON object and never close it, or start a numbered list of ten items and stop at seven when it hits the output limit. The writing is the thinking - there is no separate step where it works out the answer and then types it.
The context window is shared
Every model has a maximum context length in tokens. The detail that catches people: that budget covers the prompt and the generated response together. If a model's window is 128,000 tokens and your prompt is 120,000, the model has only about 8,000 left to answer, no matter how high you set the output limit.
Overflow the window and the API either rejects the request outright or silently drops the oldest tokens, so your carefully written system prompt quietly falls off the front. A concrete failure: you paste a long transcript, ask for a summary, and get a reply that stops mid-sentence with a finish reason of "length" - the prompt ate the budget. Track prompt size, set the output limit deliberately rather than leaving it at the maximum, and for long chat sessions trim or summarize old turns before they push you over. Windows vary enormously between models, from a few thousand tokens to over a million.
Temperature and other sampling knobs
Temperature reshapes the probability distribution before a token is drawn. At 0 the API takes the most likely token nearly every time - good for extraction, classification, and code, where you want the same answer twice. Raise it and lower-probability tokens get picked more often, which reads as more creative or more erratic depending on your point of view.
The common range is 0 to 2 for some vendors and 0 to 1 for others; a few newer models have narrowed or fixed the range, so check the current reference rather than assuming. top_p (nucleus sampling) is a related knob that restricts the candidate pool to the smallest set of tokens whose probabilities sum to p. Vendors generally suggest tuning one of temperature or top_p, not both at once.
Gotcha: temperature 0 is not a guarantee of identical output. Floating-point arithmetic, batching, and mixture-of-experts routing still introduce small run-to-run variation, so do not build a system that assumes byte-identical responses.
Streaming
Without streaming, the call returns once the whole response is ready - several seconds for a long answer. With streaming, the server holds the connection open and sends each chunk as it is generated, usually as Server-Sent Events. The user sees the first words in a fraction of a second, which is why chat UIs feel responsive and why streaming is worth the wiring for anything a person watches live.
What streaming does not do: make total generation faster, or reduce cost - you still pay for every output token. It also makes some things harder. You do not know the full response, or the final token count, until the stream ends. If you are parsing structured output you are accumulating a partial string that is not valid JSON until the last chunk arrives. And an error can occur halfway through, after you have already shown the user text, so your UI needs a way to walk that back.
What you actually pay for
Cost is per token, split into input and output, and output tokens are usually priced several times higher than input. That changes how you optimize: a long prompt is cheaper than a long answer of the same length, so it often pays to send more context and ask for a terse response.
Two line items reduce the input cost. Prompt caching lets you reuse an unchanged prefix - a big system prompt, a fixed set of instructions - across calls at a large discount, as long as the prefix is byte-identical each time. Batch endpoints run non-urgent work asynchronously at a discount. On the output side, the levers are a lower maximum-tokens limit, an instruction to be brief, and stop sequences so the model does not ramble past the point you cared about. Log the usage object every provider returns - input, output, and cached token counts - so you can see where the money goes instead of guessing.