Working with LLM APIs A developer application sends a JSON request through an HTTPS API to a hosted language model and receives text and usage metadata. Three learning goals: send a request, shape the context, build a reliable application. APPLIED AI FOR DEVELOPERS · PART 3 Working with LLM APIs In Part 2, we looked inside the model. Now we connect it to an application.
Send a request HTTP, authentication and JSON.
Shape the context Instructions and conversation history.
Build for reliability Validate, measure and handle failures.
YOUR APPLICATION Explain this code… Prompt + instructions Model + output limit { "model": "…", "input": "…", "store": false } Response LLM PROVIDER API endpoint Language model
Inference → output tokens Text + usage metadata
REQUEST HTTPS POST /responses
RESPONSE JSON
TODAY'S ROUTE Request → response → context → reliable application
WORKING WITH LLM APIs · 2026 1 / 16 Recap: From Inside the Model to Your Application Part 2 recap: tokens become embedding vectors; causal attention, position and MLPs build context across decoder blocks; vocabulary scores become next-token probabilities; generation appends a token and repeats with fixed weights. Context is finite and fluent output needs verification. Part 3 connects this mechanism to an application through an API. APPLIED AI FOR DEVELOPERS · PART 3 Recap: From Inside the Model to Your Application Part 2 opened the black box. Four ideas explain what happens behind an LLM API.
1 Tokens → vectors 2 Layers build context 3 Predict the next token 4 Append and repeat bank → Token IDs map to learned embedding vectors. Causal attention + position MLP + residual paths Repeat across decoder blocks. Each position uses its past. bright new unclear Vocabulary scores → probabilities. Illustrative distribution. The future of AI is bright Choose a token; predict again. Inference keeps weights fixed. Context is finite. Your app selects the history and evidence to include. Fluent does not mean verified. Ground important claims and validate the result. PART 3 Your application → LLM API → this mechanism → useful, checked output WORKING WITH LLM APIs · 2026 2 / 16 What Is an LLM API? Your application calls a hosted API over HTTPS. The API service handles authentication, request validation, model routing and usage accounting. The language model runs inference and returns generated output through the API. The API is the interface; the model supplies the learned capabilities. APPLIED AI FOR DEVELOPERS · PART 3 What Is an LLM API? A programmable interface that lets your application use a language model over the network.
YOUR APPLICATION
You own the workflow User interface + app logic
HTTP client Prompt, instructions, model ID
API SERVICE The contract around inference
Authentication Request validation
Model routing Limits + usage Endpoint + schema + operational rules
LANGUAGE MODEL The learned capability
Runs inference on your input
REQUEST HTTPS + JSON RESPONSE Text + usage API = interface + service How you access and operate the capability.
Model = learned parameters + inference What generates the output from your context.
A chat product adds a ready-made user interface. An API lets you build your own. WORKING WITH LLM APIs · 2026 3 / 16 The Anatomy of an API Request Five parts of an OpenAI Responses API request: HTTPS endpoint and POST method; Authorization header with a runtime API key; model selection; user input; and output and storage controls. Matching numbered labels connect the explanations to a JSON request example. No real key is included. APPLIED AI FOR DEVELOPERS · PART 3 The Anatomy of an API Request One HTTPS request combines a destination, credentials and a structured JSON body.
1 Destination POST starts a generation request. Use the provider's HTTPS endpoint. 2 Authentication The header carries your API key. Supply it at runtime; never hard-code it. 3 Model Choose the capability to call. The model must be accessible to your key. 4 Input Send the user's prompt or input items. Add instructions and context when needed. 5 Controls Bound output; choose storage behavior. Supported options depend on the model. HTTP REQUEST · OPENAI RESPONSES API 1 POST https://api.openai.com/v1/responses 2 Authorization: Bearer RUNTIME_KEY Content-Type: application/json { 3 "model": "gpt-5.4-mini", 4 "input": "Explain an API in 3 sentences.", 5 "max_output_tokens": 400, "store": false } NEXT: SEND IT Enter a temporary key and prompt on slide 5 — then inspect the response.
WORKING WITH LLM APIs · 2026 4 / 16 APPLIED AI FOR DEVELOPERS · PART 3
Your First LLM API Call LIVE DEMO 02 WRITE YOUR REQUEST
Prompt Send → Clear Check browser CORS
03 · OpenAI Response POST /v1/responses → JSON → text
The model's response appears here. Input tokens: — · Output tokens: — · Time: —
Direct browser → OpenAI. No proxy. CORS has not been checked.
Key stays in this page's runtime only; Clear removes it and aborts the request. Browser tools and page scripts can access it. Use a temporary key and revoke it afterwards. Prompt is sent to OpenAI; store:false does not guarantee zero data retention.
WORKING WITH LLM APIs · 2026 5 / 16 Understanding the API Response A simplified raw Responses API JSON object with three highlighted areas: completion status, generated text inside message output content, and input/output token usage. Extract text by item type, check incomplete responses and refusals, and distinguish server token accounting from browser elapsed time. All example values are illustrative. APPLIED AI FOR DEVELOPERS · PART 3 Understanding the API Response The response is a structured result — generated text is only one part of it.
RESPONSE BODY · SIMPLIFIED JSON { "status": "completed", "output": [{ "type": "message", "content": [{ "type": "output_text", "text": "An API lets apps communicate." }] }], "usage": { "input_tokens": 24, "output_tokens": 38 } } 1 2 3 1 Check the outcome Read status before trusting the result. Handle incomplete output, refusals and errors. 2 Extract the generated text output[] → message → content[] → output_text Collect matching text parts; render them as text. Do not assume output[0] is always a message. 3 Read actual token usage Input: 24 tokens Output: 38 tokens Elapsed time is measured separately by the browser. RAW REST ≠ SDK HELPER The SDK's output_text helper collects text; raw fetch returns typed output items. Illustrative JSON and token counts; not a measured demo response.
WORKING WITH LLM APIs · 2026 6 / 16 Instructions and Conversation Context Instructions define application behavior. Input supplies the user question and relevant context. A JSON example separates top-level instructions from input messages containing user and assistant history. Retrieved documents are untrusted data, and important results need validation. APPLIED AI FOR DEVELOPERS · PART 3 Instructions and Conversation Context Tell the model how to behave — and supply the information it needs for this request.
01 · INSTRUCTIONS Define application behavior. Task, audience, constraints and format. Keep guidance explicit and consistent.
02 · INPUT + CONTEXT Supply the question and evidence. Current user message Relevant earlier user / assistant turns Documents or tool results, when needed Your app decides what to include.
RESPONSES API · REQUEST BODY EXCERPT { "instructions": "Explain APIs clearly. Be concise.", "input": [ { "role": "user", "content": "What is an API?" }, { "role": "assistant", "content": "An API lets applications communicate." }, { "role": "user", "content": "Now give me a GET request example." } ] } KEEP THE BOUNDARY Retrieved text is untrusted data. Keep secrets out of prompts. Instructions guide behavior; validate important outputs in your application.
WORKING WITH LLM APIs · 2026 7 / 16 Controlling Model Behavior Choose a suitable model, bound output, and understand temperature: low values favor likely tokens with less variation; high values allow more varied choices. Temperature is an API parameter only for supporting models and configurations. Lower temperature does not guarantee identical answers. Define acceptance criteria, evaluate representative cases and validate important results. APPLIED AI FOR DEVELOPERS · PART 3 Controlling Model Behavior Use the controls your model supports — then measure whether the result meets your needs. 1 2 3 Choose the capability MODEL Task-appropriate model Match model strengths to your task. Compare quality, latency and cost. Evaluate on representative examples. Bound the output max_output_tokens Set an explicit upper limit. Request a schema when supported. Validate the returned structure. A token limit may truncate the answer. Temperature API parameter: temperature Low · e.g. 0.2 Favors likely tokens; less variation. High · e.g. 0.8 More varied token choices. Only where supported by the model and its selected configuration. EXPECT VARIATION Lower temperature does not promise identical answers. API BODY EXCERPT { "temperature": 0.2 } Not a truth or quality control. Check support; omit if unsupported. Slide 5 does not set it yet. WORKING WITH LLM APIs · 2026 8 / 16 Multi-Turn Conversations A concrete second request resends instructions, the first user question, the assistant answer and the new user question. A colored JSON excerpt matches the conversation shown on the left. This illustrates application-managed context; provider state is another supported approach. The live demo sends independent requests. APPLIED AI FOR DEVELOPERS · PART 3 Multi-Turn Conversations For the follow-up, your app sends the earlier exchange together with the new question.
THE CONVERSATION
USER · FIRST QUESTION What is an API?
ASSISTANT · FIRST ANSWER An API lets applications communicate.
USER · FOLLOW-UP Now show me a GET request example.
The next call receives all three messages. No automatic memory is assumed.
SECOND REQUEST · BODY EXCERPT { "instructions": "Explain APIs clearly. Be concise.", "input": [ { "role": "user", "content": "What is an API?" }, { "role": "assistant", "content": "An API lets applications communicate." }, { "role": "user", "content": "Now show me a GET request example." } ] } ANOTHER OPTION Use supported provider state, e.g. previous_response_id; resend needed instructions. Check retention and context growth. Slide 5 still sends independent requests.
WORKING WITH LLM APIs · 2026 9 / 16 Streaming Responses Illustrative timelines compare full output shown at completion with streaming text deltas rendered as they arrive. Streaming improves perceived responsiveness, not necessarily total time. Set stream true, parse server-sent events, join deltas and handle completion, cancellation and errors. Network chunks are not event boundaries. Slide 5 remains non-streaming. APPLIED AI FOR DEVELOPERS · PART 3 Streaming Responses Show text as it arrives, instead of waiting for the entire response to finish.
SAME TASK · TWO WAYS TO DISPLAY THE OUTPUT
NON-STREAMING Request Waiting for completion… An API lets apps communicate.
STREAMING An API lets apps communicate. Completed + final usage ↑ First visible text ↑ Completion Illustrative timing only. Earlier visible output does not guarantee a shorter total generation time. 01 · REQUEST { "stream": true } Enable streaming in the JSON body. 02 · READ + RENDER Parse SSE events; append text deltas. Network chunks ≠ complete events. 03 · FINISH SAFELY Handle completion and errors. Support cancellation; render as text. OUR LIVE DEMO Slide 5 uses a non-streaming request. Streaming requires additional event-handling code. WORKING WITH LLM APIs · 2026 10 / 16 Tokens, Latency and Cost Tokens, latency and cost. Different models have different input and output rates; the same token usage can have different costs. Output is usually more expensive per token, but long input can cost more in total. Sum token count times applicable rate across categories. Compare total cost per task with quality and latency. Token counts, cached input and reasoning affect accounting. Chart proportions are illustrative. APPLIED AI FOR DEVELOPERS · PART 3 Tokens, Latency and Cost Measure the whole request: how much it uses, how long it takes and what it costs. 01 · TOKENS Count input and output INPUT Instructions History Prompt OUTPUT Generated tokens Read usage metadata from the API. Tokens are not the same as words. 02 · LATENCY Follow the elapsed time Network → input processing → model work → output transfer Streaming: first text ≠ completion. Demo time: browser wall-clock duration. 03 · COST Different models, different prices Same token usage ≠ same cost. Each model has its own input / output rates. Output usually costs more per token. But long input can cost more in total. Cached input has separate rates. Cost = token count × applicable rate Sum categories; use current model pricing. TOTAL COST MATTERS Input cost + output cost; divide token counts by 1M when using per-million rates. COMPARE PER TASK Quality + latency + total cost Token counts can vary by model. Cached input and reasoning affect accounting; demo timing is not a billing quote.
WORKING WITH LLM APIs · 2026 11 / 16 Handling Errors and Limits Diagnose errors before retrying. Invalid requests, model access and authentication need configuration fixes. Rate-limit errors may warrant bounded exponential backoff with jitter, but quota errors need account action. Service errors may be temporary. A timeout has an unknown completion outcome. Browser fetch failures can arise from network, CORS or browser policy. Never expose keys in logs or errors. APPLIED AI FOR DEVELOPERS · PART 3 Handling Errors and Limits Classify the failure first. A retry only helps when the cause is temporary.
SIGNAL WHAT TO CHECK APPLICATION ACTION 400 / 404 Request, parameters, endpoint or model access Fix the request or configuration. 401 / 403 Key validity, project permissions and access Fix credentials / access; avoid blind retries. 429 Rate limit or exhausted quota? Read the error. Rate limit: backoff. Quota: account action. 5xx Temporary provider-side failure Use bounded retries with backoff + jitter. TIMEOUT The request may still have completed Cancel locally; check outcome before repeating. BROWSER FETCH FAILED? Network, CORS or browser policy may block the call. JavaScript cannot always distinguish these causes. RETRY POLICY · ILLUSTRATIVE Wait → longer wait → stop after a bounded limit Add jitter; respect provider retry guidance and deadlines. SAFE FAILURE Show useful messages. Never log API keys. Keep request IDs for diagnosis.
WORKING WITH LLM APIs · 2026 12 / 16 Choosing the Right Model Define task requirements, test candidate models on the same representative examples and compare quality, latency and total cost per successful task. A conceptual comparison matrix uses placeholders rather than benchmark claims. First satisfy quality and operational constraints, then optimize cost and speed. Model access and supported features depend on the account and provider. APPLIED AI FOR DEVELOPERS · PART 3 Choosing the Right Model Choose for your task — using measured results, rather than the model name alone.
01 · DEFINE THE REQUIREMENTS What must the model deliver? Quality and acceptable failure rate Response-time target Budget per completed task Output format and tool support Privacy and deployment constraints 02 · TEST CANDIDATES ON THE SAME TASKS Include normal inputs, hard cases and expected failures. Candidate Quality Latency Cost / task Model A Measure Measure Measure Model B Measure Measure Measure Model C Measure Measure Measure Template only — no invented scores or universal model ranking. 03 · CHOOSE Meet the quality threshold first; then optimize total cost and response time. CHECK BEFORE DEPLOYING Model access + parameter support + limits → re-evaluate when things change
WORKING WITH LLM APIs · 2026 13 / 16 Different API Providers Three access patterns: direct hosted APIs such as OpenAI, Anthropic and Google; cloud platforms such as Microsoft Azure and Amazon Bedrock; and self-hosted model serving. Compare authentication, schemas, streaming, tools, structured output, retention and usage accounting. Similar HTTP interfaces do not imply identical behavior. APPLIED AI FOR DEVELOPERS · PART 3 Different API Providers The request travels over HTTP — but the service contract and operating model can differ. 01 · DIRECT HOSTED API 02 · CLOUD PLATFORM 03 · SELF-HOSTED Call the provider directly. Examples: OpenAI, Anthropic, Google Provider manages model serving. Use its credentials, schema and billing. Use a cloud access layer. Microsoft Azure · Amazon Bedrock Cloud identity and deployment options. Model availability and endpoints vary. Operate the serving stack. Your infrastructure + a model server Own hardware, scaling and monitoring. Check model license and capabilities. CHECK BEFORE SWITCHING Authentication + schemas Streaming + tools + output formats Retention + usage accounting KEY IDEA A similar HTTP interface does not guarantee identical behavior. Test your integration. WORKING WITH LLM APIs · 2026 14 / 16 Building a Simple LLM Application Build a code explainer: a user submits a snippet through the interface; the backend validates input, adds instructions and calls the model with server-held credentials; the response is validated and displayed. Production adds access control, timeouts, budget limits, monitoring and evaluations. The slide 5 browser-key demo is a temporary teaching setup. APPLIED AI FOR DEVELOPERS · PART 3 Building a Simple LLM Application Example: a code explainer that turns a submitted snippet into a clear explanation. 01 · USER INTERFACE const total = items.reduce( (sum, x) => sum + x, 0); Collect the code snippet. Show loading, answer or error. Render generated text safely. 02 · YOUR BACKEND Validate input + add instructions Choose model + bound output Call the provider with server-held key Check the response before returning it Own the workflow and operational controls. 03 · LLM PROVIDER Instructions + snippet → inference → explanation Generate the explanation. Return output + usage metadata. Your app validates important results. BEFORE PRODUCTION Access + input limits Timeouts + budgets Logs without secrets Quality evaluations TEACHING DEMO Slide 5 uses a temporary browser key. A production app keeps credentials on the server. YOUR EXERCISE Build the code explainer; test useful answers, invalid input and provider failures. WORKING WITH LLM APIs · 2026 15 / 16 Key Takeaways and What's Next Four takeaways: understand the API contract, design instructions and context, measure quality latency and cost, and build for failure. Next lesson, Part 4: Prompting and Structured Output, covering system prompts, few-shot prompting, JSON outputs, schemas and reliable prompt design. APPLIED AI FOR DEVELOPERS · PART 3 Key Takeaways & What's Next One request is easy. A reliable application needs engineering. 01 Understand the contract Endpoint, authentication and JSON. Read typed output; handle completion. 02 Design the context Give clear instructions and relevant history. Treat retrieved content as untrusted data. 03 Measure the system Compare quality, latency and cost per task. Model prices and supported controls differ. 04 Build for failure Validate output and protect credentials. Handle limits, errors and uncertain outcomes. NEXT · PART 4 Prompting & Structured Output System prompts · few-shot prompting · JSON outputs · schemas · reliable prompt design TAKE IT WITH YOU Start with one useful task. Define success. Test the failures as well as the happy path.
WORKING WITH LLM APIs · 2026 16 / 16 Arrow keys: navigate · Home / End: first / last slide · Download PDF: saved edition of all 16 slides · Live demo on slide 5 is interactive in the web edition only.