Applied AI for Developers – Part 1: LLM Fundamentals I – From Text to Prediction

Understand how Large Language Models turn text into responses. Text is split into tokens, token IDs become vectors, and the model predicts a distribution for the next token. Select one token, append it to the context and repeat. This module builds the mental model developers need to work effectively with LLMs. Part II explores Transformers, attention, context and limits.

An LLM is a neural network trained on large amounts of text to process and generate language. Given context, it predicts a probability distribution for the next token. A decoding strategy selects one token; that token is appended and the process repeats. Example: “Explain how solar energy works.” Tokenization → neural network → token selection → append and repeat. Unlike a database query that retrieves a stored record, generation uses learned patterns and the supplied context; memorized training content may sometimes be reproduced.

The model is the language engine. A complete AI application combines it with data and knowledge (documents, databases and APIs), retrieval and search, prompting and orchestration, tools and integrations, guardrails and security, a UI or UX, and evaluation and monitoring. The value comes from combining the model with your data, tools, processes and user experience to solve real problems.

Given some text, predict what comes next. The model generates one token at a time. Example context: “The future of renewable energy is”. Illustrative next-token probabilities: bright 0.68, important 0.17, exciting 0.06 and uncertain 0.03, with the remaining probability mass on other vocabulary tokens. Choose a token based on the distribution, append it to context, and repeat until a stop condition. The model predicts likely continuations, not a guaranteed right answer.

Tokens may be whole words, word parts, punctuation, whitespace or other byte sequences. Text is split into tokens, which map to token IDs in the tokenizer vocabulary. A learned embedding lookup maps IDs to vectors, and the Transformer processes them to predict the next-token distribution. The sentence “The future of renewable energy is bright.” and all splits, IDs and probabilities on this slide are illustrative and depend on the tokenizer.

tokenizer = load_tokenizer(model)
tokens = tokenizer.encode(text)
print(tokens)
print(len(tokens))

The same sentence may produce different token splits and counts with different tokenizers. These illustrative examples show 8, 9 and 13 tokens, not measured outputs from named models. The third count matches the 13 units displayed. Vocabulary, algorithm, language, whitespace, rules, model and version affect tokenization. Always test the exact tokenizer you deploy. The code uses a conceptual load_tokenizer(model) helper to encode text and inspect token IDs and token count.

Tokens affect context limits, API costs, latency and retrieval chunking. The 7,500 / 8,192 context budget is illustrative. Budget input and output according to the model. Excess context may cause rejection or truncation depending on the system. For 2,000 input and 1,000 output tokens, total usage is 3,000 tokens; cost is 2 times the input rate plus 1 times the output rate when rates are quoted per 1,000 tokens. Be concise, summarize, filter, control output and monitor usage, costs and latency.

Text becomes tokens and then integer token IDs. Each token ID indexes the tokenizer vocabulary. The IDs shown are illustrative, including the end-token ID. Token IDs are not meanings or semantic distances. Input text is tokenized, IDs map to learned vectors and are processed by the model; generated IDs are decoded to tokens and text. The output fragment renewable energy is corresponds to the displayed IDs 54321, 12589 and 318.

Token IDs select rows in a learned embedding matrix. These embeddings are starting vectors. Transformer layers combine information across token positions to produce contextual representations. The three-dimensional illustration is only a projection of a high-dimensional space, with arbitrary learned dimensions. In river bank and bank account, the same token ID and initial embedding are used for bank, but its contextual representation differs. All IDs and vector values shown are illustrative.

Messages and supplied data form the context: system or developer instructions, user prompt, retrieved context or tool results, and included conversation history. Serialization and tokenization precede Transformer processing. Generation parameters such as temperature, top-p or top-k, output limits and stop conditions separately control decoding. Relevant context can improve reliability without guaranteeing correctness. Only content included in the request consumes context-window tokens.

The final hidden state is projected to vocabulary logits. Softmax turns logits into a categorical distribution summing to one. The illustrative top five probabilities are 0.327, 0.128, 0.097, 0.056 and 0.031. Top-k keeps these candidates, whose total mass is 0.639, and renormalizes before sampling. The rare-token probability 0.000001 equals 0.0001 percent. Next-token likelihood is not factual confidence.

LLMs generate autoregressively, one token at a time. Current input is processed into a next-token distribution. Decoding selects a token, which is appended before the next iteration. Greedy decoding selects the highest probability; top-k and nucleus sampling restrict the candidate set before sampling. Temperature, candidate filters, output limits and stop sequences control generation. A sentence-ending period is not necessarily an end-of-sequence token; generation may continue.

Greedy decoding selects the token with highest probability. Top-k and top-p sampling filter candidates and renormalize before sampling. Temperature rescales logits; repetition penalties adjust scores and vary by implementation. Parameter values and text examples are illustrative, not guarantees. Higher temperature or wider candidate sets do not guarantee creativity or factual accuracy.

Responses are shaped by learned parameters, current context and external systems called through tools. Learned patterns are not a verified knowledge database. Truth, completeness, freshness, source fidelity and consistency are not guaranteed. RAG retrieves relevant material into the model context; it does not automatically make a response correct or current. Provide relevant evidence, use tools for calculations and live data, ask for citations or abstention where appropriate, validate important outputs, and apply permissions, monitoring and evaluation.

The complete flow is input text → tokenization → token IDs → token embeddings → Transformer processing → next-token probabilities → selection → append and repeat → generated output. Token IDs and vector values are illustrative and tokenizer dependent. Training taught statistical patterns; inference processes the current context without normally updating the model's learned weights. The schematic loop reuses an expanding context; practical implementations can cache earlier computations. Use this mental model to understand behavior, evaluate outputs, debug problems and build reliable applications.

Text becomes tokens; token IDs map to learned embeddings; layers build context; the model predicts probabilities; decoding selects the next token; append and repeat; the generated output emerges one token at a time. Predictions depend on both learned parameters and the current context. Outputs are not guaranteed to be correct or coherent. The tutorial series continues with LLM Fundamentals II, embeddings and semantic search, building with LLMs, and production and best practices. Continue to Part II via its linked module title or the presentation toolbar.