Work in progress. This note is a living summary of my LLM fundamentals study material. Sections will be added and refined over time.

Personal summary of the core ideas behind how LLM-based systems work: what actually goes into a request, why prompt order matters, how the context window budget works, what to do when prompting alone is not enough, and what the model file physically is.

# Anatomy of a Chat Request

Every chat request is assembled from five components, and your application code is the assembler:

1. System prompt - standing instructions: persona, format rules, refusal policy.
2. Tool definitions - JSON schemas for callable functions, sent on every request so the model can produce a valid call.
3. Prior conversation turns - the model is stateless; the “memory” the user feels is your code replaying history on every call.
4. Retrieved chunks - RAG snippets pulled from a vector store at query time.
5. New user message - what the user just typed.

Key realizations:

• The messages array is only a serialization format. The provider concatenates entries, stitches in role markers (like <|im_start|>system), and tokenizes everything into one flat stream. The model only ever sees tokens.
• Tool schemas are not free. Three function definitions cost a few hundred tokens on every request, every turn.
• Retrieved chunks are big. A handful of doc snippets is usually the largest single source in a request, so RAG quality beats RAG quantity.
• The context window must hold all components plus the model’s reply - anything you ship in is something the reply cannot use.

# Order Matters: Lost in the Middle

Attention is not uniform across the prompt. Accuracy is high for facts near the start, high near the end, and drops sharply for facts in the middle - a pattern known as “lost in the middle.”

Practical implications:

• Standing instructions go at the top (strongest attention zone).
• The user’s new question goes at the bottom (the other strong zone).
• Reference material in the middle is fragile. Inject RAG chunks just before the current question, not buried under old conversation turns.

Debugging tip: when the model answers wrong despite a correct retrieved chunk, check where the chunk landed before blaming retrieval. Print the assembled prompt. Look at the position. Move it.

# The Context Window Is a Shared Budget

The advertised “128k context” covers input + output together. A prompt that eats 120k tokens leaves at most 8k for the reply.

This compounds into why long conversations get expensive:

• The replay tax: every prior turn rides along on every future call.
• Attention compute grows with the square of sequence length - O(N^2).

# The Five Exits (When Prompting Runs Out)

When prompt tweaking stops paying off, there are exactly five ways out:

1. Retrieval (RAG) - the model needs facts it never saw. No prompt rewrite invents facts.
2. Fine-tuning - behavior learnable from examples, not from a handful in context (style, domain judgment, structured-output reliability).
3. A smaller specialized model - a 70B chat model is the wrong tool for a single-purpose classifier; smaller + fine-tuned is faster, cheaper, often more accurate on the narrow job.
4. Agentic orchestration - split one overloaded prompt into a planner plus specialized sub-prompts.
5. Deterministic post-validation - for guarantees the model cannot give (JSON validity, schema conformance, numeric ranges). Generate, validate, retry. Reliability comes from the loop around the model.

# Latency: TTFT vs Total

LLM latency is two numbers, not one:

• TTFT (time-to-first-token) - roughly prefill time (processing all input tokens in parallel) plus one decode step.
• Total latency - TTFT plus the rest of the decode loop (output tokens generated one at a time, roughly 40-100 tok/s on hosted models).

They scale independently:

• TTFT scales with input length (an 8k prompt ~700 ms prefill; a 64k prompt ~15 s).
• Total scales with output length (a 200-token reply ~4 s of decode regardless of prompt size).

Which SLO matters depends on the audience:

• Human reading in real time - TTFT is the SLO. Use streaming; target p95 TTFT under ~1 s.
• Program consuming the output - total latency is the SLO. Skip streaming; optimize output length.

Track p95 for both - the average hides the tail, p99 is noise.

# What the Weights Actually Are

A trained model is just a file: a flat array of floating-point numbers plus a small JSON describing them. No executable, no runtime. Think of it like an mmap’d index file: loaded once, queried many times, where the “query” is pushing a token sequence through.

Size is one multiplication: parameters x bytes per parameter.

Inside the file of a modern decoder-only transformer (rough split):

• Feed-forward (MLP) ~66% - where most learned patterns (“knowledge”) live.
• Attention projections ~31% - Q, K, V, O matrices per layer.
• Embedding ~3% - vocabulary lookup.

This split comes back in fine-tuning: LoRA selectively retrains attention projections because they are the cheaper handle.

# The Mental Toolkit

Four pieces of mental machinery cover most real conversations:

• The shapes - token IDs are integers; an embedding is a ~4,000-float vector; attention is an N x N matrix; output is a probability distribution over a 30k-200k entry vocabulary.
• The cost functions - attention is O(N^2) in sequence length (why context windows are bounded); feed-forward is O(N * dim^2) (why bigger models cost more per token); KV cache is O(N * layers * dim * 2 bytes) (why long contexts crush throughput).
• The vocabulary - tokens, embeddings, attention, KV cache, prefill, decode, softmax, temperature, top-p, RAG, fine-tuning, LoRA, hallucination, prompt injection.
• The threat model - the five categories of “the LLM did something we did not want”: hallucination, prompt injection, jailbreaks, context overflow, cost overrun.

More sections to come as the study continues (retrieval, fine-tuning, serving math, agents, evaluation).