Transformers as Information Pipelines

Why pipeline thinking beats memorising layers

Architecture blogs often list components — multi-head attention, layer norm, feed-forward blocks — without showing how data flows. Pipeline thinking fixes that: at each stage, ask what shape the data is in, what changed, and what failure modes appear.

This maps directly to production work. When you integrate an embedding API, a reranker, and an LLM, you already run a pipeline. Transformers are the canonical template for how modern NLP stacks modularise those stages.

Stage 1: Tokenisation and embeddings

Integers representing tokens pass through an embedding matrix, producing dense vectors that capture semantic and syntactic hints learned during training. Positional encodings (or rotary embeddings) inject order — attention alone is permutation-sensitive without them.

Quality issues here are subtle: wrong tokenizer for your domain, out-of-vocabulary splits, or normalisation mismatches between training and serving data can degrade downstream layers even when the model weights are fine.

• Embeddings map discrete tokens to continuous space where similarity is meaningful.
• Position information tells the model that "dog bites man" differs from "man bites dog."
• Vocabulary and tokenizer choice are part of the model contract — not interchangeable across checkpoints.

Stage 2: Encoder blocks — building representations

Encoder stacks (BERT-style) read the full input bidirectionally and output contextualised vectors per token. Each self-attention layer lets every token attend to every other token, building representations that depend on surrounding context — ideal for classification, extraction, and semantic search encoders.

Encoders do not generate text token-by-token; they produce rich features you consume in downstream tasks. In RAG systems, the encoder stage often lives inside your embedding model.

• Self-attention complexity grows with sequence length — long documents need chunking or sparse methods.
• Encoder outputs are often pooled (CLS token or mean pooling) for sentence-level similarity.
• Fine-tuning encoders is common for domain-specific retrieval where general embeddings underperform.

Stage 3: Decoder blocks — autoregressive generation

Decoder-only models (GPT-style) use masked self-attention so each position sees only prior tokens during training. At inference, they append one token at a time. This is the dominant architecture for chat assistants and code generators.

Decoder stacks trade bidirectional context for generative flexibility. That is why they excel at open-ended completion but may need retrieval or tools for factual grounding.

• Causal masking enforces left-to-right generation during training.
• KV caching stores prior computations during inference — critical for latency at long outputs.
• Instruction tuning adapts decoder behaviour without changing the fundamental autoregressive loop.

Encoder-decoder and cross-attention

Seq2seq transformers (T5, BART, original translation models) encode the full source sequence, then decode target tokens while attending to encoder outputs via cross-attention. This pattern still appears in summarisation, translation, and some multimodal systems where one modality encodes and another decodes.

When choosing architectures for a product, match pipeline shape to task: encode-only for search and labelling, decode-only for assistants, encoder-decoder when transforming one full sequence into another under tight format constraints.

• Cross-attention lets each decoding step pull relevant parts of the encoded input.
• Many "agent" workflows chain encoder-style retrieval with decoder-style generation.
• Hybrid pipelines are normal — monolithic end-to-end models are the exception in enterprise deployments.