AICASSINDRA · LEARN BY EXPERIMENTING

Transformer math & SLMs labs

Explore the interactive side of AICassindra. Read a topic, test an idea, and follow the results.

Choose a topic, explore the model, and compare what changes. Each experiment explains its assumptions and limitations.

Browse the collection below.

Transformer math & SLMs

Autoregressive generation: one decision at a time

Predict → select → append → repeat. Inspect how each output becomes the next input.

Interactive lab
Transformer math & SLMs

Backpropagation: the Chain Rule

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Beam Search: Pruning an Explicit Probability Tree

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Blocked multiplication: inspect the traversal order

Keep the arithmetic fixed and change the order in which matrix tiles are visited.

Interactive lab
Transformer math & SLMs

Cache Hits, Conflict Misses and Reuse

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Compare greedy, top-k and top-p decoding

Hold the fixture fixed while changing candidate selection and sampling policy.

Interactive lab
Transformer math & SLMs

Complete Transformer Block: Executable Tensor Trace

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

CPU SLM Stack: Trace a Dense Kernel

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Cross-Entropy Loss Surface

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

DataLoader Pipeline: Bounded Prefetch and Compute Stalls

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Dot Product Geometry (2D)

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

DPO Loss and Reference-Relative Preference

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Embedding lookup: track IDs to stored rows

Trace each coordinate copied from a six-token embedding table.

Interactive lab
Transformer math & SLMs

End-to-End Recipe: Train, Export, Validate, Serve

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Feed-Forward Expansion with Actual Activations

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

FlashAttention Tiling: Inspect Online Softmax

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Forward and Backward Arithmetic Counts

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

FP32, BF16 and FP16: inspect range and rounding

Enter an actual value. Inspect its sign, exponent and fraction bits, then pause at the boundaries where a format rounds, underflows or overflows.

Interactive lab
Transformer math & SLMs

From logits to a selected token

Inspect every arithmetic stage between an unnormalized score and a token.

Interactive lab
Transformer math & SLMs

Gradient Accumulation and Unequal Microbatches

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Gradient Clipping in Action

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Inference Latency: Conditional Resource Bounds

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

KV cache growth: count every stored position

Follow context growth and reconcile cache bytes with model dimensions.

Interactive lab
Transformer math & SLMs

LayerNorm and RMSNorm: inspect row statistics

Compare mean-centering with root-mean-square scaling using the same input.

Interactive lab
Transformer math & SLMs

LayerNorm Statistics

Calculate population statistics, epsilon-stabilized normalization and explicit per-feature affine output.

Interactive lab
Transformer math & SLMs

Learning-rate schedule: warmup and exact decay

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Loss curves: inspect evidence before diagnosis

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Low-rank adapters: compute the base and the update

Inspect XW and (alpha/r)(XA)B separately, then add their actual outputs.

Interactive lab
Transformer math & SLMs

Matrix multiplication: follow every accumulator

Choose a row and column. Watch their products build one output entry.

Interactive lab
Transformer math & SLMs

MoE gate arithmetic and capacity accounting

Inspect actual top-k decisions, expert capacity and weighted scalar outputs.

Interactive lab
Transformer math & SLMs

Multi-Head Split + Concat

One big projection reshapes into h heads then back.

Interactive lab
Transformer math & SLMs

Multi-token prediction heads

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Optimizer steps: SGD, momentum and Adam

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Perplexity Calculator

Audit target-token likelihoods, counted losses and perplexity without invented model rankings.

Interactive lab
Transformer math & SLMs

Positional Encoding Curves

Sinusoidal at different dimensions = different frequencies.

Interactive lab
Transformer math & SLMs

Q4_K Block Layout

Quantize 256 weights into one Q4_K super-block, pack every byte the way llama.cpp does, and see how one outlier coarsens only its own sub-block.

Interactive lab
Transformer math & SLMs

Repetition penalty: inspect signed-logit changes

Apply a sign-aware penalty only to tokens already present in the context.

Interactive lab
Transformer math & SLMs

Residual Gradient Flow

Compute signed derivatives through plain and residual scalar chains.

Interactive lab
Transformer math & SLMs

RoPE Extension Strategies

Compute every rotary frequency before and after context extension and count the pairs that would rotate through angles the model never saw in training.

Interactive lab
Transformer math & SLMs

SIMD Register — AVX-512 + AMX

Run one u8 x s8 dot product through scalar, AVX-512, AVX2, VNNI and AMX instruction shapes and watch lanes fill, saturate and accumulate.

Interactive lab
Transformer math & SLMs

SLM Architecture Comparison

Put two published small-model configurations side by side and compute what their choices cost in KV cache at a context length you pick.

Interactive lab
Transformer math & SLMs

SLM Parameter Breakdown

Count every weight of a published small model from its config, split into embedding, attention, FFN and norms, and check it against a memory budget.

Interactive lab
Transformer math & SLMs

Speculative Decoding Acceptance

Watch draft tokens accepted with probability min(1, p/q) or replaced from the residual, and check that the output still follows the target model.

Interactive lab
Transformer math & SLMs

Stable softmax and temperature

Inspect exponentiation and normalization without overflow-prone direct exponentials.

Interactive lab
Transformer math & SLMs

Tied embeddings: reuse the output weights

Look a token up in an 8-token embedding table, score every token with the same matrix, and size what tying saves in real models.

Interactive lab
Transformer math & SLMs

Tokenizer Compression Comparison

Train a byte-level BPE tokenizer on a short English text right here, encode your own text with it, and see why other languages and code cost more tokens.

Interactive lab
Transformer math & SLMs

Training memory allocation ledger

Explore the controls, compare outcomes and record your observations.

Interactive lab
Transformer math & SLMs

Weight Initialization Distributions

Push a batch of random inputs through a stack of freshly initialised layers and watch whether the activations keep their scale, vanish or explode.

Interactive lab
Transformer math & SLMs

Weight Layout on Disk (GGUF / SafeTensors)

Lay out a five-tensor model in GGUF v3 or safetensors and read the exact byte offsets of its header, metadata, tensor table, padding and data.

Interactive lab