AICASSINDRA · LEARN BY EXPERIMENTING

Large language models labs

Explore the interactive side of AICassindra. Read a topic, test an idea, and follow the results.

Choose a topic, explore the model, and compare what changes. Each experiment explains its assumptions and limitations.

Browse the collection below.

Large language models

A/B capacity: sample size changes operational exposure

Compare control and candidate traffic volume alongside accepted outcomes.

Interactive lab Large language models

A2A architecture: isolate one child-agent failure

Fan one request out to three remote agents, break the stock agent, and see whether the join keeps or throws away the healthy results.

Interactive lab Large language models

A2A interop: inspect the selected interface contract

Compute ordered interface selection, tenant propagation and version checks from a local card.

Interactive lab Large language models

A2A load balancing: keep a local task on its owning server

Send eight A2A requests from two clients through a load balancer and see whether each follow-up lands where its task state lives.

Interactive lab Large language models

ADK application overview: check authority at the tool boundary

Let an agent propose six tool calls for a customer and see which ones a system instruction, a before-tool callback or per-tool checks actually stop.

Interactive lab Large language models

ADK code executor: reject a forbidden import

Scan an editable script for static and computed imports, apply an allowlist or denylist, and see whether a file-system effect could run.

Interactive lab Large language models

ADK Java and A2A: validate a discovered peer before handoff

Inspect a trusted Agent Card and one whose endpoint origin differs.

Interactive lab Large language models

ADK Java core: keep session state tenant-scoped

Write one tenant fact into ADK-style session state and see which sessions, users and tenants can read it under each key prefix.

Interactive lab Large language models

After-tool callback: redact returned content

A customer lookup returns more than the agent needs. Redact it in an after-tool callback and count what still reaches the model, and what the agent can no longer answer.

Interactive lab Large language models

Agent A/B assignment: expose the candidate deliberately

Route a fixed request cohort into control and candidate agents, then inspect the observed answers.

Interactive lab Large language models

Agent accessibility: preserve keyboard focus through an error dialog

Open an error dialog from the keyboard, follow every Tab stop, close it, and see where focus and announcements end up.

Interactive lab Large language models

Agent alerting: open an incident only after enough failures

Watch an error threshold become an alert on a numbered request stream.

Interactive lab Large language models

Agent architecture: propagate trace context through a failing tool

Follow one user request through gateway, agent, model, a tool on a worker thread and an inventory service that times out, and see whether the failure stays in the user’s trace.

Interactive lab Large language models

Agent as tool: cap a delegated child’s permissions

A support agent calls a summarizer agent as a tool. Choose the credential it carries and watch which steps, including injected ones, actually run.

Interactive lab Large language models

Agent batching: throughput grouping versus request wait

Group twelve request arrivals into batches of two or four.

Interactive lab Large language models

Agent billing: one invoice from validated usage

Follow individual meter records into invoice lines, one credit application and a separate collection/refund ledger.

Interactive lab Large language models

Agent cancellation and resumption: restore from a checkpoint

Interrupt an eight-step task and inspect exactly which work replays.

Interactive lab Large language models

Agent Card discovery: follow the request

Follow the request, inspect the card, and see exactly why discovery succeeds or stops.

Interactive lab Large language models

Agent Card discovery: invalid and unavailable cards

Follow the request, inspect the card, and see exactly why discovery succeeds or stops.

Interactive lab Large language models

Agent Card discovery: trace changes over time

A remote agent changes its Agent Card three times in 45 minutes; follow five client calls through a cached card and see which ones go stale.

Interactive lab Large language models

Agent Card specification: inspect fields and client compatibility

Editable A2A 0.3.0 JSON, exact field failures and explicit client decisions.

Interactive lab Large language models

Agent chaos gate: survive a regional outage

Take the primary region down for thirty seconds and see whether health checks, routing TTLs, secondary capacity and session replication let the agent keep serving.

Interactive lab Large language models

Agent CI/CD: four evidence gates before promotion

See which build, contract, evaluation, or canary defect escapes a shallow pipeline.

Interactive lab Large language models

Agent CI/CD: keep a failing integration test from release

Push one commit through four integration tests with retries, a quarantine list and a required check, and see whether a real regression reaches release.

Interactive lab Large language models

Agent code execution: terminate a runaway script

Run a modeled loop against a deadline and compare a supervisor kill, cooperative cancellation, an iteration budget and no limit at all.

Interactive lab Large language models

Agent context failure: answer unknown when an early fact is evicted

Fill a bounded context window turn by turn, evict the earliest fact, and see whether the agent answers correctly, says unknown, or guesses.

Interactive lab Large language models

Agent cost control: reject an over-budget call

Admit eight model calls against a session budget and compare after-the-fact checks, input-only estimates and reserved worst-case estimates.

Interactive lab Large language models

Agent dashboards: separate error rate from tail latency

Inspect twelve requests through two different dashboard lenses.

Interactive lab Large language models

Agent debate: let a counterexample change the decision

Let an opponent search the release evidence, then see whether the judge rule lets one verified counterexample change the decision.

Interactive lab Large language models

Agent debugging: trace the wrong tool output

Six of twenty quote runs ship a wrong total after a tool update. See whether your traces pin the cause on the tool result, and whether a contract stops it at the boundary.

Interactive lab Large language models

Agent deployment: route only after readiness

Follow worker lifecycles and the invocations they own. Test readiness, shutdown deadlines and the point at which results commit.

Interactive lab Large language models

Agent evaluation: inspect every decision

Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.

Interactive lab Large language models

Agent event release gate: protect a required consumer field

Change the agent event schema in a producer release and see which gate notices that a downstream consumer can no longer parse it.

Interactive lab Large language models

Agent events: deduplicate a redelivered effect

Deliver eight agent events with a lost acknowledgement or a consumer crash, and see which dedupe key and window keep exactly one ledger effect per event.

Interactive lab Large language models

Agent feature flags: inspect stable targeting

Evaluate twelve named subjects; inspect allocation, overrides and error fallbacks.

Interactive lab Large language models

Agent feedback UI: record a versioned, actionable signal

Edit a fictional draft, test duplicate submission and inspect a local review queue.

Interactive lab Large language models

Agent filesystem: read versus write authority

Edit real JSON arguments, inspect policy gates and watch an approved local function change a synthetic ledger.

Interactive lab Large language models

Agent framework landscape: choose by constraints, not popularity

Score four agent framework archetypes for a workload, and see when popularity outvotes a requirement the workload cannot give up.

Interactive lab Large language models

Agent hierarchy: inspect a broken delegation chain

Send a billing task down a root, lead and specialist chain, break one link and see whether the failure is reported or silently swallowed.

Interactive lab Large language models

Agent history: inspect committed state changes

Two browser tabs change one agent session at once while the agent streams a tentative discount; inspect which revisions the history really committed.

Interactive lab Large language models

Agent host lifecycle: initialize resources and drain work

Follow worker lifecycles and the invocations they own. Test readiness, shutdown deadlines and the point at which results commit.

Interactive lab Large language models

Agent hotfix: verify the fix without skipping regression evidence

Replay an incident fixture, patch assertion, adjacent contracts, and a canary rollback check.

Interactive lab Large language models

Agent identity: trust and audience boundaries

Eight agents call a payments agent with bearer tokens. Tune the receiver’s verification policy and see which forged, replayed or expired tokens get in.

Interactive lab Large language models

Agent inspection API: query a trace without exposing forbidden fields

Compute object access, response fields and event pages from fictional records.

Interactive lab Large language models

Agent integration testing: execute four component contracts

Execute isolated in-memory probes and compare expected objects with observed tenant, timeout, storage and callback results.

Interactive lab Large language models

Agent invocation lifecycle: admit, execute, commit

Follow worker lifecycles and the invocations they own. Test readiness, shutdown deadlines and the point at which results commit.

Interactive lab Large language models

Agent logs: redact sensitive fields before emission

Five structured agent log records carry an e-mail, a bearer token, a card number in free text and a nested phone number; choose where and how they are redacted.

Interactive lab Large language models

Agent map tool: route policy versus traffic ETA

Compare a faster toll road with a slower toll-free route under a jam.

Interactive lab Large language models

Agent market: assign work under skill and budget constraints

Inspect eligible bids, award costs, and tasks left unassigned by a tight cap.

Interactive lab Large language models

Agent marketplace: compare eligible offers transparently

Inspect seven synthetic registry entries. Apply hard eligibility gates before ranking and proposing a compatible local contract.

Interactive lab Large language models

Agent memory compression: preserve the user’s report preference

Inspect whether a concise-bullets preference survives a long incident conversation.

Interactive lab Large language models

Agent memory hierarchy: keep a preference across sessions only when approved

Move two statements through working memory, session state and durable memory, then return a week later as the same or a different person.

Interactive lab Large language models

Agent pipeline: ordered transformations and failure propagation

Push ten invoices through six ordered stages, inject a fault into one, and see whether bad output is stopped, quarantined or published.

Interactive lab Large language models

Agent progress UI: make a stalled tool visible

Compare milestone updates with a silent final-only spinner.

Interactive lab Large language models

Agent red-team test: expose instruction injection in a document

Hide the same injected instruction in five places in a document, test only some of them, and compare the suite verdict with what production would see.

Interactive lab Large language models

Agent release train: hold promotion after canary errors

Separate artifact checks, sample sufficiency and canary error-rate gates before promotion.

Interactive lab Large language models

Agent release: blue/green and rollback

Switch a streaming agent from blue to green, then roll back when green fails, and count dropped streams, lost sessions and rollback time.

Interactive lab Large language models

Agent runtime capacity: every invocation needs an owner

Follow worker lifecycles and the invocations they own. Test readiness, shutdown deadlines and the point at which results commit.

Interactive lab Large language models

Agent sandboxing: deny outbound network from untrusted code

Run untrusted code that tries six outbound paths and see which ones each egress policy, resolver and loopback setting actually closes.

Interactive lab Large language models

Agent scoring: threshold and error cost

An agent auto-approves refund requests below a risk score; sweep the threshold and compare the most accurate setting with the cheapest one.

Interactive lab Large language models

Agent serving scale: compare fixed and elastic capacity

Serve an hour of agent traffic with a load spike and compare fixed replicas with an autoscaler that needs time to start new ones.

Interactive lab Large language models

Agent shell execution: restrict commands before they run

Six commands proposed by an agent meet a denylist, an executable allowlist or validated arguments, run through a shell or as an argument vector.

Interactive lab Large language models

Agent simulator: replay the same case before comparing versions

Compare a baseline and a candidate agent on simulated conversations, with fresh or paired scenario seeds and a deterministic or sampling user simulator.

Interactive lab Large language models

Agent skills: load only trusted instructions for the task

Five skills are installed for a PDF-to-CSV task; choose which bodies enter the context and see what an untrusted or tampered skill can make the agent do.

Interactive lab Large language models

Agent smoke testing: find a broken startup before release

Run four concrete probes and see when a shallow green check lets a broken agent escape.

Interactive lab Large language models

Agent soak test: expose a slow memory leak

Run a modeled load test for hours, sample the heap, fit a growth rate and see whether a slow leak is exposed or missed.

Interactive lab Large language models

Agent swarm: elect one owner for a duplicated task

Broadcast one refund task to two workers, stall the first one, and see whether a lease, renewal and a fencing token keep the refund to exactly one.

Interactive lab Large language models

Agent team: independent specialists and a final check

Have legal, finance and security specialists review an eight-clause contract, and see what shared notes, voting and a final coverage check change.

Interactive lab Large language models

Agent telemetry events: validate correlation fields

Quarantine events whose trace or tenant context is missing.

Interactive lab Large language models

Agent types: choose sequential or parallel work by dependency

Compose four sub-agents as sequential, parallel or mixed workflows and see which reads find their inputs in shared state.

Interactive lab Large language models

Agent unit testing: exercise boundaries and repeated effects

Compare a happy-path unit check with schema, limit, and idempotency fixtures.

Interactive lab Large language models

Agent user analytics: respect opt-out before collecting events

Process actual fictional event records and compare storage with a time-window report.

Interactive lab Large language models

Agent UX: clarify the report target

Resolve the intended tenant, check permission, then inspect the exact records used in the report.

Interactive lab Large language models

Agent voting: majority decision with a recorded dissent

Three reviewer agents vote on promoting a prompt; one finds failing safety tests. See which rule, voting order and decision record keep that dissent useful.

Interactive lab Large language models

Agent web browsing: keep page text out of the instruction channel

Let an agent read three shop pages, one carrying an injected instruction, and see which defence layer, if any, stops it.

Interactive lab Large language models

Agno-style remote agent: bound the Java adapter wait

Call an Agno-style remote agent from a Java adapter with a small thread pool and watch hung calls pin threads until a timeout frees them.

Interactive lab Large language models

Analogical prompting: test whether a pattern transfers

Borrow a design from a packet router or a batch import, map each element onto the real problem, and see whether the release tests notice what did not transfer.

Interactive lab Large language models

Application result caching: reuse without changing the answer

Eight questions from five users hit an answer cache while a price changes; see which cache keys and lifetimes change the answers people get.

Interactive lab Large language models

Approval workflow: bind review to operation intent

Edit real JSON arguments, inspect policy gates and watch an approved local function change a synthetic ledger.

Interactive lab Large language models

Artifact capacity: count stored content bytes

Follow exact content through assembly, a conditional write and a scoped version read.

Interactive lab Large language models

Artifact service contract: assemble, commit and read

Write a three-section report into a versioned artifact store while another reader loads it, and see which version that reader actually gets.

Interactive lab Large language models

Async state: prevent a lost update

Let parallel branches reserve books, stock handling and packing against one shared budget, and watch overlapping awaits erase each other’s debits.

Interactive lab Large language models

Asynchronous work: worker capacity versus dependencies

Run six local transformations. Watch dependency gates, worker assignments, retries and the exact artifacts produced.

Interactive lab Large language models

Attention masks: permitted query-key pairs

Explore the controls, compare outcomes and record your observations.

Interactive lab Large language models

Audit trail: reconcile actions with record effects

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

AutoGen-style team: stop a conversation that never terminates

Run a writer and a critic in round-robin and see which termination condition fires: a keyword, a message cap, a token budget, or none.

Interactive lab Large language models

BaseLlm interface: normalize a provider response shape

A vendor renames its response fields mid-run. See whether your BaseLlm adapter maps them, leaks the raw JSON to callers, or silently returns blanks.

Interactive lab Large language models

BeeAI-to-Java bridge: preserve a renamed response field

Rename a field in a Python agent response and see whether the Java client fails loudly, silently returns nothing, or keeps working.

Interactive lab Large language models

Benchmark audit: inspect reference labels and scores

Grade two agents on twelve items, then apply audit fixes one at a time and watch the leaderboard order change.

Interactive lab Large language models

Benchmark design: show denominators and failures

Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.

Interactive lab Large language models

Bidirectional agent: bounded request and response queues

Follow named requests and acknowledgements through two bounded queues.

Interactive lab Large language models

BigQuery-style query budget: filter before scanning

Estimate the bytes a BigQuery-style query reads from a partitioned table, then cap it, prune it and drop unneeded columns.

Interactive lab Large language models

Blue/green observability: measure both versions

Mirror a slice of traffic to the green candidate and see whether the release gate is measuring green or a blended dashboard.

Interactive lab Large language models

Blue/green release: inspect the cutover

Route individual requests, observe both versions and stop candidate traffic when the measured release gate fails.

Interactive lab Large language models

Bulkheads: reserved capacity versus borrowing

Share ten workers between a search tool and a payments tool, slow the search backend down for thirty seconds, and see whether payments keep working.

Interactive lab Large language models

Calendar boundaries: adjacent is not overlapping

A calendar tool checks five meeting requests against an attendee’s busy events; see how interval ends, time zones and a required gap change the answers.

Interactive lab Large language models

Calendar tool: detect overlapping intervals

Check a proposed meeting against four existing events and see how time zones, endpoint rules, buffers and tentative events decide the answer.

Interactive lab Large language models

Canary exposure: speed versus blast radius

A bad candidate goes through a canary ladder; trade the first step’s size, bake time and sample requirement against how many users see failures.

Interactive lab Large language models

Canary failure: stop a bad candidate

Route individual requests, observe both versions and stop candidate traffic when the measured release gate fails.

Interactive lab Large language models

Canary release: compare observed cohorts

Route individual requests, observe both versions and stop candidate traffic when the measured release gate fails.

Interactive lab Large language models

Capacity planning: find the saturation point

Find the saturation boundary and distinguish a longer queue from more service capacity.

Interactive lab Large language models

Chain of verification: catch an unsupported delivery claim

Verify a four-claim delivery answer against the order record, and compare independent questions with a verifier that merely restates the draft.

Interactive lab Large language models

Checkpointing: recover the gap after a snapshot

Follow concurrent drafts into committed revisions, inspect the checkpoint and reconstruct state after a simulated crash.

Interactive lab Large language models

CI evaluation gate: accept or reject a fixture run

Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.

Interactive lab Large language models

Circuit breaker: in-flight capacity

Break a dependency, watch the gate protect it, and decide when traffic can safely return.

Interactive lab Large language models

Circuit breaker: protected vs bypassed

Break a dependency, watch the gate protect it, and decide when traffic can safely return.

Interactive lab Large language models

Circuit breaker: recovery tuning

Trip a breaker on a failing tool, then tune its window, wait and probe budget so traffic returns once, without hammering a dependency that is still warming up.

Interactive lab Large language models

Circuit breaker: recovery verification

Verify a candidate circuit-breaker configuration with a fault-injection drill, then replay the real incident and see whether the gate told the truth.

Interactive lab Large language models

Citation check: does the cited source support the claim?

Inspect a reference identifier, source existence, numerical claim, and source version.

Interactive lab Large language models

Cloud tools: keep the agent service role least-privileged

Give an agent service account a role, let an injected instruction ask for destructive calls, and see what the role allows beyond the job.

Interactive lab Large language models

Code agent: gate a patch on regression evidence

See a boundary-condition change caught or released by a test gate.

Interactive lab Large language models

Collaborative tasks: worker limits and shared results

Run six local transformations. Watch dependency gates, worker assignments, retries and the exact artifacts produced.

Interactive lab Large language models

Concurrent tools: isolate failures before joining results

Run four tool calls on a small thread pool, make one fail, hang or stall, and see what the join returns and when.

Interactive lab Large language models

Conditional object writes: protect the live generation

Follow exact content through assembly, a conditional write and a scoped version read.

Interactive lab Large language models

Configuration and secrets: bind before activation

Edit real JSON layers. Trace each winning value and compare the candidate with the configuration that is actually active.

Interactive lab Large language models

Configuration precedence: trace the winning value

Edit real JSON layers. Trace each winning value and compare the candidate with the configuration that is actually active.

Interactive lab Large language models

Container packaging: pin the tested image

Trace a release from its registry reference to the exact platform artifact that was tested.

Interactive lab Large language models

Container termination drill: observe an ignored signal

Follow worker lifecycles and the invocations they own. Test readiness, shutdown deadlines and the point at which results commit.

Interactive lab Large language models

Content filter threshold: a quoted phrase versus a harmful request

Score twelve messages with a fictional harm classifier and see why one threshold blocks reporters and novelists while still missing a politely worded harmful request.

Interactive lab Large language models

Content moderation pipeline: find the missed boundary

Inspect request, tool, candidate, and private-data checks before release.

Interactive lab Large language models

Context compaction: retain the retry policy or lose the answer

Watch eight turns enter a bounded window and inspect the exact retained messages.

Interactive lab Large language models

Context compression: compare space saved with evidence retained

Replay a deployment conversation and test whether a region fact survives.

Interactive lab Large language models

Continuous evaluation: compare frozen fixtures

Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.

Interactive lab Large language models

Cooperative tasks: inspect scheduling and cancellation

Run six local transformations. Watch dependency gates, worker assignments, retries and the exact artifacts produced.

Interactive lab Large language models

Cost attribution: trace a tenant invoice to meter events

Trace a $100 provider bill to tenant invoices, including sub-agent calls, retried attempts and a shared vector database.

Interactive lab Large language models

Crew-style task assignment: name one owner before work starts

Post six tasks to a crew of agents and count customer emails: broadcast and racing claims send duplicates, a crash without a lease strands work.

Interactive lab Large language models

Cross-session memory: scope before similarity

Two users ask what they decided. A shared memory index ranks everyone’s memories by similarity; see what happens when the user filter comes after ranking.

Interactive lab Large language models

Customer credits: offset once and carry the remainder

Apply one service credit across three monthly invoices, crash and re-run a billing job, and check that every dollar of credit is offset exactly once.

Interactive lab Large language models

Data agent: expose a malformed row before aggregation

Parse an editable CSV before aggregating it and compare strict validation with lenient coercion that turns 14.3k into 14.

Interactive lab Large language models

Data map: trace primary, backup and export copies

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

Data policy: separate use, export and deletion

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

Data-asset dependencies: execute and inspect a finite DAG

Run six local transformations. Watch dependency gates, worker assignments, retries and the exact artifacts produced.

Interactive lab Large language models

Data-flow review: map copies before assessing risk

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

Dataset-driven evaluation: choose the test slice

Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.

Interactive lab Large language models

Dead-letter review: quarantine and redrive

Follow four messages through retry, quarantine, expiry and selective redrive while checking the committed-effect ledger.

Interactive lab Large language models

Debugging hallucinations: compare a claim with its source

Split an answer into claims, find the best supporting source sentence, and check numbers and citations before the answer ships.

Interactive lab Large language models

Debugging latency: inspect work and waiting

Four requests arrive together. Inspect the fourth request’s trace and separate the time spent working from the time spent waiting for workers and connections.

Interactive lab Large language models

Delegated result gate: verify before accepting a report

Six worker agents report back on a release review. Decide what the coordinator checks before a report enters the release decision.

Interactive lab Large language models

Delegation contract: join only successful artifacts

Delegate a price lookup to three regional sub-agents and decide which returned artifacts a coordinator may join into one answer.

Interactive lab Large language models

Deletion: primary storage is only one copy

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

Deployment fixture gate: account for every invocation

Route twenty requests across two workers, send SIGTERM to one, and check that every invocation either commits or is rejected in a way a client can retry.

Interactive lab Large language models

Deployment gate: evidence before promotion

Route individual requests, observe both versions and stop candidate traffic when the measured release gate fails.

Interactive lab Large language models

DevOps agent: audit a release decision

Bind tests to a candidate, enforce approval and inspect a numerical health gate.

Interactive lab Large language models

Disaster recovery: test that a backup can restore

Take scheduled backups, inject a hidden defect, lose primary storage and find out which backup actually restores.

Interactive lab Large language models

Documentation agent: answer only with a supporting passage

Score eight versioned documentation passages against a question and answer only when a passage for the user’s version supports it.

Interactive lab Large language models

DSPy-style optimization: do not select on training examples

Score candidate prompt programs, choose one by training or development accuracy, and compare the choice with held-out test accuracy.

Interactive lab Large language models

Dunning: retry collection without rebilling

Follow individual meter records into invoice lines, one credit application and a separate collection/refund ledger.

Interactive lab Large language models

Dynamic reload: keep the complete last-good revision

Edit real JSON layers. Trace each winning value and compare the candidate with the configuration that is actually active.

Interactive lab Large language models

Edge deployment: fit the model inside the worker budget

Place an edge classifier, a JVM agent and its sessions across a Worker isolate, a container and a region, and see what fits, what sleeps and what is lost.

Interactive lab Large language models

Email observability: receipt versus repeated delivery

Send four emails through a slow provider with a tool timeout, retries and idempotency keys, then compare reported status with what customers received.

Interactive lab Large language models

Email review: invalidate approval after edits

Approve a send_email call, let the agent edit it, and see whether the approval still describes the message that is actually sent.

Interactive lab Large language models

Email tool: review intent before an effect

Edit real JSON arguments, inspect policy gates and watch an approved local function change a synthetic ledger.

Interactive lab Large language models

Embedding cache failure: the transform version belongs in the key

Watch concrete requests fill a bounded cache. Inspect stored vectors, expiry, eviction and incorrect reuse.

Interactive lab Large language models

Environment configuration: convert strings explicitly

Edit real JSON layers. Trace each winning value and compare the candidate with the configuration that is actually active.

Interactive lab Large language models

Episodic memory: retrieve a past incident

Follow real ranking calculations from stored memory cards through scope filters into a bounded context tray.

Interactive lab Large language models

Evaluation datasets: expose coverage gaps

Build an evaluation set from logs and see whether it has enough cases in the intents where a candidate regresses.

Interactive lab Large language models

Evaluation framework: inspect metric trade-offs

Score ten agent runs by tool trajectory and answer overlap, and compare each verdict with a person’s: strict scorers fail good runs, loose ones ship bad ones.

Interactive lab Large language models

Evaluation harness: repeat a deterministic run

Run the same ten recorded cases several times and find out whether sampling and a live tool clock make the gate decision change between runs.

Interactive lab Large language models

Evaluation integration: scores into decisions

Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.

Interactive lab Large language models

Evaluation observability: errors behind the score

A candidate agent raises the aggregate pass rate while one group of users gets much worse; decide which gate notices.

Interactive lab Large language models

Execution plan: dependencies, deadlines and acceptance

Schedule a six-step plan on a few workers, inject one defective artifact, and see whether checking acceptance early or late meets the deadline.

Interactive lab Large language models

Failure propagation: client state versus server state

Charge one order through an orchestrator and a remote payments agent, inject a fault, and compare what the client believes with what the server committed.

Interactive lab Large language models

Fault injection: lose an acknowledgement after a side effect

Inject a lost response after the server commits, then follow retries, deduplication and the committed-effect ledger.

Interactive lab Large language models

Few-shot library: select examples that cover the query

Choose prompt examples from a ten-example library by recency, shuffle or similarity, and see when the chosen set covers the ticket.

Interactive lab Large language models

File tools: bound the virtual workspace

Edit real JSON arguments, inspect policy gates and watch an approved local function change a synthetic ledger.

Interactive lab Large language models

Final summary: reconcile the inputs behind the result

Run six local transformations. Watch dependency gates, worker assignments, retries and the exact artifacts produced.

Interactive lab Large language models

Firestore-style document access: compare path owner to caller

Send six agent requests against users/{uid}/notes paths and compare the path owner with the signed-in caller under each access path and rule.

Interactive lab Large language models

Flat agent team: direct handoff or broadcast

Route four fixed tasks across peers and inspect the cost of broad coordination.

Interactive lab Large language models

Flowise-style Agentflow: validate a disconnected tool node

Edit a four-node agent graph, validate reachability before running it, and see what the agent answers when its tool is disconnected.

Interactive lab Large language models

Free tier: apply allowances to the correct meter

Follow individual meter records into invoice lines, one credit application and a separate collection/refund ledger.

Interactive lab Large language models

Function calling: structured arguments into a local function

Take one model-proposed function call through a declared schema, the function calling mode and argument validation before a local function runs.

Interactive lab Large language models

Function dispatch: inspect a validated file operation

Follow one model-proposed file path through joining, normalisation and symbolic-link resolution, and see which containment check stops it.

Interactive lab Large language models

Function tools: validate before dispatch

Send one refund call through a tool with three side effects and compare validating every argument first with checking each one just before it is used.

Interactive lab Large language models

Function-call inspection: validate a state mutation

The model asks to change a shipping address. Choose which checks stand between its function call and the order record.

Interactive lab Large language models

GAIA-style benchmark: keep evaluation answers out of tuning

Tune an agent on a GAIA-style validation set, let it browse and remember, and watch the reported score drift away from what it can really do.

Interactive lab Large language models

Gemini integration: catch missing credentials before request

Validate credentials, backend settings and the model id at startup, or discover the same mistake through rejected calls.

Interactive lab Large language models

Gemini LLM tool call: validate typed arguments before dispatch

The model fills count_items with arguments of the wrong type; compare loose coercion with strict validation and a repair turn.

Interactive lab Large language models

Gemini multimodal input: send pixels when the question needs them

Ask three questions about a chart and decide whether to send OCR text, pixels or both, at what size and crop, and what that costs in tokens.

Interactive lab Large language models

Gemini thinking budgets: an executed search analogy

Execute a bounded graph traversal and distinguish an untested route from a verified answer.

Interactive lab Large language models

Gemini variant comparison: quality versus latency

Run the same twelve prompts against two fictional variants and inspect the paired observations.

Interactive lab Large language models

Gemini-style safety thresholds: inspect category decisions

Set a threshold for each harm category on a security-awareness tutor and see which legitimate lessons get blocked, which harmful replies get through, and what the learner sees.

Interactive lab Large language models

Generalist agent: route a specialized question to a qualified path

Route eight requests by classifier confidence and see which regulated questions end up with an unqualified generalist.

Interactive lab Large language models

GitOps rollout: validate a desired manifest

Edit the manifest in Git, validate it or not, then watch sync, prune, health and self-heal decide what actually runs.

Interactive lab Large language models

Governance gates: holds and retention exceptions

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

Graceful shutdown: stop admitting, then drain

Follow worker lifecycles and the invocations they own. Test readiness, shutdown deadlines and the point at which results commit.

Interactive lab Large language models

Graph of thoughts: verify the artifacts you combine

Trace two sorting branches into an aggregate and expose a sorted result that lost data.

Interactive lab Large language models

Griptape-style workflow: honor a parent-output dependency

Two child tasks need a parent task’s summary and risk flags. Launch them on a dependency edge, all at once, or by polling, and see what they read.

Interactive lab Large language models

Ground truth: make the rubric explicit

Build a truth set from two annotators’ labels, measure their agreement with Cohen’s kappa, and see how a policy change makes old labels wrong.

Interactive lab Large language models

How an LLM answers a question

Watch Qwen2.5-0.5B-Instruct turn a question into tokens, IDs, scores and an answer, one real token at a time, with probabilities captured from the model on a laptop CPU.

Interactive lab Large language models

Human evaluation: resolve grading disagreement

Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.

Interactive lab Large language models

Incident recall: relevance versus recency

Follow real ranking calculations from stored memory cards through scope filters into a bounded context tray.

Interactive lab Large language models

Integrating guardrails at agent and tool boundaries

Send six requests through input, tool-argument, tool-result and output checks and see which boundary catches each harm, and which benign request is blocked.

Interactive lab Large language models

Interrupted agent effect: reuse a receipt after a lost ACK

Crash an agent between creating an invoice and recording it, resume, and count how many invoices and emails one order produced.

Interactive lab Large language models

Judge evaluation: audit recorded quality scores

Compare twelve recorded judge scores with human reference labels, fill the confusion matrix and compute Cohen’s kappa before trusting the judge.

Interactive lab Large language models

KV cache memory: storage versus recomputation

Size the key/value cache of a fictional 32-layer decoder from heads, precision, context and batch, and compare it with recomputing the prefix every step.

Interactive lab Large language models

Layered guardrails: compare one check with four boundaries

Five conversations carry defects at the request, a tool call, a tool result and the response; compare one request check with four boundaries.

Interactive lab Large language models

LlmAgent release gate: approve a high-impact tool action

An agent proposes a lookup, a refund and an email, plus one injected refund; choose which calls run alone and which wait for a person.

Interactive lab Large language models

Load testing: throughput and admission outcomes

Drive an agent endpoint with a traffic ramp and see whether overload is rejected at the door, queued, or served after the client has already given up.

Interactive lab Large language models

Long-running task: inspect progress without premature success

Run six local transformations. Watch dependency gates, worker assignments, retries and the exact artifacts produced.

Interactive lab Large language models

Long-term memory: retain previous sessions

A user mentioned her city, diet and allergy across three sessions over 200 days. Choose what memory keeps and see what the agent knows on day 230.

Interactive lab Large language models

Matrix agent organization: resolve dual ownership

Inspect product and domain owner votes, distinguish absence from rejection, and count only the decisions already reviewed.

Interactive lab Large language models

Memory bank: pack a bounded context

Follow real ranking calculations from stored memory cards through scope filters into a bounded context tray.

Interactive lab Large language models

Memory forgetting: age and superseded facts

Follow real ranking calculations from stored memory cards through scope filters into a bounded context tray.

Interactive lab Large language models

Memory recall: inspect the retrieval score

Follow real ranking calculations from stored memory cards through scope filters into a bounded context tray.

Interactive lab Large language models

Memory retrieval: inspect missing evidence

Filter eight stored memories by tenant, kind and age, rank them into a bounded context tray, and find out why the needed evidence is missing.

Interactive lab Large language models

Middleware ordering: inspect the stopping gate

Four hooks guard a refund tool: tracing, a cache, an amount normaliser and the refund policy. Reorder them and see which one stops the chain first.

Interactive lab Large language models

Model fallback: bound attempts and preserve the deadline

Separate an eligible routing failure from an incompatible backup or a request that ran out of time.

Interactive lab Large language models

Model settings: validate a candidate before activation

Push a candidate model configuration with one defect through schema, catalog and probe checks, and see what happens to live traffic when a check is missing.

Interactive lab Large language models

Moderation release gate: block on the wrong error type

Gate a new moderation policy on its eval results and see an accuracy rule ship a candidate that misses four times as many severe cases.

Interactive lab Large language models

Multi-agent workflow: explicit handoffs and verified output

Hand research from one agent to a writer as a raw transcript or as a typed, sourced contract, then decide what a reviewer must check before publishing.

Interactive lab Large language models

Multi-region A2A: resume a task after home-region failure

Commit a ten-step A2A task in a home region, replicate it to a standby, fail the home region and resume: count lost steps and duplicate side effects.

Interactive lab Large language models

Multimodal agent: align frames before describing an event

Sample video frames, place the audio event on the same clock, and see whether the description keeps cause before effect.

Interactive lab Large language models

Observability CI: detect a missing tool span

A refactor loses the refund tool’s span. Decide which conversations CI replays and which telemetry check runs, and see what reaches production.

Interactive lab Large language models

Observe the complete record lifecycle

Follow one customer record through primary storage, a search index, an analytics export and nightly backups, and compare what lineage reports with what exists.

Interactive lab Large language models

Output grounding check: inspect every factual sentence

Check every sentence of a cited answer against the passage it cites, and see which verifier catches a dropped condition, an unrelated source and an uncited claim.

Interactive lab Large language models

Payment operation: inspect attempts, receipts and refunds

A charge times out after the card was taken, a bank asks for 3-D Secure, and a refund is requested twice. Read the provider’s records against what the agent says.

Interactive lab Large language models

Privacy engineering failure: deleted data returns

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

Procedural memory: retrieve an action sequence

Follow real ranking calculations from stored memory cards through scope filters into a bounded context tray.

Interactive lab Large language models

Processing inventory: identify purpose and copies

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

Progressive rollout: expand candidate exposure

Step a defective candidate from 1% to 100% of traffic behind an error-rate gate, and measure how many users it hurts before it is stopped.

Interactive lab Large language models

RAG grounding: test retrieval, freshness, support, and injection

Retrieve from six documents, filter stale and untrusted ones, and check whether the refund answer is current, supported or injected.

Interactive lab Large language models

Refunds: never return more cash than was collected

A $100 order paid with a card and a gift card is fully returned; check that each tender gets back no more than it paid, even with a coupon and a retried request.

Interactive lab Large language models

Registry design: budget and freshness are separate gates

Inspect seven synthetic registry entries. Apply hard eligibility gates before ranking and proposing a compatible local contract.

Interactive lab Large language models

Regression testing: inspect changed predictions

Inspect each recorded score, compare its decision with the reference label and watch the confusion matrix fill.

Interactive lab Large language models

Request hedging: race a backup against a slow primary

Send a backup request when the primary is slow, and weigh the tail latency saved against extra load and duplicate writes.

Interactive lab Large language models

Retention: expire records at the policy boundary

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

Retry safety: backoff and side effects

Retry one payment tool call through a lost response, a temporary outage and a permanent error, and count the charges that really happened.

Interactive lab Large language models

Role-based agent team: permission at the action boundary

Watch authorized tasks, revoked roles, and cross-role actions under two policies.

Interactive lab Large language models

Rollback: preserve evidence and stop exposure

A bad prompt release gives wrong refund answers; choose the alert window and rollback method, and decide what happens to its evidence and sessions.

Interactive lab Large language models

Safety evaluation: inspect false positives and false negatives

Run four labeled examples against a threshold and compare the exact error cells.

Interactive lab Large language models

Search tool: validate a bounded local query

The model proposes six searches over a local document store, two steered by an injected instruction. Validate the limit, the path and the query before anything runs.

Interactive lab Large language models

Search-tool integration: prefer current evidence over a stale top hit

Inspect ranking, freshness, citation, and the chosen document.

Interactive lab Large language models

Seat billing: quantity and effective date

Follow individual meter records into invoice lines, one credit application and a separate collection/refund ledger.

Interactive lab Large language models

Seat proration: charge the remaining period

Add five seats and remove three during a 31-day October, and compare each proration rule with the contract’s daily amount.

Interactive lab Large language models

Secret references: pinned versions and tenant boundaries

Edit real JSON layers. Trace each winning value and compare the candidate with the configuration that is actually active.

Interactive lab Large language models

Semantic memory: retrieve stored facts

Follow real ranking calculations from stored memory cards through scope filters into a bounded context tray.

Interactive lab Large language models

Server-side agent: authorize before reading a tenant record

A client agent for tenant acme asks a server agent for four invoices; choose where the tenant comes from and whether ownership is checked before or after the read.

Interactive lab Large language models

Session recovery: replay recorded transitions

Record a short ticket-booking conversation as session events, crash the process, and rebuild state by replaying only what was persisted.

Interactive lab Large language models

Session resumption: recover a consistent revision

Follow concurrent drafts into committed revisions, inspect the checkpoint and reconstruct state after a simulated crash.

Interactive lab Large language models

Session state: separate a snapshot from history

Keep a cart, a unit preference and a sign-in token across ten turns and a restart, and see which belong in state, which in history, and which nowhere.

Interactive lab Large language models

Shadow deployment: compare without serving v2

Discarding a candidate response does not undo its tool or session effects.

Interactive lab Large language models

Shadow testing: observe isolated candidate failures

Mirror checkout requests to a candidate, compare responses, and check that the shadow neither emails customers nor slows them down.

Interactive lab Large language models

Sharing an artifact: read permission is not write permission

Follow exact content through assembly, a conditional write and a scoped version read.

Interactive lab Large language models

Short-term context: current session only

Follow real ranking calculations from stored memory cards through scope filters into a bounded context tray.

Interactive lab Large language models

Speculative Decoding

Generate 48 tokens with a draft model and a target verifier, and see how draft length, acceptance rate and draft cost decide whether it is faster.

Interactive lab Large language models

Startup validation: fail closed on invalid settings

Edit real JSON layers. Trace each winning value and compare the candidate with the configuration that is actually active.

Interactive lab Large language models

State diff: compare drafts, revisions and commits

Follow concurrent drafts into committed revisions, inspect the checkpoint and reconstruct state after a simulated crash.

Interactive lab Large language models

State export: preserve a reproducible replay

Export a ten-step agent session and replay it: see which bundle contents keep the replay faithful and which make it re-issue a real credit.

Interactive lab Large language models

State management: concurrent intent and revisions

Follow concurrent drafts into committed revisions, inspect the checkpoint and reconstruct state after a simulated crash.

Interactive lab Large language models

State replay: inspect deterministic reconstruction

Follow concurrent drafts into committed revisions, inspect the checkpoint and reconstruct state after a simulated crash.

Interactive lab Large language models

Subject export: verify identity and scope

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

Subject requests: preserve the identity boundary

Process an access or erasure request for Ana Ruiz across four stores, and see what the verification method and search key release, and to whom.

Interactive lab Large language models

Subscription changes: inspect a prorated downgrade

Separate a seat adjustment from the money already collected and the refund actually requested.

Interactive lab Large language models

Synthetic monitoring: one-region alert versus quorum

Probe three regions for thirty minutes through a probe-host blip and a real outage, and compare single-region, quorum and dual-vantage alerting.

Interactive lab Large language models

Task-DAG scheduling: measure the parallel critical branch

A daily report DAG has three parallel branches; choose which task to speed up and how many worker slots run, and see which branch decides the finish time.

Interactive lab Large language models

Temperature sampling: reshape and inspect probabilities

Scale eight fixture logits by a temperature, compute stable or naive softmax, and draw seeded samples to see when the tail starts to speak.

Interactive lab Large language models

Tenant billing: isolate usage and event identity

Meter eight usage events from two tenants whose clients reuse event ids, and see which tenant each invoice charges.

Interactive lab Large language models

Tenant capacity: scale the constrained partition

Let one tenant burst on a shared partition, then compare uniform scaling, targeted scaling, isolation and a per-tenant rate limit.

Interactive lab Large language models

Tenant configuration: scoped overrides and source tracking

Merge global defaults, the pro plan and one tenant’s overrides into an effective agent configuration, and track which layer set each value.

Interactive lab Large language models

Tenant data: principal versus resource ownership

An agent reads invoices, an HR file and a user’s folder for ana at acme. Decide where the tenant comes from, what is checked and with whose credential.

Interactive lab Large language models

Tenant isolation: prevent cross-tenant access

Inspect credential claims, policy gates and the resource decision. Change a condition and follow the exact denial reason.

Interactive lab Large language models

Tenant migration: compare before cutover

Copy a tenant to a new store while writes continue, compare source and target, and decide whether the cutover may proceed.

Interactive lab Large language models

Tenant offboarding: stop late writes

Follow actual fictional records through primary storage, a backup, an export and a restore.

Interactive lab Large language models

Tenant onboarding: validate, provision and retry

Inspect the exact resource ledger after validation, allocation and repeated readiness probes.

Interactive lab Large language models

Tenant registry: do not route across scope

Two tenants register agents with the same names as each other and as a platform agent. Resolve five requests and see which reach another tenant.

Interactive lab Large language models

Tenant service targets: count every offered request

Follow each request from arrival to completion, rejection or timeout.

Interactive lab Large language models

Tenant usage analysis: compare isolated meter totals

Attribute nine usage events to two tenants for one billing day and see how API keys, missing claims and time zones move usage between tenants.

Interactive lab Large language models

Token-cost ledger: inspect input and output charges

Split cached input and thinking output, then keep incomplete costs visibly unresolved.

Interactive lab Large language models

Tool callbacks: block before execution

Gate six tool calls with a before-tool hook, and see how its return value decides whether the model explains, retries, lies or the whole turn dies.

Interactive lab Large language models

Tool design: separate schema, authority and effect

Edit real JSON arguments, inspect policy gates and watch an approved local function change a synthetic ledger.

Interactive lab Large language models

Tool plugin policy: transform or reject a call

Edit real JSON arguments, inspect policy gates and watch an approved local function change a synthetic ledger.

Interactive lab Large language models

Top-k and nucleus sampling: inspect retained candidates

Filter six next-token distributions, three peaked and three flat, with top-k, top-p or min-p, then draw from what is kept.

Interactive lab Large language models

Trace integration: follow an outgoing traceparent

Follow one agent request through an HTTP tool call, an inventory service and a queue, and see where its trace context survives.

Interactive lab Large language models

Traffic shifting: follow each routed request

Shift traffic from v1 to v2 in three steps while four users hold multi-turn sessions, and follow which version serves every turn.

Interactive lab Large language models

Undo state: append a compensating transition

Undo the last steps of an order that reserved stock, charged a card and sent an email, and compare deleting history with appending its reversal.

Interactive lab Large language models

Usage cost tracking: quantities before totals

Follow individual meter records into invoice lines, one credit application and a separate collection/refund ledger.

Interactive lab Large language models

Usage metering: detect duplicates and invalid records

Run ten October meter records for one tenant through dedupe, validation, unit conversion and period assignment, and compare the invoice with the correct 6,200 tokens.

Interactive lab Large language models

Usage reports: reconcile finalized and deferred records

Follow individual meter records into invoice lines, one credit application and a separate collection/refund ledger.

Interactive lab Large language models

Workflow inspection: states, workers and intermediate data

Two parallel researchers write findings into shared session state; see which findings survive to the report step.

Interactive lab Large language models

Workflow runtime: inspect ready, running and blocked states

Schedule six dependent tasks on a small worker pool, fail an API task, and watch ready, running, failed, blocked and cancelled states.

Interactive lab Large language models

Working memory: assemble the active context

Follow real ranking calculations from stored memory cards through scope filters into a bounded context tray.

Interactive lab

Views are my own and do not represent any employer, past or present.