What an 8 GB GPU Actually Ran: Local Model Results

Choose a model, then test your assumptions

Start with a prediction: Which model would you choose if a response must be correct and finish within your latency budget? Explore the recorded runs below, then check whether your first choice survives both constraints.

Try a separate synthetic answer check

Simulation, not a recorded model response: choose an integer and a proposed label. Predict whether a valid JSON object can still contain the wrong answer. This exercise uses only your selected number and label, locally in the browser.

{"label":"even"}

Explain your result: why would accepting valid JSON alone miss this failure? Return to the measured runs and compare the valid-output count with the correct-answer count.

What was actually run

These measurements were recorded at 2026-10-28T04:03:38.308408+00:00 on NVIDIA GeForce RTX 4060 Laptop GPU with 8188 MiB total device memory and driver 592.82. Ollama version 0.40.11 performed the two inference trials. The models were already installed locally; no personal dataset or remote inference provider was used. GPU residency was confirmed through the runtime after warmup. That observation does not establish that every operation ran exclusively on the GPU.

Each model received 24 distinct synthetic integer-parity prompts after one excluded warmup request. Inputs were integers from 100 through 123. The required response was a JSON object with one label: even or odd. The structured-output schema constrained valid labels, while the evaluator independently checked correctness. Concurrency was one, the context limit was 2,048 tokens and the output limit was 32 tokens. Temperature was zero and the requested seed was 42.

Inference results

Wall latency measures a complete local HTTP generation request. It includes request handling and generation; it is not time to first token. The output-rate column divides total generated tokens by the sum of the runtime-reported generation durations. Short constrained answers make token rates noisy, so use these numbers for this task and configuration only. The sample size is shown because a percentile from a small single trial is not a long-term service guarantee.

Adapter-training outcome

Training status: completed. Training, if completed, used a separate Transformers/PyTorch CUDA process after the Ollama models were unloaded. It is LoRA over an FP16 base model, not QLoRA, and it is not a vLLM serving experiment. Failed or unavailable training produces no invented accuracy, duration or memory figure.

{
  "method": "LoRA FP16 frozen base; not QLoRA",
  "model": "Qwen/Qwen2.5-0.5B-Instruct",
  "model_revision": "7ae557604adf67be50417f59c2c2f167def9a775",
  "training_steps": 32,
  "optimizer_steps": 16,
  "trainable_parameters": 270336,
  "training_elapsed_s": 11.307847000018228,
  "peak_torch_allocated_mib": 1070.95947265625,
  "baseline": {
    "samples": 12,
    "correct": 6,
    "accuracy": 0.5,
    "mean_latency_s": 0.38881636667550384
  },
  "adapted": {
    "samples": 12,
    "correct": 6,
    "accuracy": 0.5,
    "mean_latency_s": 0.2688223000053161
  },
  "versions": {
    "torch": "2.9.1+cu128",
    "transformers": "4.57.1",
    "peft": "0.17.1",
    "accelerate": "1.11.0"
  },
  "limitations": [
    "Small held-out set and shared prompt template; not broad generalization evidence",
    "One run and no high-precision versus quantized comparison",
    "Latency includes each complete generation; not time to first token"
  ]
}

Interpret the comparison carefully

The inference models differ in size as well as their packaged representation. This is therefore a local model-selection observation, not an isolated four-bit versus eight-bit comparison. Neither row proves that a compression format caused a speed or quality difference. The parity task has a deterministic answer and no sensitive content, but it is much narrower than general language understanding or reliable agent behavior.

A valid schema is also not proof of a correct answer. The report retains both counts to expose that distinction. Failed classifications stay in the denominator. A poor result is published as observed rather than removed to improve the score. No training examples are reused as held-out adapter evaluation values, although the shared instruction template limits how much can be concluded about generalization.

Reproduce and inspect the evidence

The safe aggregate JSON result file records model labels, quantization metadata, runtime versions, request settings, sample counts and the measured aggregates. It contains no raw generated responses, account identifiers, user prompts, credentials, local filesystem paths or machine serial numbers. The experiment scripts use fixed synthetic tasks; the public report is constructed from an explicit field schema rather than copying application logs.

Reproduction requires the same model artifacts, software versions and workload settings. The result file records the available revision for adapter training and the installed model labels for Ollama; it does not claim identical artifacts across unrelated machines. Device power state, shared GPU usage, background work and runtime revisions can change timings. Repeat the trial before treating a small latency difference as meaningful.

Read the quantization guide for weight-memory arithmetic, then use the Local LLM learning path to distinguish calculated simulations from these measured observations. Runtime references: Ollama generation fields and PEFT LoRA concepts.