ScalaCheck Property Testing, in depth: generators, properties, shrinking, seeds and stateful testing

How a property run works

A property is a function from generated inputs to a result, evaluated in a loop. Each iteration takes a pseudo-random seed and a size parameter, asks a generator for a value and evaluates the property. The size grows during the run, so early values are tiny and later ones larger: small inputs find shallow bugs cheaply, larger ones explore deeper.

Each evaluation passes and counts toward the required successes, 100 by default; is undecided because a ==> precondition was false, so the input is discarded; or fails by returning false or throwing. On the first failure ScalaCheck starts shrinking: it asks a Shrink instance for smaller candidates, keeps any that still fail, and repeats until none does. The report holds the original and shrunk values and enough to rerun the sequence.

So a passing property has been checked on a sample, not proven, and is exactly as strong as its generator; and because a seed drives the run, a failure is deterministic once you have the seed.

Generators: Gen and Arbitrary

A Gen[A] produces values of A from a seed and a size, and composes like any functional value: for-comprehensions build records, Gen.oneOf and Gen.frequency pick from alternatives, Gen.choose from a range, Gen.listOf builds collections, and Gen.sized reads the current size. Recursive structures need Gen.lzy or a depth budget derived from the size, or generation can recurse without bound.

An Arbitrary[A] wraps a generator as an implicit, found through implicit resolution when you write forAll { (a: A) => ... }. Define domain generators as plain values first; pass them explicitly, as in forAll(orderGen) { o => ... }, when a property needs a specific slice of the domain, so a distant implicit cannot change what a test checks.

import org.scalacheck.{Arbitrary, Gen}

final case class Money(cents: Long, currency: String)
final case class Order(id: String, lines: List[Money])

object Generators {
  val currency: Gen[String] = Gen.oneOf("EUR", "USD", "GBP")

  // Bias toward edge values: zero and boundaries are where arithmetic breaks.
  val cents: Gen[Long] = Gen.frequency(
    5 -> Gen.choose(0L, 100000L),
    1 -> Gen.const(0L),
    1 -> Gen.oneOf(Long.MaxValue / 2, 1L)
  )

  val money: Gen[Money] = for {
    c   <- cents
    cur <- currency
  } yield Money(c, cur)

  // Gen.sized lets ScalaCheck grow collections as the run progresses.
  val order: Gen[Order] = Gen.sized { n =>
    for {
      id    <- Gen.identifier
      k     <- Gen.choose(0, n min 20)
      lines <- Gen.listOfN(k, money)
    } yield Order(id, lines)
  }

  implicit val arbOrder: Arbitrary[Order] = Arbitrary(order)
}

Note the bias in cents. Uniform random longs almost never hit zero, one or overflow boundaries, the values that break arithmetic; weighting them in is the cheapest improvement most generators can get. Types constrained as refined types deserve generators that produce only valid values, not filters.

Property patterns that find real bugs

Most people stall because they cannot think of a property. A few patterns cover most code:

• Round trip. Decode after encode gives back the input: serialisers, parsers with printers, compression.
• Invariant. Something stays true after an operation: sorted output is ordered and a permutation of the input; a balance never goes negative.
• Model or oracle. Compare the optimised code with a slow, obviously correct one, such as a cache against a Map.
• Idempotence and commutativity. Normalising twice equals normalising once; independent updates commute.
• Metamorphic relations. When you cannot compute the answer, relate two answers: adding a filter never returns more rows.
• Algebraic laws. Monoid and functor laws; the Cats ecosystem publishes law suites built on ScalaCheck.

import org.scalacheck.{Prop, Properties}
import org.scalacheck.Prop.{forAll, propBoolean}
import Generators._

object CodecSpec extends Properties("Codec") {

  // Round trip: the most valuable property you will ever write.
  property("decode . encode == identity") = forAll { (o: Order) =>
    Codec.decode(Codec.encode(o)) == Right(o)
  }

  // Invariant, with labels so a failure says which clause broke.
  property("total is order independent") = forAll { (o: Order) =>
    val a = Pricing.total(o)
    val b = Pricing.total(o.copy(lines = o.lines.reverse))
    (a == b) :| s"forward=$a reverse=$b"
  }

  // Precondition with ==> : inputs that fail it are discarded, not passed.
  property("single-currency totals keep the currency") = forAll { (o: Order) =>
    (o.lines.nonEmpty && o.lines.map(_.currency).distinct.size == 1) ==> {
      Pricing.total(o).currency == o.lines.head.currency
    }
  }

}

The :| operator labels a clause so the failure report says which one broke. The ==> operator discards inputs; use it for rare exclusions only, because a precondition that rejects most inputs makes the run give up, and giving up is a failure. And Prop.classify and Prop.collect print the input distribution, which is how you discover that 90% of your orders were empty.

Shrinking, and when it misleads

A raw counterexample is often a list of forty random elements. Shrinking proposes smaller candidates, such as a list with elements removed or a number closer to zero, keeps the first that still fails, and repeats until every candidate passes. The result is locally minimal.

Default shrinkers are chosen by type, not by the generator that produced the value. If your generator produces ports from 1024 to 65535, the default Int shrinker will propose 0 and negatives, and the shrunk value may fail for a different reason than the original. Recent versions filter candidates through a generator's suchThat filter, but a generator built from choose and map carries none, so do not rely on it.

Three fixes: give a domain type a Shrink that stays in range; use forAllNoShrink when shrunk values are meaningless, such as opaque tokens; or make the property total over the wider type, since production will eventually send the values your generator excluded.

import org.scalacheck.Shrink
import org.scalacheck.Prop.forAllNoShrink

// Option 1: keep shrinking, but shrink only within the domain you generate.
// Shrink the offset above the lower bound, so every candidate is still a valid port.
implicit val shrinkPort: Shrink[Port] = Shrink { p =>
  Shrink.shrink(p.value - 1024).filter(_ >= 0).map(d => Port(d + 1024))
}

// Option 2: turn shrinking off for this property when shrunk values are meaningless.
property("handshake") = forAllNoShrink(sessionGen) { s => Handshake.run(s).isRight }

Configuration, seeds and reproducing a failure

A run is controlled by Test.Parameters: by default 100 successful tests, a maximum discard ratio of 5, sizes up to 100 and one worker. Raise minSuccessfulTests for critical code in a nightly job, and lower maxSize when large inputs are slow. Several workers speed up CPU-bound properties but must not share mutable fixtures.

When a property fails in CI the report includes the seed; MUnit's integration prints it with the override that replays it, and plain ScalaCheck takes it through withInitialSeed. Once fixed, copy the shrunk counterexample into an example-based regression test, because a future generator change could silently stop producing it.

import org.scalacheck.Test
import org.scalacheck.rng.Seed

// Reproduce a CI failure locally with the seed it printed.
val params = Test.Parameters.default
  .withMinSuccessfulTests(500)
  .withMaxSize(50)
  .withInitialSeed(Seed.fromBase64("<seed printed by the failing run>").get)

val result = Test.check(params, RleSpec.properties.head._2)
println(result.status)

Worked example: a bug the suite could not see

Consider a run-length encoder that writes each run as a count followed by the character. It has a bug: counts of ten or more are written as their last digit only, so ten a characters encode as 0a. A round trip property is the obvious test.

object Rle {
  // Encodes runs as "<count><char>", e.g. "aaab" -> "3a1b".
  def encode(s: String): String =
    if (s.isEmpty) ""
    else {
      val sb = new StringBuilder
      var run = 1
      for (i <- 1 to s.length) {
        if (i < s.length && s(i) == s(i - 1)) run += 1
        else { sb.append(run.toString.last).append(s(i - 1)); run = 1 } // BUG: keeps last digit
      }
      sb.toString
    }

  def decode(e: String): String =
    e.grouped(2).map(p => p(1).toString * p(0).asDigit).mkString
}

object RleSpec extends Properties("Rle") {
  // A two-letter alphabet makes long runs likely; the default String generator almost never would.
  val runny: Gen[String] = Gen.listOf(Gen.oneOf('a', 'b')).map(_.mkString)

  property("round trip") = forAll(runny) { s => Rle.decode(Rle.encode(s)) == s }
}

Written with the default Arbitrary[String], the property passes every time: default strings draw from a large character range, so ten equal adjacent characters essentially never occur. Adding Prop.classify(maxRun(s) >= 10, "long run") shows the long-run bucket is empty. The generator could not reach the bug, so green meant nothing.

With the two-letter alphabet in runny, runs of ten appear within the first few hundred evaluations. The property fails on a long random string, shrinking deletes and simplifies characters, and the report converges on ten identical characters, the smallest trigger. The exact printed form depends on versions, but the lesson is stable: check every distribution, and prefer domain-shaped generators to type-shaped ones.

The fix writes the full count with a delimiter; the regression test assertEquals(Rle.decode(Rle.encode("a" * 10)), "a" * 10) stays next to the property.

Stateful testing with Commands

Caches, pools and queues fail through sequences of operations, not single inputs. The org.scalacheck.commands.Commands trait generates command sequences, runs them against the real system under test, and checks each result against a simple immutable model.

You define a model State, a Sut type, how to create and destroy a system under test, and how to generate the next command from the model state. Each command runs against the real system, updates the model in nextState, and checks the real result in postCondition. A failing sequence is shrunk to the shortest one that still fails.

import org.scalacheck.Gen
import org.scalacheck.Prop
import org.scalacheck.commands.Commands
import scala.util.Try

// Model: an immutable Map. System under test: the real LRU cache, capacity 3.
object CacheSpec extends Commands {
  type State = Vector[(String, Int)]          // most recently used last
  type Sut   = LruCache[String, Int]

  def canCreateNewSut(s: State, inits: Traversable[State], running: Traversable[Sut]) = true
  def newSut(s: State): Sut = new LruCache[String, Int](capacity = 3)
  def destroySut(sut: Sut): Unit = ()
  def initialPreCondition(s: State): Boolean = s.isEmpty
  def genInitialState: Gen[State] = Gen.const(Vector.empty)

  val key = Gen.oneOf("a", "b", "c", "d", "e")   // small key space forces evictions
  def genCommand(s: State): Gen[Command] =
    Gen.oneOf(for { k <- key; v <- Gen.choose(0, 9) } yield Put(k, v), key.map(Get(_)))

  case class Put(k: String, v: Int) extends UnitCommand {
    def run(sut: Sut): Unit = sut.put(k, v)
    def nextState(s: State): State = (s.filterNot(_._1 == k) :+ (k -> v)).takeRight(3)
    def preCondition(s: State) = true
    def postCondition(s: State, success: Boolean): Prop = success
  }

  case class Get(k: String) extends SuccessCommand {
    type Result = Option[Int]
    def run(sut: Sut): Option[Int] = sut.get(k)
    def nextState(s: State): State = s.find(_._1 == k) match {
      case Some(e) => s.filterNot(_._1 == k) :+ e   // a hit refreshes recency
      case None    => s
    }
    def preCondition(s: State) = true
    def postCondition(s: State, r: Option[Int]): Prop = r == s.find(_._1 == k).map(_._2)
  }
}
// In a suite: property("lru behaves like the model") = CacheSpec.property()

Five keys and capacity three force evictions within a few commands. A typical find is a get that fails to refresh recency, shrunk to three or four commands. Commands can also run sequences in parallel and check the results against some serial order, a practical thread-safety probe, though slower.

Running it in your test framework

MUnit offers munit-scalacheck with a ScalaCheckSuite; assertions inside forAll work because a thrown assertion error is a failure. ScalaTest offers ScalaCheckPropertyChecks through scalatestplus. Integrations are published per framework and ScalaCheck version combination, so match them.

// build.sbt
libraryDependencies += "org.scalameta" %% "munit-scalacheck" % "<version>" % Test

// test file
import munit.ScalaCheckSuite
import org.scalacheck.Prop.forAll

class RleSuite extends ScalaCheckSuite {
  property("round trip") {
    forAll(RleSpec.runny) { (s: String) => assertEquals(Rle.decode(Rle.encode(s)), s) }
  }
}

For effectful code, keep generation pure and run the effect inside the property. Cats Effect and ZIO publish property helpers that handle the plumbing; prefer them to blocking on effects by hand.

Failure modes and trade-offs

The trade-off: a good property replaces dozens of examples but costs more runtime and design effort. Keep examples for documentation and regressions, properties for general rules.

What to do next

1. Pick one serialiser or parser in your codebase and write a round trip property for it today.
2. Add Prop.classify or Prop.collect to every existing property once and look at the distribution; fix generators that never reach edge values.
3. Move domain generators into a shared Generators object and give constrained types their own Shrink instances.
4. Make sure CI prints the failing seed, and document how to replay it locally with withInitialSeed or your framework's override.
5. Turn every shrunk counterexample you fix into a named example-based regression test.
6. Choose one stateful component, such as a cache or a rate limiter, and test it against a model with Commands.