ScalaTest Framework, in depth: suite styles, matchers, fixtures, async tests, eventually and patience, tags and parallel runs

How a ScalaTest run works

sbt talks to ScalaTest through the standard test-framework interface. When you run Test / test, sbt compiles the test sources and asks ScalaTest which classes are suites: concrete classes that extend Suite and have a public no-argument constructor. sbt passes along any arguments you gave after --, and ScalaTest uses them to filter tests by tag or name.

For each selected suite, ScalaTest instantiates the class. That matters: the class body runs at construction time, and in most styles the body is where tests are registered, so code you write directly in the class body runs once during discovery, not before each test. Then run calls beforeAll if the suite mixes in BeforeAndAfterAll, runs each test through runTest, which calls withFixture, which calls the test body. The body returns an Assertion (or a Future[Assertion] in async suites), and the outcome is Succeeded, Failed, Canceled or Pending. Each step sends events to reporters, which is how the console, JUnit XML files and IDEs all see the same results.

Setting it up

// build.sbt
libraryDependencies ++= Seq(
  "org.scalatest"     %% "scalatest"       % "3.2.19"   % Test,
  // optional: property checks through ScalaCheck
  "org.scalatestplus" %% "scalacheck-1-18" % "3.2.19.0" % Test
)

// Smaller footprint: depend only on the style and matchers you use.
// "org.scalatest" %% "scalatest-funsuite"       % "3.2.19" % Test,
// "org.scalatest" %% "scalatest-shouldmatchers" % "3.2.19" % Test,

The scalatest artifact pulls in every style and matcher. Since 3.2.0, ScalaTest is also published as modules such as scalatest-funsuite and scalatest-shouldmatchers, so you can depend only on what you use. The ScalaCheck integration lives in a separate scalatestplus artifact whose name carries the ScalaCheck version (scalacheck-1-18 here) and whose version carries the ScalaTest version plus a fourth number. Check the current 3.2.x releases when you add them; sbt itself is covered in the sbt guide.

Picking a style, and only one

Styles differ only in how you name and nest tests. Every style supports the same assertions, matchers and fixtures.

import org.scalatest.funsuite.AnyFunSuite
import org.scalatest.wordspec.AnyWordSpec
import org.scalatest.matchers.should.Matchers

class SlugSuite extends AnyFunSuite with Matchers:
  test("lower-cases and joins words with hyphens") {
    Slug("Hello World") shouldBe "hello-world"
  }
  test("drops characters outside a-z and 0-9") {
    Slug("C'est la vie!") shouldBe "cest-la-vie"
  }

class CartSpec extends AnyWordSpec with Matchers:
  "A Cart" when {
    "empty" should {
      "have a zero total" in {
        Cart.empty.total shouldBe BigDecimal(0)
      }
    }
    "given a discount code" should {
      "never go below zero" in {
        Cart.empty.add(Item("pen", 2)).applyCode("ALL100").total shouldBe BigDecimal(0)
      }
    }
  }

A sensible default is AnyFunSuite for unit tests plus one specification style, AnyWordSpec or AnyFlatSpec, if your team likes reading tests as behaviour. Write that choice down and create a base trait, for example trait UnitSuite extends AnyFunSuite with Matchers, so every suite gets the same mix-ins.

Assertions and matchers

Plain assert(a == b) in ScalaTest is a macro: on failure it reports both sides of the comparison, so you get 3 did not equal 4 instead of an empty assertion error. assertResult(expected)(actual) labels which side is which, and assertThrows[T] and intercept[T] check exceptions; intercept returns the exception so you can inspect it.

Matchers add a readable vocabulary. With org.scalatest.matchers.should.Matchers you write x shouldBe 3, list should contain (42), s should startWith ("Ac"), x should be >= 0 and d shouldBe 1.0 +- 0.01. must.Matchers is identical with the word must. A few helper traits remove boilerplate that otherwise hides intent:

import org.scalatest.{Inside, Inspectors, OptionValues, EitherValues}

class ParserSuite extends AnyFunSuite with Matchers
    with Inside with Inspectors with OptionValues with EitherValues:

  test("parses a valid order line") {
    val r = Parser.order("1001|Acme|250.00")
    r.isRight shouldBe true
    inside(r.value) { case Order(id, customer, amount) =>
      id shouldBe 1001L
      customer should startWith ("Ac")
      amount shouldBe BigDecimal("250.00") +- BigDecimal("0.001")
    }
  }

  test("every parsed amount is non-negative") {
    val orders = sample.flatMap(Parser.order(_).toOption)
    forAll(orders) { o => o.amount should be >= BigDecimal(0) }
  }

  test("rejects a short line with a useful message") {
    Parser.order("1001|Acme").left.value should include ("expected 3 fields")
  }

  test("an unknown id throws") {
    val e = intercept[NoSuchElementException] { repo.get(-1L) }
    e.getMessage should include ("-1")
  }

Inside pattern-matches a value and reports which part failed. Inspectors.forAll checks every element of a collection and reports the index and value of the one that failed, which is far better than a loop of asserts. OptionValues and EitherValues give .value and .left.value, which fail the test with a clear message instead of throwing NoSuchElementException from .get.

Fixtures: shared, per-test and loaned

Test setup comes in three sizes, and ScalaTest has a mechanism for each. Expensive resources shared by a whole suite, such as a test database container, go in beforeAll and afterAll from BeforeAndAfterAll. Per-test setup and teardown go in beforeEach/afterEach or, better, an override of withFixture, which wraps the test and can use try/finally. Resources only some tests need are best passed with the loan pattern: a method that creates the resource, lends it to a function, and cleans up afterwards.

import org.scalatest.{BeforeAndAfterAll, Outcome}

class OrderRepoSuite extends AnyFunSuite with Matchers with BeforeAndAfterAll:

  private var db: TestDatabase = _                    // expensive: one per suite

  override def beforeAll(): Unit = db = TestDatabase.start()
  override def afterAll(): Unit  = db.stop()

  // Runs around every test: give each test a clean schema, always clean up.
  override def withFixture(test: NoArgTest): Outcome =
    db.resetSchema()
    try super.withFixture(test)
    finally db.truncateAll()

  // Loan pattern: only tests that need a temp directory ask for one.
  // (os.* comes from the os-lib library, "com.lihaoyi" %% "os-lib".)
  def withTempDir(body: os.Path => Any): Unit =
    val dir = os.temp.dir()
    try body(dir) finally os.remove.all(dir)

  test("saves and reloads an order") {
    val repo = OrderRepo(db.dataSource)
    repo.save(Order(1L, "Acme", BigDecimal(10)))
    repo.get(1L).customer shouldBe "Acme"
  }

  test("exports orders to a file") {
    withTempDir { dir =>
      OrderRepo(db.dataSource).exportTo(dir / "orders.csv")
      os.exists(dir / "orders.csv") shouldBe true
    }
  }

Do not put setup in the class body. It runs once when the suite is constructed, its state is shared by every test, and a test that changes it changes the next test's starting point. For cases where every test needs its own fixture value as a parameter, FixtureAnyFunSuite lets tests receive it directly, at the cost of a little more ceremony.

Async code: async suites, ScalaFutures and Eventually

There are two ways to test code that returns a Future. Async suites such as AsyncFunSuite let a test return Future[Assertion]; ScalaTest waits for it without blocking a thread. By default these suites run one test at a time on a serial execution context, so tests are still deterministic about ordering within the suite. Ordinary suites can mix in ScalaFutures and call .futureValue or whenReady, which block until the future completes or the patience timeout expires.

For conditions that become true some time later, such as a message being indexed or a cache being refreshed, Eventually retries a block until it passes or the timeout expires. Both use a PatienceConfig with a timeout and a retry interval:

import org.scalatest.funsuite.AsyncFunSuite
import org.scalatest.concurrent.{Eventually, ScalaFutures}
import org.scalatest.time.{Millis, Seconds, Span}
import scala.concurrent.Future

// Tests return Future[Assertion]; nothing blocks.
class PriceClientAsyncSuite extends AsyncFunSuite with Matchers:
  test("fetches a price") {
    client.price("ABC").map(p => p should be > BigDecimal(0))
  }

// Blocking style in an ordinary suite, with explicit patience.
class IndexerSuite extends AnyFunSuite with Matchers
    with ScalaFutures with Eventually:

  implicit override val patienceConfig: PatienceConfig =
    PatienceConfig(timeout = Span(5, Seconds), interval = Span(50, Millis))

  test("a submitted document becomes searchable") {
    indexer.submit(Doc("42", "parallel streams")).futureValue shouldBe Accepted
    eventually {
      search("parallel").map(_.id) should contain ("42")
    }
  }

This replaces Thread.sleep, the most common cause of slow and flaky Scala suites: a sleep is too long on a fast machine and too short on a loaded CI runner, while eventually returns as soon as the condition holds and waits up to a clear limit. Set patience per suite to what the system really needs, and raise it on CI with a scaling factor instead of editing every test. How execution contexts behave under test is covered in futures and execution contexts.

Table-driven and property checks

When the same rule must hold for many inputs, list them in a table instead of copying a test. TableDrivenPropertyChecks runs the body for every row and reports the row that failed. When you can state a rule that holds for all inputs, ScalaCheckPropertyChecks generates inputs for you:

import org.scalatest.prop.TableDrivenPropertyChecks
import org.scalatestplus.scalacheck.ScalaCheckPropertyChecks

class SlugTableSuite extends AnyFunSuite with Matchers
    with TableDrivenPropertyChecks with ScalaCheckPropertyChecks:

  private val cases = Table(
    ("input",          "expected"),
    ("Hello World",    "hello-world"),
    ("  padded  ",     "padded"),
    ("a--b",           "a-b"),
    ("",               "")
  )

  test("known inputs") {
    forAll(cases) { (in, out) => Slug(in) shouldBe out }
  }

  test("slugs are idempotent") {
    forAll { (s: String) => Slug(Slug(s)) shouldBe Slug(s) }
  }

Tables document the cases someone thought of; properties find the ones nobody did. Use both: a table for known edge cases and regressions, a property for invariants such as idempotence or round-trips. Generators, shrinking and seeds are covered in the ScalaCheck article linked above.

Tags, selection and parallel runs

Tags let you run subsets of a suite. Declare a tag object, attach it to tests, and select or exclude it from the runner. The runner flags are -n to include a tag, -l to exclude one, -z to select tests whose names contain a substring, and -oD to print durations on the console reporter. From sbt, put them after --:

import org.scalatest.Tag
object Slow extends Tag("com.example.Slow")
object Db   extends Tag("com.example.Db")

test("rebuilds the full index", Slow, Db) { /* ... */ }

// sbt shell
// Test / testOnly *RepoSuite                       // one suite (glob)
// Test / testOnly *RepoSuite -- -z "reloads"       // tests whose name contains "reloads"
// Test / testOnly -- -l com.example.Slow           // everything except Slow
// Test / testOnly -- -n com.example.Db -oD         // only Db tests, print durations

Parallelism happens at two levels. sbt runs different suites in parallel by default, controlled by Test / parallelExecution, so suites must not share mutable global state such as a fixed port or a single database schema. Within a suite, tests run sequentially unless the suite mixes in ParallelTestExecution, which runs each test in its own instance of the suite class, so instance state is not shared. Turn on in-suite parallelism only for suites whose tests are independent and slow enough to benefit; the usual fix for a slow suite is removing sleeps and real network calls.

Failure modes

• Setup code in the class body shared by every test, so test order changes results.
• Thread.sleep instead of eventually, making suites slow locally and flaky on CI.
• Default patience too short for CI, causing timeouts that never happen on laptops.
• Blocking with Await.result inside async suites, which can deadlock on the serial execution context.
• Suites that share a port, file or schema failing only when sbt runs them in parallel.
• Calling .get on Option or Either in tests, producing NoSuchElementException instead of a readable failure.
• Mixed styles and matcher families across a codebase, so every suite reads differently.
• Slow tests never tagged, so the fast local loop is as slow as CI.

What to do next

1. Pick one unit-test style and at most one specification style, and create a base trait that mixes in Matchers and the helpers you use.
2. Replace .get and bare loops in assertions with OptionValues, EitherValues, Inside and Inspectors.
3. Move setup out of class bodies into beforeAll, withFixture or loan methods.
4. Replace every Thread.sleep in tests with eventually or futureValue, and set PatienceConfig per suite.
5. Turn repeated tests into tables, and add one property for each core invariant.
6. Tag slow and integration tests, and make the default local command exclude them with -l.
7. Check that suites can run in parallel: no fixed ports, shared files or shared schemas.
8. Run with -oD on CI, sort by duration and fix the slowest suites first.