Scala Closures, in depth: environments, inspecting captures, lifetime, threads, serialization, non-local return and cost

First principles: free variables and environments

Take the expression (x: Int) => x + offset. The name x is bound by the lambda itself; offset is free, meaning it refers to something defined outside the lambda. A function with no free variables is just code. A function with free variables is only meaningful together with a binding for each one, and that pair (code plus bindings) is what the term closure means. The set of bindings is the environment.

On the JVM there is no stack frame that can outlive its method, so the environment has to be copied into a heap object. Since Scala 2.12, and in Scala 3, the compiler does this in two steps. First, it lifts the lambda body into a private synthetic method on the enclosing class, with a name like $anonfun$process$1, whose parameters are the captured values followed by the lambda's own parameters. Second, at the point where the lambda appears, it emits an invokedynamic call to LambdaMetafactory, which produces an object implementing Function1 (or the relevant SAM type) whose fields hold the captured values. Calling apply on that object calls the lifted method with the fields plus the argument.

Three consequences follow. The environment holds whatever the body mentions, resolved when the closure is created. It lives exactly as long as the function object is reachable. And a closure that mentions nothing from outside needs no fields, so the runtime can reuse one instance.

What exactly gets captured: the rules that matter in practice

The capture rules reduce to a question: what does the body refer to after name resolution? A local val or a method parameter is captured by value. A local var is rewritten into a scala.runtime.IntRef, ObjectRef or similar cell that the closure and the enclosing method share; a @volatile var becomes VolatileIntRef or VolatileObjectRef, which gives visibility but still not atomic read-modify-write. Anything reached through the enclosing object, whether a field, a method or an inner object, is reached through this, so the closure captures this.

Less obvious cases follow from the same rule. A by-name parameter p: => A is a Function0; a closure that mentions p captures the thunk and re-evaluates it each time. A local lazy val is captured as a holder, so the closure may trigger its initialisation later, on another thread. An implicit or given parameter used by the body is captured like an explicit one, which is how an ExecutionContext or a logger sneaks into an environment.

The useful habit: list a lambda's free names and ask of each one whether it is a local, a cell, or reached through this.

Seeing the environment: compiler output and SerializedLambda

You do not have to guess. The compiler can print its trees after the lifting phase. In Scala 2 pass -Xprint:lambdalift (2.13 also accepts -Vprint:lambdalift); in Scala 3 the phase is lambdaLift, so pass -Xprint:lambdaLift. Look for the synthetic $anonfun method: its leading parameters are the captures, and a leading $this parameter means the enclosing instance is captured. javap -p -c on the compiled class shows the same thing at bytecode level.

At runtime, because Scala lambdas are serializable, each has a writeReplace method returning a java.lang.invoke.SerializedLambda that names the implementation method and lists every captured argument, the same trick Spark's closure cleaner uses:

import java.lang.invoke.SerializedLambda

final case class Captures(impl: String, args: Seq[AnyRef])

def captures(f: AnyRef): Captures = {
  val m = f.getClass.getDeclaredMethod("writeReplace")
  m.setAccessible(true)
  val sl = m.invoke(f).asInstanceOf[SerializedLambda]
  Captures(
    impl = s"${sl.getImplClass}.${sl.getImplMethodName}",
    args = (0 until sl.getCapturedArgCount).map(sl.getCapturedArg)
  )
}

class Pricer(val rate: Double, val audit: Vector[String]) {
  def viaField: Double => Double = x => x * rate         // captures this
  def viaLocal: Double => Double = { val r = rate; x => x * r } // captures a Double
}

val p = new Pricer(1.2, Vector.fill(100000)("event"))
captures(p.viaField).args.map(_.getClass.getSimpleName)  // Seq(Pricer)
captures(p.viaLocal).args                                 // Seq(1.2)

Use that helper as an assertion in tests of code that registers long-lived callbacks or ships functions to a cluster; it catches a new field reference the day it is written, not months later in a heap dump.

Lifetime: how a small closure pins a large graph

The environment lives as long as the function object does, and function objects get stored in long-lived places: listener registries, caches, scheduled tasks, metric gauges, retry policies. A closure that captured this keeps the entire enclosing object reachable, and everything it references.

A typical incident: a request handler registers metrics.gauge("queue")(() => queue.size), where queue is a field of a per-request object that also holds the parsed body. The registry is global, so every request leaves behind a closure holding a request object. The heap grows with traffic, and a heap dump shows thousands of instances whose path to the GC root runs through a synthetic class whose name contains $$Lambda.

The fixes are mechanical. Bind what the body needs to a local val before the lambda. Make every registration return a handle that an owner must close. Avoid function values as map keys: two identical lambdas are distinct objects, so the entry you try to remove is never found, the identity trap described in Scala Methods vs Functions. For weak semantics, hold the target in a WeakReference inside the closure.

Closures across threads

A closure passed to a Future, a parallel collection or a thread pool runs later and possibly concurrently, so its environment is shared state under the Java memory model, as discussed in Scala Futures and ExecutionContext.

Captured vals are safe if the referenced objects are immutable. Captured var cells are plain fields mutated without synchronisation, so var n = 0; (1 to 1000).par.foreach(_ => n += 1) loses updates. A closure that runs on another thread should capture only immutable values; shared mutable state should be an explicit AtomicLong, LongAdder or concurrent map, chosen deliberately rather than created by the compiler.

Actors add a sharper version. In an Akka or Pekko classic actor, sender() and the actor's fields are only valid while the current message is processed. A future.onComplete callback that calls sender() or mutates a field runs after the actor has moved on. Capture val replyTo = sender() first, and send results back as a message with pipeTo.

Closures across machines

Spark serialises the closure you pass to map and ships it with each task, so everything in the environment must be serializable and is shipped every time. Task not serializable almost always means the environment contains a non-serializable this; the quieter failure is a serializable but large enclosing object riding along with every task.

The local-val fix works here too. Move stateless helpers into an object, copy configuration into locals, and build clients and connections inside mapPartitions so each executor creates its own. Large read-only data belongs in a broadcast variable, covered in Spark Broadcast Variables.

Control flow inside closures: return, boundary and break

A return inside a lambda returns from the enclosing method, not the lambda. Because the lambda body is a separate JVM method, the compiler throws a NonLocalReturnControl carrying the value and catches it in the enclosing method. A catch { case _: Throwable => } in between swallows it (NonFatal correctly lets it pass), and if the lambda runs after the enclosing method has returned, for example in a Future, there is no frame to return to and the exception surfaces somewhere unrelated.

Scala 3 deprecated returning from nested anonymous functions in 3.2.0, and 3.3.0 added scala.util.boundary with break as the explicit, lexically scoped replacement; when the break is in the same method the compiler can turn it into a jump:

import scala.util.boundary, boundary.break

// Before: non-local return, deprecated since Scala 3.2.0
def firstNegativeOld(xs: List[Int]): Option[Int] = {
  xs.foreach { x => if (x < 0) return Some(x) }
  None
}

// After: explicit boundary, same result, no hidden handler
def firstNegative(xs: List[Int]): Option[Int] =
  boundary:
    for x <- xs do
      if x < 0 then break(Some(x))
    None

// Often simpler still: xs.find(_ < 0)

In Scala 2, use collection methods (find, collectFirst, exists) or a tail-recursive loop instead of return inside lambdas, and never write return inside a callback that runs asynchronously.

What closures cost

A non-capturing lambda costs one shared instance, created on the first call to its invokedynamic site. A capturing lambda allocates a small object each time the expression is evaluated. The JIT often removes that allocation through inlining and escape analysis, but only when the call site is monomorphic enough to inline.

The second cost is megamorphic dispatch: inside map, f.apply(x) sees every lambda class ever passed to map, and once a call site has seen more than two receiver types HotSpot stops inlining through it. The third is boxing: generic A => B code passes Int as java.lang.Integer, although Scala 2's Function1 is specialized for a few primitive types.

These costs matter only in inner loops a profiler has pointed at. There, hoist the closure out of the loop, use a while loop over an array, or use Scala 3 inline methods, which inline the lambda body so no function object exists. Everywhere else, write the clear version.

Worked example: making a pricing service safe to ship and cheap to keep

Here is a service class with three closure problems at once: a metrics gauge that captures the whole service, a Spark job whose lambda captures the service, and a counter mutated from a parallel loop.

class PricingService(conf: Conf, client: HttpClient, metrics: Metrics) {
  private val rates: Map[String, Double] = loadRates(conf)
  private var priced = 0

  metrics.gauge("rates")(() => rates.size)              // pins service forever

  def priceAll(orders: RDD[Order]): RDD[Double] =
    orders.map(o => o.total * rates(o.region))          // captures this: not serializable

  def priceLocal(orders: Seq[Order]): Seq[Double] =
    orders.par.map { o => priced += 1; o.total * rates(o.region) }.seq  // races
}

After the refactor every lambda captures only immutable locals, the gauge is owned and closable, and the counter is a concurrent object chosen on purpose:

class PricingService(conf: Conf, client: HttpClient, metrics: Metrics) extends AutoCloseable {
  private val rates: Map[String, Double] = loadRates(conf)
  private val priced = new java.util.concurrent.atomic.LongAdder

  private val gauge = { val n = rates.size; metrics.gauge("rates")(() => n) }
  def close(): Unit = gauge.close()

  def priceAll(orders: RDD[Order]): RDD[Double] = {
    val r = orders.sparkContext.broadcast(rates)       // ship once per executor
    orders.map(o => o.total * r.value(o.region))       // captures only the broadcast handle
  }

  def priceLocal(orders: Seq[Order]): Seq[Double] = {
    val (r, counter) = (rates, priced)                  // locals, not fields
    orders.par.map { o => counter.increment(); o.total * r(o.region) }.seq
  }
}

Then assert with the captures helper that the lambda passed to orders.map captures exactly one broadcast handle, so the build fails if someone writes rates inside it again.

Failure modes at a glance

What to do next

1. Grep your code for lambdas passed to registries, schedulers, gauges and caches; for each, list its free names and check whether any resolves through this.
2. Add the SerializedLambda captures helper to your test utilities and assert on the captures of every closure that is stored long-term or shipped to a cluster.
3. Compile one module with -Xprint:lambdalift (Scala 2) or -Xprint:lambdaLift (Scala 3) and read three of its lifted methods, so the transformation stops being abstract.
4. Search for vars captured by lambdas that run on other threads and replace them with immutable values or explicit concurrent types.
5. Replace every return inside a lambda with boundary/break on Scala 3.3+ or with a collection method on Scala 2, and turn on deprecation warnings so new ones fail review.
6. Only after a profiler points at a hot loop, hoist closures out of it or use Scala 3 inline; do not micro-optimise elsewhere.