Scala For-Comprehensions, in depth: the desugaring rules, guards and withFilter, value definitions, sequential Futures, mixing effects and making your own type work

A for-comprehension is a rewrite, not a loop

The Scala language specification defines for-comprehensions by translation. The compiler replaces the syntax with ordinary method calls and nested lambdas. Scala 2 performs the rewrite in the parser, so scalac -Xprint:parser shows you the desugared code. Scala 3 does it during typing, so -Xprint:typer is the flag to use there.

The rewrite is purely syntactic, so any type with suitably named methods works, and the meaning of the for depends entirely on those methods. On a List flatMap means for each element; on an Option it means continue only if a value is present; on an Either it means stop at the first Left; on a Future it means when this completes, start the next step.

The five rules

Five left-to-right rewrites cover almost every for-comprehension; the compiler applies them until no for syntax remains.

// 1. One generator with yield
for (x <- xs) yield f(x)            ==>  xs.map(x => f(x))

// 2. Several generators: every one but the last becomes flatMap
for (x <- xs; y <- ys(x)) yield (x, y)
                                    ==>  xs.flatMap(x => ys(x).map(y => (x, y)))

// 3. A guard becomes withFilter on the generator before it
for (x <- xs if p(x)) yield x       ==>  xs.withFilter(x => p(x)).map(x => x)

// 4. A value definition is carried along in a tuple
for (x <- xs; y = g(x)) yield x + y ==>  xs.map { x => val y = g(x); (x, y) }
                                              .map { case (x, y) => x + y }

// 5. No yield: the same shape with foreach, result is Unit
for (x <- xs; y <- ys) println(x + y)
                                    ==>  xs.foreach(x => ys.foreach(y => println(x + y)))

Two observations follow directly. The lambdas nest, so a name bound by an early generator is in scope for every later line, which is why each step can use the results of the previous ones. And without yield the whole chain is built from foreach, which returns Unit: a for without yield over an Option or a Future runs side effects and throws the result away.

Worked example: an order lookup in Either

Suppose a discount service has to find a user, find an order, check that the order belongs to the user and compute a discount. Each lookup can fail with a typed error. With Either the for-comprehension reads like the happy path while still stopping at the first failure.

final case class User(id: Long, email: String, tier: String)
final case class Order(id: Long, userId: Long, totalCents: Long)

sealed trait LookupError
case object UserNotFound   extends LookupError
case object OrderNotFound  extends LookupError
case object NotYourOrder   extends LookupError

def findUser(id: Long): Either[LookupError, User] = ???
def findOrder(id: Long): Either[LookupError, Order] = ???

def discountFor(userId: Long, orderId: Long): Either[LookupError, Long] =
  for {
    user  <- findUser(userId)
    order <- findOrder(orderId)
    _     <- Either.cond(order.userId == user.id, (), NotYourOrder)
    rate   = if (user.tier == "gold") 10 else 0
  } yield order.totalCents * rate / 100

Desugared, this is findUser(userId).flatMap(user => findOrder(orderId).flatMap(order => Either.cond(...).map(_ => ...)), with the value definition rate folded into the final step. If findUser returns Left(UserNotFound), flatMap on a Left returns that Left without calling the lambda, so findOrder is never invoked.

Notice the ownership check. A first attempt usually writes if order.userId == user.id as a guard. That does not compile for Either, because a guard needs withFilter and Either does not have one: if the condition failed there would be no error value to put in the Left. Either.cond (or filterOrElse) supplies that error explicitly, which is the honest version of the guard. The page on Scala Either goes further into error ADTs and accumulating errors instead of stopping at the first.

Guards, patterns and withFilter

A guard if p becomes withFilter on the preceding generator. On collections withFilter is lazy and avoids building an intermediate collection. On Option it simply turns Some into None when the predicate fails. Types that cannot represent an empty result without extra information, such as Either and Future with its failed state, either lack a useful withFilter or implement it by failing: Future.withFilter produces a failed Future with a NoSuchElementException when the predicate is false, which surfaces far from the line that caused it.

Patterns on the left of an arrow interact with the same method. In Scala 2, a generator such as (a, b) <- pairs is translated with a withFilter that drops elements not matching the pattern, even when the pattern can never fail. That is why destructuring a tuple inside a for over Either fails to compile in Scala 2; the community plugin better-monadic-for exists largely to remove that filter. Scala 3 checks whether the pattern is irrefutable for the static type and omits the filter when it is. For a pattern that genuinely can fail, such as Some(x) <- maybes, Scala 3 asks you to write case Some(x) <- maybes to say that filtering is intended; depending on the source level, leaving the case out is a warning or an error.

Rule of thumb: guards on collections and Option; an explicit error-producing step on Either, Try and effect types.

Value definitions, their tuples, and Scala 3.8

A line such as rate = expr inside a for is a value definition. The classic translation cannot simply put a val inside the next lambda when a guard or another generator follows, so it pairs the new value with the bound variables in a tuple and destructures it in the next step. On a hot path over a large collection this means one tuple allocation per element per value definition, plus a pattern match.

Scala 3.8 stabilised SIP-62, known as better fors, which was previewed in 3.7. It changes the translation in three ways that matter day to day. A for may now begin with a value definition, so for { x = 1; y <- Some(2) } yield x + y is legal. A value definition that is not followed by a guard is desugared without the intermediate tuple. And when the final generator binds a variable that the yield returns unchanged, the trailing identity map is dropped, which matters for long-running effect loops because that last map used to keep a growing chain of continuations alive. On Scala 2 and earlier Scala 3 versions the classic rules still apply, so check your compiler version before relying on the new shape, and use -Xprint to confirm.

Futures: sequential by construction

The most expensive misunderstanding of for-comprehensions concerns Future. A Future starts running when it is created, and the second generator of a for is inside the lambda passed to the first one's flatMap. So the second Future is not even created until the first completes.

import scala.concurrent.{ExecutionContext, Future}

def price(sku: String)(implicit ec: ExecutionContext): Future[BigDecimal] = ???
def stock(sku: String)(implicit ec: ExecutionContext): Future[Int] = ???

// Sequential: stock() is not called until price() has completed.
def slow(sku: String)(implicit ec: ExecutionContext) =
  for {
    p <- price(sku)
    s <- stock(sku)
  } yield (p, s)

// Concurrent: both Futures start here, the for only waits for them.
def fast(sku: String)(implicit ec: ExecutionContext) = {
  val pf = price(sku)
  val sf = stock(sku)
  for {
    p <- pf
    s <- sf
  } yield (p, s)
}

If price and stock each take 80 ms, slow takes about 160 ms and fast about 80 ms. The for in fast still desugars to pf.flatMap(p => sf.map(s => (p, s))), but both computations were already in flight before the chain was built. pf.zip(sf) says the same thing more directly. Lazy effect types such as Cats Effect IO and ZIO do not start until run, so assigning them to vals first does not make them concurrent; there you use parTupled, parMapN or zipPar. Thread pools and blocking, which decide how much real concurrency you get, are covered in Futures and ExecutionContext.

Why effect types must line up

Each flatMap must return the type its receiver expects: Option.flatMap needs a function returning an Option, Future.flatMap needs one returning a Future. So every generator in one for must produce the same outer type, and the first generator decides what that type is. A for that starts with a Future[User] and then binds from an Option[Address] does not compile, and the error message is confusing because it describes desugared code you never wrote.

One asymmetric exception: an Option can appear after a collection generator, because an implicit conversion lets it act as an Iterable: for { x <- List(1, 2); y <- Option(x) } yield y compiles and returns a List. Swap the order and it fails, because Option.flatMap will not accept a function returning a List.

When you genuinely have nested effects, such as Future[Option[User]] or IO[Either[Error, A]], you have three options: lift every step into the outer type (for example with Future.successful or by converting the Option to a failed Future), use a monad transformer such as Cats OptionT or EitherT whose flatMap threads both layers, or switch to an effect type with a typed error channel, such as ZIO, where the second layer is built in.

Making your own type work

Because the rewrite only looks for method names, you can make any type usable in a for by giving it map and flatMap, plus withFilter if you want guards and foreach if you want a for without yield. The example below carries a log alongside an optional result.

// A tiny result type that works in for-comprehensions with no type class.
final case class Checked[+A](value: Option[A], log: Vector[String]) {
  def map[B](f: A => B): Checked[B] = Checked(value.map(f), log)

  def flatMap[B](f: A => Checked[B]): Checked[B] = value match {
    case Some(a) => val next = f(a); Checked(next.value, log ++ next.log)
    case None    => Checked(None, log)
  }

  def withFilter(p: A => Boolean): Checked[A] =
    Checked(value.filter(p), if (value.exists(p)) log else log :+ "guard failed")
}

def step(name: String, v: Int): Checked[Int] = Checked(Some(v), Vector(s"$name=$v"))

val r = for {
  a <- step("a", 2)
  b <- step("b", a * 10)
  if b > 5
} yield a + b
// r == Checked(Some(22), Vector("a=2", "b=20"))

Two disciplines keep custom types honest. The methods should obey the usual laws: map(identity) changes nothing, and flatMap is associative, so regrouping steps cannot change the answer. And if your type cannot represent a filtered-out value meaningfully, leave out withFilter so that guards fail at compile time instead of at runtime.

Failure modes

• Accidental sequencing. Independent Futures written as consecutive generators run one after another. Symptom: latency equals the sum of the calls. Fix: create them first or use zip.
• Silent discard without yield. A for over a Future without yield returns Unit and the caller cannot wait for or observe failures. Fix: always yield for effect types.
• Guard on Future. A false guard yields a failed Future with NoSuchElementException, logged far from the cause. Fix: fail with a domain error explicitly.
• Hidden cross products. Two collection generators multiply: 10,000 by 10,000 is 100 million iterations behind two innocent lines. Fix: join through a map or index first.
• Value definitions recomputed or wasted. Expensive work in a value definition before a guard runs for every element, including discarded ones. Fix: put the guard first, or compute inside the yield.
• Stack depth with custom monads. A recursive for over a naive custom flatMap can overflow the stack. Library effect types trampoline; hand-written ones often do not.

Trade-offs: when to use one and when not to

Use a for for two or more dependent steps in one effect type. Prefer a plain map for one step, zip or applicative combinators for independent steps, and a fold or a while loop for performance-critical numeric code over arrays, where closure and tuple allocation show up in profiles. For heavy effect programs, Cats Effect and ZIO make for-comprehensions the main composition tool, with runtimes built to make long flatMap chains cheap and stack safe.

What to do next

1. Desugar three for-comprehensions from your own codebase by hand, then check yourself with -Xprint:parser on Scala 2 or -Xprint:typer on Scala 3.
2. Search for fors over Future with independent generators and start those Futures before the for or use zip.
3. Replace guards on Either, Try and Future with steps that fail with a named error.
4. Check which compiler version you run; on Scala 3.8 or later, simplify fors that worked around the old alias rules.
5. If you have nested effects, pick one strategy per codebase: lifting, transformers or a typed-error effect type.
6. Read Scala Option for the combinators that are clearer than a single-step for.