Why architecture matters here
Architecture matters here because a semaphore is usually standing between your application and a resource that has a hard capacity limit — and getting the bound wrong fails in opposite, equally bad ways. Set the permit count too high and the semaphore provides no protection: all N+1 threads rush a connection pool that only has N connections, the pool blocks or errors, and you have merely moved the contention somewhere less observable. Set it too low and you throttle throughput below what the resource can actually sustain, leaving capacity idle while requests queue. The permit count is a capacity contract, and it must match the real downstream limit.
The reason to reach for a semaphore rather than an ad-hoc counter is that the increment/decrement and the blocking must be atomic together. A hand-rolled 'if count > 0 then count-- else wait' has a race: two threads can both read count as 1, both decrement, and both proceed, over-subscribing the resource. The semaphore's job is to make the test-and-decrement and the enqueue-if-unavailable a single indivisible operation, which requires either a lock or a lock-free compare-and-swap loop underneath. This is the entire value proposition — correct, atomic admission control — and it is why you should use the platform's semaphore rather than reinventing it.
Semaphores also give you a place to attach backpressure and fairness policy. Because blocked threads sit in a queue, you can decide whether they are served in arrival order (fair) or whether an arriving thread may 'barge' ahead of the queue when a permit happens to be free (non-fair, higher throughput). And because acquire can offer a timed or non-blocking variant (tryAcquire), the semaphore becomes a natural admission gate that can shed load — reject or degrade — rather than block unboundedly, which is often the difference between a slow service and a hung one.
There is one more architectural reason to prefer a semaphore over informal throttling: it makes the concurrency limit a single, named, observable object. When admission control is scattered across ad-hoc counters, retry loops, and thread-pool sizes, no one can say what the system's actual concurrency ceiling is or why. A semaphore concentrates that policy in one place with one number that can be logged, alerted on, tuned, and reasoned about. In a service with several downstream dependencies, giving each its own semaphore isolates their limits — a slow dependency exhausts only its own permits and sheds load for that path, leaving unrelated paths fully available. This bulkheading is a primary defense against one slow dependency dragging down an entire service.
The architecture: every piece explained
At the core is a single atomic integer, the permit count, initialized to the maximum concurrency N. acquire() attempts to atomically decrement it: in a lock-free implementation it reads the current value, and if it is positive it compare-and-swaps it down by one; success means the caller holds a permit and proceeds. If the count is zero, the caller cannot decrement and must block. release() atomically increments the count and, if any threads are waiting, wakes one. The invariant is that (permits currently held) + (permit count) never exceeds N.
Blocking is handled by a wait queue. In Java's Semaphore, this is the AbstractQueuedSynchronizer (AQS): a threads that cannot get a permit is wrapped in a node and appended to a CLH-style FIFO queue, then parked (the OS deschedules it so it consumes no CPU). When release increments the count, it unparks the queue head, which re-attempts the acquire. AQS holds the shared state (the permit count) and the queue together, so the decrement and the enqueue are coordinated under one consistent synchronizer rather than two racing structures.
The fairness policy decides what happens in the race between an arriving thread and a queued waiter when a permit is released. A non-fair semaphore lets a newly arriving thread grab the just-freed permit immediately (barging), even though older threads wait — this maximizes throughput because it avoids the context-switch of waking a parked thread, at the cost of possible starvation of unlucky waiters. A fair semaphore forces arrivals to check the queue first and go to the back if anyone is waiting, guaranteeing FIFO service and no starvation, but paying more context switches. A binary semaphore is simply one initialized to a count of 1; it resembles a mutex but differs in that any thread may release it, so it lacks a mutex's ownership and reentrancy semantics and should not be used where those matter.
End-to-end flow
Trace a request-handling thread that needs one of ten database connections guarded by a semaphore initialized to 10. It calls acquire(); the count is 7 (three connections in use), so the CAS decrements it to 6 and the thread proceeds immediately, checks out a connection, runs its query, and — in a finally block — checks the connection back in and calls release(), incrementing the count back to 7. No blocking occurred; the semaphore simply tracked outstanding usage. This is the common fast path: acquire, use, release, all without touching the wait queue.
Now suppose all ten connections are in use and the count is 0. Three more request threads call acquire(); each finds the count at zero, fails the CAS, enqueues itself in the wait queue, and parks. They consume no CPU while blocked. When one of the in-flight requests finishes and calls release(), the count would go to 1, but instead of leaving it there the synchronizer hands the permit directly to the head of the wait queue: it unparks the first waiter, which wakes, acquires the permit (count stays effectively allocated), checks out the freed connection, and runs. The other two remain parked until further releases.
Consider the load-shedding variant. A thread calls tryAcquire(50, MILLISECONDS) instead of blocking indefinitely. If a permit becomes available within 50ms it proceeds; if not, tryAcquire returns false and the thread takes an alternate path — return a 503, serve a cached/degraded response, or push the work to a queue. This converts unbounded queuing into bounded latency: under overload the service sheds excess load quickly rather than accumulating a growing backlog of blocked threads, each holding memory and, worse, often holding other resources while they wait.
The load-shedding path also composes with retries in a way worth understanding. If a caller responds to a tryAcquire failure by immediately retrying in a tight loop, it converts backpressure into a busy-wait storm that makes the overload worse. The correct pattern is to treat a permit-acquisition timeout as a first-class rejection: return the error up the stack, let a higher layer apply jittered backoff or route to a fallback, and shed the request cleanly. In this way the semaphore is not merely a limiter but the point at which the system decides, under overload, what work to not do — and making that decision decisively is what keeps a saturated service responsive for the requests it does accept.