Core concept
async-profiler is a low-overhead, event-driven sampling profiler for Java and other JVM languages. It operates as a native agent running outside the JVM, periodically interrupting the application to collect stack traces. This design makes it production-safe compared to bytecode instrumentation tools like JProfiler or Yourkit. The key insight behind async-profiler is that you don't need to instrument code to see where time is spent—sampling at OS level captures the truth of what the CPU is executing at any moment.
The profiler captures call stacks at configurable intervals (typically 10-100Hz depending on workload), building a hierarchical view of where CPU time and wall-clock time are spent. Unlike traditional JVM profilers that use JVM-level hooks, async-profiler uses OS-level mechanisms: perf_event_open() on Linux, dtrace on macOS, and ETW (Event Tracing for Windows) on Windows. This approach captures not just Java code but also time spent in native libraries and JVM internals (GC, JIT compilation). For a microservice handling 10,000 requests per second, sampling at 100Hz means collecting ~100 stack traces per second, creating a statistically representative picture without collecting millions of traces.
A key strength of async-profiler is visibility into the entire call stack: from application code down through library internals, the JVM runtime, and OS-level system calls. If your application is spending time in Java String operations that delegate to native memory access, or if the GC is running unexpectedly, or if page faults are causing context switches—all of these appear in the flame graph. Other JVM profilers often hide these layers or miss them entirely.
Output formats include FlameGraph SVG (interactive visualization), plain text profiles, Java Flight Recorder (JFR) format for integration with JDK Mission Control, and JSON for programmatic analysis. The flame graph shows call hierarchy with width proportional to CPU time, making hot paths immediately obvious. Real-world profiling sessions can show whether bottlenecks are in application code, library code, or JVM internals like garbage collection. The SVG output is particularly powerful for collaborative analysis: you can save it, share it via email, and open it in any browser without additional tools.
Comparison to other profilers. JProfiler and Yourkit use bytecode instrumentation or JVMTI hooks, adding 5-20% overhead even when collecting minimal data. JFR (Java Flight Recorder), built into the JDK since Java 11, offers similar functionality but operates at lower overhead (~2-3%) by using kernel integration. async-profiler complements JFR: it excels at CPU and wall-clock profiling, while JFR has better memory allocation tracking. The choice depends on what you're investigating—CPU hotspots favor async-profiler; memory leaks favor JFR or heap analysis tools.
How it works
Startup and attachment. async-profiler is attached to a running JVM using the JVMTI (JVM Tool Interface) agent mechanism. Unlike startup agents that must be configured at JVM launch, async-profiler can be attached dynamically to already-running processes using async-profiler attach $PID or via jcmd commands. The native agent library (libasyncProfiler.so for Linux, .dylib for macOS, .dll for Windows) is loaded into the JVM process with full access to the process memory space. This dynamic attachment capability is invaluable for production debugging—you can start profiling a problematic service without restarts or configuration changes.
Sampling mechanism and signal handling. The profiler installs OS-level signal handlers (SIGPROF on Unix-like systems) that fire at regular intervals determined by the sampling frequency. When a signal arrives, the async-profiler signal handler interrupts the current thread's execution and captures its call stack. The challenge here is that the handler runs asynchronously—it can interrupt the thread at any point, possibly in the middle of JVM internal operations. To avoid deadlock or corruption, the signal handler must use lock-free, signal-safe operations only.
Stack unwinding is highly platform-specific. On Linux with DWARF debug information available, the profiler uses that metadata to walk the stack correctly. When debug info is missing, it falls back to frame-pointer-based unwinding or, for JIT-compiled code, uses internal JVM structures. The JVM provides the AsyncGetCallTrace API (internal but stable) that async-profiler calls to obtain the Java portion of the stack. Combining platform unwinding with JVM unwinding yields the full picture: native library code → JVM internals → application Java code.
Stack aggregation and in-memory tracking. Raw stack traces are not stored individually; instead, each unique call stack is collected into a hash table and a count is incremented. Sampling at 100Hz for a 10-minute profile generates approximately 60,000 samples. If the same hot code path appears in 50,000 of those samples, the aggregated data stores only one entry with count=50,000. This in-memory aggregation keeps heap usage minimal (typically under 100MB even for long-running profiles). The trade-off is that fine-grained timing information (exact duration of individual calls) is lost, but aggregate statistics are preserved.
Profiling modes. async-profiler supports multiple sampling events: cpu (CPU cycles), wall (wall-clock time, captures blocking I/O), allocations (object allocation rate), and cache-misses (if hardware performance counters are available). CPU profiling shows where the CPU spends time executing instructions. Wall-clock profiling captures all time, including sleep and I/O blocking, useful for identifying where requests spend time even if not consuming CPU. Allocation profiling tracks new object creation, essential for garbage collection tuning.
Output formats and rendering. After profiling ends, async-profiler converts the aggregated data into multiple output formats. FlameGraph SVG is the most user-friendly: it's a self-contained, interactive visualization where width represents sample count (time proportion). Hover over frames to see exact percentages; click to zoom into subtrees. JFR format allows integration with JDK Mission Control, a GUI tool for post-analysis and comparison with other profiles. JSON export enables programmatic analysis—for example, automated detection of performance regressions by comparing profiles across CI/CD builds. Plain text output is useful for automation and log aggregation.
Trade-offs + gotchas
Overhead and measurement impact. At 100Hz CPU sampling, async-profiler adds negligible overhead (typically <1% for CPU profiling). This is orders of magnitude lower than instrumentation-based tools like JProfiler (5-20% overhead). Wall-clock or memory allocation profiling requires additional tracking—collecting allocation site information on every new object can add 2-5% overhead. Sampling frequency is configurable: running at 10Hz reduces overhead further but may miss short-duration hot spots. The key insight is that profiling should not distort the behavior being measured, and async-profiler succeeds where instrumentation fails.
Accuracy vs. sampling bias. Sampling introduces statistical bias: very short-lived functions and I/O-bound code with minimal stack time may be underrepresented or completely absent from profiles. A function that executes in 10 nanoseconds and runs trillions of times per second can still be invisible at 100Hz sampling if threads are mostly blocked elsewhere. This is not a bug but a fundamental characteristic of statistical sampling. The upside is that it provides qualitative insights ("where is time going") accurately, even if absolute precision on specific methods is sacrificed. For comparative analysis—did this code change make things faster or slower—sampling provides high fidelity because the bias is consistent.
Platform dependencies and permission requirements. async-profiler relies on OS-level profiling APIs that vary significantly by platform. On Linux, perf_event_open() requires kernel 4.6+ and either root privileges or CAP_SYS_ADMIN capability. Systems with strict security hardening (SELinux, AppArmor) may require additional policy changes to allow profiling. macOS support requires JDK 11+ and Xcode command-line tools; M-series ARM Macs have better support with recent JDK versions. Windows support uses ETW (Event Tracing for Windows) and is less mature than Linux support, with occasional edge cases in complex scenarios. Containers and cloud VMs present additional challenges: many orchestration platforms disable perf_event_open by default or restrict it to privileged pods. Testing profiler attachment in your exact deployment environment is essential before relying on it for production troubleshooting.
Reproducibility and statistical confidence. Because sampling is probabilistic, profiles are never identical run-to-run. The 7th-ranked hot function in one profile might be 5th-ranked in another due to random variation. For CI/CD pipelines using profiles for regression detection, collect multiple profiles (at least 3-5) and analyze aggregate statistics rather than treating individual profiles as ground truth. Aim for at least 1000-2000 samples per profile to achieve statistical confidence that hot spots are real and not artifacts of sampling variance. Profile duration matters: a 30-second profile of a service handling 1,000 RPS may contain only a few hundred Java samples, whereas a 5-minute profile contains 30,000+ samples, enabling reliable detection of smaller bottlenecks.
JIT compilation and warmup effects. Profiles taken during JVM warmup (first few minutes of execution) can be misleading because code is still being JIT-compiled. Methods appear hot during warmup because the interpreter is running them, but they disappear from profiles once they're JIT-compiled to efficient native code. Best practice: let the application warm up for 1-2 minutes before capturing profiles. Some tools auto-detect JIT compilation events and can be configured to exclude them from profiles.
Integration with monitoring and CI/CD. async-profiler can export profiles as JSON or JFR, enabling integration with automated regression detection. However, setting up production profiling requires careful infrastructure: where are profiles stored, how are they managed, who has access? For one-off troubleshooting, async-profiler is straightforward. For continuous production profiling, consider profiling services like Datadog's Continuous Profiling or Grafana Pyroscope, which handle storage, aggregation, and comparison automatically. JFR integration with JDK Mission Control works but is sometimes clunky; the native async-profiler flame graph output is often more intuitive.
Symbol resolution and debug information. To map addresses back to method names, the profiler needs symbol information. For Java methods, this comes from the JVM itself. For native libraries (OpenSSL, zlib, OS system calls), the profiler needs access to debug symbols (.so files with DWARF info or separate .debug files). Stripped binaries or minimal Docker images without debug symbols result in numeric addresses instead of function names in the flame graph. Solution: keep debug symbols available, or build with -g compiler flags. Most major distributions provide debug symbol repositories (Ubuntu: -dbgsym packages).