VisualVM, in depth: connecting safely, reading samplers honestly and solving leaks with heap dumps

Where VisualVM fits among Java profiling tools

Several tools overlap, and choosing the wrong one wastes an afternoon. VisualVM is the interactive generalist: point it at a JVM and look. It is strongest at heap dump analysis and at the first, exploratory look at a misbehaving process. It is weakest at low-overhead, always-on recording in production, where Java Flight Recorder belongs, and at precise CPU attribution, where async-profiler is better because it does not suffer from safepoint bias.

These combine well. VisualVM 2.x can open .jfr recordings and .hprof heap dumps, so a common workflow is to capture on the server with jcmd and analyse on your laptop. For recording strategy see Java Flight Recorder, and for sampling without safepoint bias see async-profiler.

How VisualVM reaches a JVM

VisualVM finds local JVMs the same way jps does: each HotSpot JVM publishes a memory-mapped performance data file in a per-user hsperfdata directory under the system temp directory. VisualVM lists those processes, then uses the Attach API to load its agent into the target. Attach only works between processes owned by the same operating system user, and it fails if the target was started with -XX:+DisableAttachMechanism. A JVM started with -XX:+PerfDisableSharedMem does not publish the file, so it never appears in the list even though attach might work.

Remote JVMs are reached over JMX. VisualVM reads the platform MBeans for memory, threads, class loading and the operating system, can request a thread dump, and can ask the remote JVM to write a heap dump through the HotSpot diagnostic MBean. That dump is written on the remote host's disk, not yours. The third route, jstatd, is a small RMI server that exports the jvmstat counters for every JVM on a host; it gives you the monitoring graphs but no thread dumps, sampling or heap dumps. Which sampler features work over JMX has changed between releases, so check in your version before you plan an investigation around it.

What each tab actually measures

Each tab measures something different, and the differences explain most of the confusion people report.

• Overview shows the command line, JVM flags and system properties. Read it first: half of all performance tickets are a wrong -Xmx or a missing GC flag.
• Monitor plots CPU, heap used versus committed, metaspace, loaded classes and live threads, and has buttons to request a GC and take a heap dump. A healthy heap draws a sawtooth; a leak draws a sawtooth whose low points climb.
• Threads shows a timeline of thread states. Remember that a thread in a native socket read shows as RUNNABLE even though it uses no CPU, so a page full of green does not mean the CPU is busy.
• Sampler takes periodic thread dumps for CPU and periodic class histograms for memory. Thread dumps are taken at safepoints, so time spent in tight loops between safepoint polls is attributed to the nearest poll. Treat its CPU view as a strong hint, not a measurement.
• Profiler instruments bytecode: it rewrites the methods you select to record entry and exit. Results are exact call counts, but the overhead can be large and distorts the timing of small methods. Always restrict it to your own packages.

Remote JVMs and containers without a security hole

The JMX connector needs two ports: one for the RMI registry and one for the RMI server that returns the actual objects. Set them to the same number so a single tunnel or firewall rule works, and tell RMI which hostname to put into the stubs it hands back, otherwise the client is told to connect to an internal address it cannot reach.

# Start the service with a JMX endpoint bound to localhost only
java \
  -Dcom.sun.management.jmxremote.port=9010 \
  -Dcom.sun.management.jmxremote.rmi.port=9010 \
  -Dcom.sun.management.jmxremote.host=127.0.0.1 \
  -Djava.rmi.server.hostname=127.0.0.1 \
  -Dcom.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.management.jmxremote.password.file=/etc/app/jmxremote.password \
  -Dcom.sun.management.jmxremote.access.file=/etc/app/jmxremote.access \
  -Dcom.sun.management.jmxremote.ssl=false \
  -jar orders-service.jar

# From your laptop: tunnel the port, then add a JMX connection to localhost:9010
ssh -N -L 9010:127.0.0.1:9010 ops@orders-host

# Kubernetes: the same idea with port-forward
kubectl port-forward pod/orders-7c9f5 9010:9010

Binding to localhost and tunnelling keeps authentication and encryption in SSH, which is why ssl=false is tolerable here; never expose an unauthenticated JMX port, because JMX can load classes and invoke operations, which amounts to remote code execution. Inside a container, local attach also works with kubectl exec if you run a JDK tool as the same user, but VisualVM is a desktop GUI, so the tunnel is the usual route. Alternatively, skip the live connection entirely: run jcmd <pid> GC.heap_dump /tmp/app.hprof in the pod, copy the file out and open it locally.

Worked example: a slow memory leak

A pricing service needs a restart every three days because it eventually runs out of heap. Open the Monitor tab and watch for twenty minutes: the heap sawtooths as expected, but the low point after each collection rises slowly. That rising floor is the signature of objects that remain reachable, and it is the moment to stop guessing and take evidence.

1. Press Perform GC, then Heap Dump. Wait an hour of normal traffic and do it again. Two dumps taken after a collection make growth visible and filter out short-lived garbage.
2. Open the second dump, go to Objects, and sort by retained size rather than shallow size. Shallow size is the object itself; retained size is everything that would be freed if it disappeared. One ConcurrentHashMap retaining most of the heap is a far stronger lead than millions of small strings.
3. Use the compare feature against the first dump. Classes whose instance counts grew by roughly the same amount, here Quote, String and the map's node class, move together, which tells you they form one structure.
4. On one leaked Quote, ask for the path to the nearest GC root. It ends at the static field QuoteCache.CACHE, a class held by the application class loader. That is the culprit.

public final class QuoteCache {
    // Intended as a cache, behaves as a leak: nothing ever evicts entries.
    private static final Map<String, Quote> CACHE = new ConcurrentHashMap<>();

    public Quote get(String symbol, Instant at) {
        String key = symbol + "@" + at.truncatedTo(ChronoUnit.SECONDS);  // unbounded key space
        return CACHE.computeIfAbsent(key, k -> loader.load(symbol, at));
    }
}

// Fix: bound it and let entries expire (Caffeine shown; any bounded cache works)
private static final Cache<String, Quote> CACHE = Caffeine.newBuilder()
        .maximumSize(50_000)
        .expireAfterWrite(Duration.ofMinutes(5))
        .build();

The key includes a timestamp truncated to the second, so the key space never stops growing and the map never evicts. The fix bounds size and age. Verify the fix the same way you found the bug: the post-GC floor in the Monitor tab should flatten within an hour. For deciding how large the heap should be once it is no longer leaking, see Java heap tuning.

Worked example: a CPU hotspot after a release

A second ticket: p99 latency of a search endpoint doubled after a release. Open the Sampler, choose CPU, apply load and let it run for a minute. The hot-spot view shows java.util.regex.Pattern.compile near the top with a large self time, called from a validation helper. The release had replaced a precompiled static Pattern with String.matches, which compiles the pattern on every call.

Before you trust that, remember the sampler's bias. A method that compiles regexes allocates heavily and passes through safepoint polls often, so it is a method the sampler sees well, which makes this finding credible. The opposite case, a counted loop with no polls, can hide behind a neighbour. When the sampler and your intuition disagree, confirm with async-profiler or a JFR recording before you rewrite code. Here the fix is one line, restoring a static final Pattern, and the next sample should show the method gone. Allocation pressure from the same bug also shows in the GC log; reading GC logs explains how to confirm the drop in young collections.

Heap dump practice

Heap dump analysis is where VisualVM earns its keep, so a few practices make it far more productive.

• Capture dumps automatically on failure with -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath=/dumps. Give that path enough disk: a dump is roughly the size of the live heap.
• Prefer jcmd <pid> GC.heap_dump on servers. Dumping pauses the application for a time proportional to heap size, so take it from a node drained of traffic when you can.
• Use OQL, the object query language, to ask precise questions, for example select s from java.lang.String s where s.value.length > 100000 to find huge strings, or a query over a collection class filtered by its size field.
• Treat dumps as sensitive data. They contain every string in memory: tokens, passwords, customer records. Store them encrypted, restrict access and delete them when the investigation closes.
• Read retained sizes in context of memory regions. Direct buffers, thread stacks and metaspace are outside the heap and will not appear; the JVM memory layout covers where else memory goes.

Failure modes

• The JVM is not listed. Different OS user, a container namespace, PerfDisableSharedMem, or a temp directory that is not shared. Run VisualVM as the same user or use JMX.
• JMX connects and then hangs. The RMI server port differs from the registry port, or java.rmi.server.hostname points at an unreachable address. Set both ports equal and the hostname to the tunnel end.
• The profiler freezes the application. You instrumented every class, including the JDK. Restrict instrumentation to your packages, or use the sampler.
• The heap dump never finishes. The dump file was written on a remote disk that filled up, or the GUI ran out of its own heap opening a large dump. Raise VisualVM's heap in its configuration file and analyse dumps on a machine with spare memory.
• Numbers disagree with the container dashboard. VisualVM reports Java heap; the container reports resident memory, which includes metaspace, code cache, thread stacks and native allocations.
• The finding was a sampling artefact. Safepoint bias moved time to the wrong method. Confirm with a second tool before acting.

Trade-offs

VisualVM's strength is that it needs nothing in advance: no agent baked into the image, no license, no recording plan. That same live-connection model is its weakness in production, where opening a port, attaching an agent or pausing for a dump are changes that need approval. A reasonable division of labour is: JFR always on in production, async-profiler for targeted CPU and allocation questions, and VisualVM on your laptop for exploration, heap dump forensics and teaching yourself what a healthy JVM looks like. Reach for a commercial profiler when you need allocation call trees with low overhead, database call attribution across services, or team-wide snapshot sharing.

What to do next

1. Install VisualVM 2.2.2 from visualvm.github.io and run it on a JDK from the supported range, JDK 8 to 26.
2. Attach it to a local copy of your service, apply load and save screenshots of a healthy Monitor tab as your baseline.
3. Add -XX:+HeapDumpOnOutOfMemoryError and a dump path with enough disk to every production JVM.
4. Create a localhost-bound, password-protected JMX configuration and a documented SSH or port-forward procedure, and test it before an incident.
5. Take two heap dumps an hour apart in staging, compare them, and practise the retained size and path-to-GC-root workflow.
6. Run the CPU sampler once, then confirm its top three methods with async-profiler or JFR so you learn how far to trust it.
7. Write a short policy for handling dump files: where they go, who may read them and when they are deleted.