Profiling simulation performance#

This page covers how to measure where a Genesis World simulation spends its time. There are three questions worth asking, in order of increasing depth:

Throughput: how many steps per second does the whole scene run? This is the headline number, reported as FPS.
Launch latency: is the GPU actually busy, or is the CPU stalling between kernel launches? This is what limits large parallel simulations that are not yet GPU-bound.
Per-kernel time: which solver kernels dominate a step? This tells you what to optimize.

The first is built in and always available. The other two use the PyTorch profiler, which works even in a simulation that never touches PyTorch.

Benchmark against a disposable cache#

Before you measure anything, get the compilation cache out of the way. Genesis World compiles GPU kernels just-in-time and caches the results to a persistent local folder, so repeated runs of the same scene start quickly. Quadrants, the compiler, keeps its own cache in the same way. This is what you want for day-to-day work, but it distorts profiling and benchmarking: the first run pays the compilation cost and later runs read from the warm cache.

Do not wipe the persistent cache to get around this. Its effect outlives your experiment, and every future simulation is slow until the cache rebuilds. Instead, redirect both caches to a throwaway directory for the duration of a single run, by setting a few environment variables:

XDG_CACHE_HOME="$(mktemp -d)" \
GS_CACHE_FILE_PATH="$XDG_CACHE_HOME/genesis" \
QD_OFFLINE_CACHE_FILE_PATH="$XDG_CACHE_HOME/quadrants" \
python your_script.py

GS_CACHE_FILE_PATH: Genesis World’s cache directory.
QD_OFFLINE_CACHE_FILE_PATH: the Quadrants compiler cache directory.
XDG_CACHE_HOME: the base cache directory, honored on Linux only.

On Linux, XDG_CACHE_HOME alone is enough to relocate the Genesis World cache. On Windows and macOS it is ignored, so set GS_CACHE_FILE_PATH and QD_OFFLINE_CACHE_FILE_PATH explicitly as shown above.

Reading the FPS counter#

By default, Genesis World prints the achieved step rate to the terminal as the simulation runs. The output looks like this:

Running at 43,000,000.00 FPS (1,433.33 FPS per env, 30000 envs).

Three numbers, all reported per window of wall-clock time:

Total FPS: steps per second summed across every environment. This is the throughput figure to compare against benchmarks.
Per-env FPS: the total divided by the number of environments. Useful when comparing scenes with different batch sizes.
Environments: the value of n_envs passed to scene.build(). It is omitted when the scene has no batch dimension.

The rate is measured over fixed wall-clock windows and lightly smoothed with an exponential moving average, so it settles to a stable value rather than jumping every step.

Configuring the counter#

The counter is controlled by gs.options.ProfilingOptions, passed to the scene:

scene = gs.Scene(
    profiling_options=gs.options.ProfilingOptions(
        show_FPS=True,       # print the step rate each window; default True
        FPS_tracker_alpha=0.95,  # EMA momentum for the smoothed rate
    ),
)

show_FPS: whether to print the rate at all. Set it to False for quiet runs, or when the log lines interfere with your own output.
FPS_tracker_alpha: the smoothing momentum, between 0 and 1. Higher values react more slowly and read more steadily; lower values track sudden changes in speed more closely.

The scene exposes the live options as scene.profiling_options, so you can toggle the counter after construction:

scene.profiling_options.show_FPS = False

See Options system for how ProfilingOptions fits alongside the other options objects.

Measuring throughput#

The scripts in examples/speed_benchmark are the reference for measuring throughput on your own hardware. They are the source of truth for a clean benchmark setup; the excerpts below only highlight the choices that matter.

examples/speed_benchmark/franka.py runs a Franka arm across tens of thousands of environments:

gs.init(backend=gs.gpu, performance_mode=True)

scene = gs.Scene(
    rigid_options=gs.options.RigidOptions(dt=0.01),
    show_viewer=False,
)
# ... add plane and franka ...

scene.build(n_envs=30000, env_spacing=(1.0, 1.0))

Three choices make this a throughput benchmark rather than an interactive session:

performance_mode=True bakes static tensor shapes into the compiled kernels for faster stepping, at the cost of recompiling whenever the scene changes. It is worth it for a fixed benchmark or a training run, not for iterative development.
show_viewer=False runs headless. Rendering a window caps throughput at display rates and defeats the purpose.
A large n_envs keeps the GPU saturated. Throughput scales with the batch until you run out of VRAM.

examples/speed_benchmark/anymal_c.py is the equivalent for a legged robot. For the reasoning behind these settings and other ways to raise throughput, see Parallel simulation and Writing an efficient RL environment.

Timing GPU kernels with the PyTorch profiler#

The FPS counter tells you how fast, not why. To see individual GPU kernels and the gaps between them, attach the PyTorch profiler around the steps you care about. It records CUDA activity whether or not your simulation uses PyTorch tensors.

examples/speed_benchmark/timers.py wraps a fixed number of steps in a profiler:

with torch.profiler.profile(
    on_trace_ready=torch.profiler.tensorboard_trace_handler("./benchmark"),
    schedule=torch.profiler.schedule(wait=0, warmup=0, active=1),
    record_shapes=False,
    profile_memory=False,
    with_stack=True,
    with_flops=False,
):
    for step in range(500):
        scene.step()

The schedule selects which steps to record so you can skip the ones you do not care about. Its phases are:

wait: steps to ignore at the start, past any one-time initialization.
warmup: steps to run while the profiler primes itself but discards the trace.
active: steps actually recorded. One is usually enough and keeps memory low.

When you use a multi-phase schedule, advance it by calling profiler.step() once per iteration so the profiler knows which phase it is in. See the PyTorch profiler schedule documentation for details. The resulting trace opens in TensorBoard or, if you export a Chrome trace, in Perfetto.

Reading the trace#

On the GPU timeline you get the precise duration of every kernel launch. White gaps between kernels are launch overhead: the GPU sitting idle while the CPU catches up. On a large parallel simulation those gaps should be negligible; if they dominate, the simulation is CPU-bound and no amount of extra environments will help until the stalls are removed. Diagnosing and removing these stalls is the subject of Writing an efficient RL environment.

The GPU timeline has no call hierarchy of its own, so it can be hard to tell which Python-side operation a kernel belongs to. Insert a synchronization before each step to force that alignment:

qd.sync()  # block until all Quadrants GPU work completes, so the CPU and GPU timelines line up
scene.step()

Warning

Synchronizing every step serializes the CPU and GPU and can roughly halve throughput. Use it to understand a trace, then remove it before measuring speed.

Per-solver kernel timings#

To go below CUDA kernels and time blocks of work inside the rigid solver, examples/speed_benchmark/timers.py reads the constraint solver’s timers array and plots per-environment timings in the terminal:

from genesis.utils.misc import qd_to_torch

timers = qd_to_torch(scene.rigid_solver.constraint_solver.constraint_state.timers)

Run it with --profiling to attach the PyTorch profiler instead, or without to collect and plot the per-solver timings. This reaches into solver internals and is aimed at contributors optimizing the physics kernels rather than at typical simulation code.