📸 Visualization & Rendering#

Genesis’s visualization system is managed by the visualizer of the scene you just created (i.e. scene.visualizer). There are two ways for visualizing the scene: 1). using the interactive viewer that runs in a separate thread, and 2). by manually adding cameras to the scene and render images using the camera.

Viewer#

If you are connected to a display, you can visualize the scene using the interactive viewer. Genesis uses different options groups to configure different components in the scene. To configure the viewer, you can change the parameters in viewer_options when creating the scene. In addition, we use vis_options to specify visualization-related properties, which will be shared by the viewer and cameras (that we will add very soon).

Create a scene with a more detailed viewer and vis setting (this looks a bit complex, but it’s just for illustration purposes):

scene = gs.Scene(
    show_viewer    = True,
    viewer_options = gs.options.ViewerOptions(
        res           = (1280, 960),
        camera_pos    = (3.5, 0.0, 2.5),
        camera_lookat = (0.0, 0.0, 0.5),
        camera_fov    = 40,
        max_FPS       = 60,
    ),
    vis_options = gs.options.VisOptions(
        show_world_frame = True, # visualize the coordinate frame of `world` at its origin
        world_frame_size = 1.0, # length of the world frame in meter
        show_link_frame  = False, # do not visualize coordinate frames of entity links
        show_cameras     = False, # do not visualize mesh and frustum of the cameras added
        plane_reflection = True, # turn on plane reflection
        ambient_light    = (0.1, 0.1, 0.1), # ambient light setting
    ),
    renderer = gs.renderers.Rasterizer(), # using rasterizer for camera rendering
)

Here we can specify the pose and fov of the viewer camera. The viewer will run as fast as possible if max_FPS is set to None. If res is set to None, genesis will automatically create a 4:3 window with the height set to half of your display height. Also note that in the above setting, we set to use rasterization backend for camera rendering. Genesis provides two rendering backends: gs.renderers.Rasterizer() and gs.renderers.RayTracer(). The viewer always uses the rasterizer. By default, camera also uses rasterizer.

Once the scene is created, you can access the viewer object via scene.visualizer.viewer, or simply scene.viewer as a shortcut. You can query or set the viewer camera pose:

cam_pose = scene.viewer.camera_pose

scene.viewer.set_camera_pose(cam_pose)

Camera & Headless Rendering#

Now let’s manually add a camera object to the scene. Cameras are not connected to the viewer or the display, and returns rendered images only when you need it. Therefore, camera works in headless mode.

cam = scene.add_camera(
    res    = (1280, 960),
    pos    = (3.5, 0.0, 2.5),
    lookat = (0, 0, 0.5),
    fov    = 30,
    GUI    = False
)

If GUI=True, each camera will create an opencv window to dynamically display the rendered image. Note that this is different from the viewer GUI.

Then, once we build the scene, we can render images using the camera. Our camera supports rendering rgb image, depth, segmentation mask and surface normals. By default, only rgb is rendered, and you can turn other modes on by setting the parameters when calling camera.render():

scene.build()

# render rgb, depth, segmentation mask and normal map
rgb, depth, segmentation, normal = cam.render(depth=True, segmentation=True, normal=True)

If you used GUI=True and have a display connected, you should be able to see 4 windows now. (Sometimes opencv windows comes with extra delay, so you can call extra cv2.waitKey(1) if the windows are black, or simply call render() again to refresh the window.)

../../_images/multimodal.png

Record videos using camera

Now, let’s only render rgb images, and move the camera around and record a video. Genesis provides a handy util for recording videos:

# start camera recording. Once this is started, all the rgb images rendered will be recorded internally
cam.start_recording()

import numpy as np
for i in range(120):
    scene.step()

    # change camera position
    cam.set_pose(
        pos    = (3.0 * np.sin(i / 60), 3.0 * np.cos(i / 60), 2.5),
        lookat = (0, 0, 0.5),
    )
    
    cam.render()

# stop recording and save video. If `filename` is not specified, a name will be auto-generated using the caller file name.
cam.stop_recording(save_to_filename='video.mp4', fps=60)

You will have the video saved to video.mp4:

Here is the full code script covering everything discussed above:

import genesis as gs

gs.init(backend=gs.cpu)

scene = gs.Scene(
    show_viewer = True,
    viewer_options = gs.options.ViewerOptions(
        res           = (1280, 960),
        camera_pos    = (3.5, 0.0, 2.5),
        camera_lookat = (0.0, 0.0, 0.5),
        camera_fov    = 40,
        max_FPS       = 60,
    ),
    vis_options = gs.options.VisOptions(
        show_world_frame = True,
        world_frame_size = 1.0,
        show_link_frame  = False,
        show_cameras     = False,
        plane_reflection = True,
        ambient_light    = (0.1, 0.1, 0.1),
    ),
    renderer=gs.renderers.Rasterizer(),
)

plane = scene.add_entity(
    gs.morphs.Plane(),
)
franka = scene.add_entity(
    gs.morphs.MJCF(file='xml/franka_emika_panda/panda.xml'),
)

cam = scene.add_camera(
    res    = (640, 480),
    pos    = (3.5, 0.0, 2.5),
    lookat = (0, 0, 0.5),
    fov    = 30,
    GUI    = False,
)

scene.build()

# render rgb, depth, segmentation, and normal
# rgb, depth, segmentation, normal = cam.render(rgb=True, depth=True, segmentation=True, normal=True)

cam.start_recording()
import numpy as np

for i in range(120):
    scene.step()
    cam.set_pose(
        pos    = (3.0 * np.sin(i / 60), 3.0 * np.cos(i / 60), 2.5),
        lookat = (0, 0, 0.5),
    )
    cam.render()
cam.stop_recording(save_to_filename='video.mp4', fps=60)

Photo-realistic Ray Tracing Rendering#

Genesis provides a ray tracing rendering backend for photorealistic rendering. You can easily switch to using this backend by setting renderer=gs.renderers.RayTracer() when creating the scene. This camera allows more parameter adjustment, such as spp, aperture, model, etc.

Setup#

Tested on

  • Ubuntu 22.04, CUDA 12.4, python 3.9

Get submodules, specifically genesis/ext/LuisaRender.

# inside Genesis/
git submodule update --init --recursive
pip install -e ".[render]"

Install/upgrad g++ and gcc (to) version 11.

sudo apt install build-essential manpages-dev software-properties-common
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt update && sudo apt install gcc-11 g++-11
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 110
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 110

# verify version
g++ --version
gcc --version

Install cmake. We use snap instead of apt because we need its version >= 3.26. However, remember to use the correct cmake. You may have /usr/local/bin/cmake but the snap installed package is at /snap/bin/cmake (or /usr/bin/snap). Please double check the order of binary path via echo $PATH.

sudo snap install cmake --classic
cmake --version

Install dependencies,

sudo apt install libvulkan-dev # Vulkan
sudo apt-get install zlib1g-dev # zlib
sudo apt-get install libx11-dev # X11
sudo apt-get install xorg-dev libglu1-mesa-dev # RandR headers

Build LuisaRender. Remember to use the correct cmake.

cd genesis/ext/LuisaRender
cmake -S . -B build -D CMAKE_BUILD_TYPE=Release -D PYTHON_VERSIONS=3.9 -D LUISA_COMPUTE_DOWNLOAD_NVCOMP=ON # remember to check python version
cmake --build build -j $(nproc)

If you really struggle getting the build, we have some build here and you can check if your machine happens to have the same setup. The naming follows build_<commit-tag>_cuda<version>_python<version>. Download the one that matches your system, rename to build/ and put it in genesis/ext/LuisaRender.

Finally, you can run the example,

cd examples/rendering
python demo.py

You should be able to get

../../_images/raytracing_demo.png

FAQ#

  • Pybind error when doing cmake -S . -B build -D CMAKE_BUILD_TYPE=Release -D PYTHON_VERSIONS=3.9 -D LUISA_COMPUTE_DOWNLOAD_NVCOMP=ON,

    CMake Error at src/apps/CMakeLists.txt:12 (find_package):
By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "pybind11",
but CMake did not find one.

Could not find a package configuration file provided by "pybind11" with any
of the following names:

    pybind11Config.cmake
    pybind11-config.cmake

    You probably forget to do pip install -e ".[render]". Alternatively, you can simply do pip install "pybind11[global]".

  • CUDA runtime compilation error when doing cmake -S . -B build -D CMAKE_BUILD_TYPE=Release -D PYTHON_VERSIONS=3.9 -D LUISA_COMPUTE_DOWNLOAD_NVCOMP=ON,

    /usr/bin/ld: CMakeFiles/luisa-cuda-nvrtc-standalone-compiler.dir/cuda_nvrtc_compiler.cpp.o: in function `main':
cuda_nvrtc_compiler.cpp:(.text.startup+0x173): undefined reference to `nvrtcGetOptiXIRSize'
/usr/bin/ld: cuda_nvrtc_compiler.cpp:(.text.startup+0x197): undefined reference to `nvrtcGetOptiXIR'

    You need to install “system-wise” cuda-toolkit (official installation guide). You first check the cuda-toolkit,

    nvcc --version # this should be the consistent with you cuda version from nvidia-smi
which nvcc # just to check you are using the cuda-toolkit you expected

    If you don’t get proper output from nvcc, please follow the official cuda-toolkit installation guide. Yet, just as an example of installing cuda-toolkit for cuda-12.4. Download installer as in here.

    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda-repo-ubuntu2204-12-4-local_12.4.0-550.54.14-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-12-4-local_12.4.0-550.54.14-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-12-4-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-4

    Remember to set binary and runtime library path. In ~/.bashrc, add the following (note that we append the CUDA path at the end since there are also another gcc and g++ in /usr/local/cuda-12.4/bin and may not be version 11, which is required for the build),

    PATH=${PATH:+${PATH}:}/usr/local/cuda-12.4/bin
LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}/usr/local/cuda-12.4/lib64

    Remember to either restart the terminal or do source ~/.bashrc. Another type of error is,

    <your-env-path>/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_fatal_printf@GLIBC_PRIVATE'
<your-env-path>/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_audit_symbind_alt@GLIBC_PRIVATE'
<your-env-path>/genesis-test1/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_exception_create@GLIBC_PRIVATE'
<your-env-path>/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `__nptl_change_stack_perm@GLIBC_PRIVATE'
<your-env-path>/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `__tunable_get_val@GLIBC_PRIVATE'
<your-env-path>/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_audit_preinit@GLIBC_PRIVATE'
<your-env-path>/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_find_dso_for_object@GLIBC_PRIVATE'

    This may be due to the cuda-toolkit in your conda environment. Please do the following and install the system-wise CUDA,

    which nvcc
conda uninstall cuda-toolkit

    Alternatively, you can add your conda library path to the runtime library path,

    ls $CONDA_PREFIX/lib/libcudart.so # you should have this

# inside you ~/.bashrc, add
LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}/usr/local/cuda-12.4/lib64

    Lastly, remember to clear the build after doing the above fixed,

    rm -r build

  • Compiler error at cmake -S . -B build -D CMAKE_BUILD_TYPE=Release -D PYTHON_VERSIONS=3.9 -D LUISA_COMPUTE_DOWNLOAD_NVCOMP=ON,

    CMake Error at /snap/cmake/1435/share/cmake-3.31/Modules/CMakeDetermineCCompiler.cmake:49 (message):
Could not find compiler set in environment variable CC:

/home/tsunw/miniconda3/envs/genesis-test1/bin/x86_64-conda-linux-gnu-cc.
Call Stack (most recent call first):
CMakeLists.txt:21 (project)


CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage

    You are probably not using gcc and g++ version 11. Please double check (i) the version (ii) if the binary points to the path as expected (iii) the order of your binary path,

    gcc --version
g++ --version
which gcc
which g++
echo $PATH # e.g., /usr/local/cuda-12.4/bin/gcc (version = 10.5) shouldn't be in front of /usr/bin/gcc (version = 11 if you install properly with apt)

  • Import error when running examples/rendering/demo.py,

    [Genesis] [11:29:47] [ERROR] Failed to import LuisaRenderer. ImportError: /home/tsunw/miniconda3/envs/genesis-test1/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/tsunw/workspace/Genesis/genesis/ext/LuisaRender/build/bin/liblc-core.so)

    Conda’s libstdc++.so.6 doesn’t support 3.4.30. You need to move system’s into conda (reference).

    cd $CONDA_PREFIX/lib
mv libstdc++.so.6 libstdc++.so.6.old
ln -s /usr/lib/x86_64-linux-gnu/libstdc++.so.6 libstdc++.so.6