Raycaster sensors#

A raycaster measures the scene geometrically: it casts a fixed pattern of rays from the sensor’s frame, finds where each ray first hits scene geometry, and returns the hit points and their distances. Two concrete sensors share this machinery and differ only in how you interpret the result:

gs.sensors.Lidar returns the raw hit set (a point cloud and per-ray distances) for range sensing, mapping, and obstacle avoidance.
gs.sensors.DepthCamera casts a pinhole-camera ray grid and reshapes the distances into a depth image.

The ray count and directions come from a pattern (gs.sensors.RaycastPattern). Two runnable examples are the source of truth for this page: a teleoperated sensor mounted on a Go2 in examples/sensors/lidar_teleop.py, and depth cameras on rigid and deforming geometry in examples/sensors/depth_camera_custom_vverts.py.

Minimal example#

Mount a spinning lidar on a robot, add something for the rays to hit, and read the point cloud each step:

import genesis as gs

gs.init(backend=gs.gpu)

scene = gs.Scene(show_viewer=True)
scene.add_entity(gs.morphs.Plane())
robot = scene.add_entity(gs.morphs.URDF(file="urdf/go2/urdf/go2.urdf", fixed=True))
scene.add_entity(gs.morphs.Box(size=(0.5, 0.5, 1.0), pos=(2.0, 0.0, 0.5), fixed=True))

lidar = scene.add_sensor(
    gs.sensors.Lidar(
        pattern=gs.sensors.SphericalPattern(),  # 360° × 60° fov, 128 × 64 rays
        entity_idx=robot.idx,                   # mount on the robot's base link
        pos_offset=(0.3, 0.0, 0.1),             # meters, in the link frame
        max_range=20.0,                         # meters
        return_world_frame=True,
        draw_debug=True,                        # draw hit points in the viewer
    )
)

scene.build()
for _ in range(1000):
    scene.step()
    result = lidar.read()          # RaycasterReturnType(points, distances)
    points = result.points         # shape (128, 64, 3), meters, Z-up world frame
    distances = result.distances   # shape (128, 64), meters

The lidar needs geometry to cast against: a raycaster raises at build time if the rigid solver is inactive and no entity opts into visual raycasting.

How ray casting works#

A pattern is a purely local description of the rays: it fixes a start point and a unit direction for each ray in the sensor’s own frame, independent of where the sensor ends up in the world. The pattern’s return_shape (for example (n_horizontal, n_vertical) for a spherical scan) sets the layout every read preserves.

At each read(), Genesis World places the pattern in the world by composing the attached link’s pose with the sensor’s pos_offset and euler_offset, then casts every ray against an acceleration structure built over the scene’s geometry. Each ray reports the first surface it hits. A ray that hits nothing within max_range, or closer than min_range, reports no_hit_value (which defaults to max_range) so the returned tensors keep a fixed shape.

Attach a sensor to a link by setting entity_idx and link_idx_local; the rays then move with that link. Leave entity_idx unset (or None) for a world-fixed sensor, in which case pos_offset and euler_offset are applied in the world frame.

Reading data#

read() returns a RaycasterReturnType, a NamedTuple of points and distances:

result = lidar.read()
points = result.points        # shape ([n_envs,] *return_shape, 3), meters
distances = result.distances  # shape ([n_envs,] *return_shape),    meters

distances is the along-ray hit distance in meters, with misses filled by no_hit_value.
points is the hit location per ray. With return_world_frame=True it is in the world frame (Z-up, meters); with the default return_world_frame=False it is in the sensor’s local frame.

The return_shape is set by the pattern, so the trailing axes match the pattern’s own layout: (128, 64) for the default spherical scan, (height, width) for a depth camera, (n_x, n_y) for a grid. The leading [n_envs,] axis is present only when the scene is built with multiple environments (see Multiple environments).

Ray patterns#

The pattern decides what the sensor is. All three are constructed under gs.sensors and passed as the pattern argument.

Pattern	Ray layout	Typical hardware
`SphericalPattern`	azimuth × elevation scan lines	3D lidar (Velodyne, Ouster)
`DepthCameraPattern`	pinhole image grid	depth cameras (RealSense, Kinect)
`GridPattern`	parallel rays on a plane	height maps, planar sensing

SphericalPattern#

Rays fan out over a horizontal (azimuth) and vertical (elevation) field of view. Specify the pattern by ray count, by angular resolution, or by explicit angle arrays:

gs.sensors.SphericalPattern(
    fov=(360.0, 60.0),   # (horizontal, vertical) degrees; a scalar is centered on 0°,
                         # a (min, max) tuple gives an asymmetric range
    n_points=(128, 64),  # (horizontal, vertical) ray counts -> return_shape (128, 64)
    # angular_resolution=(0.25, 0.5),  # degrees per ray; overrides n_points
    # angles=(h_angles, v_angles),     # explicit angle arrays; overrides the rest
)

To model a real unit, set the fov and ray counts from its datasheet. For example, a Velodyne VLP-16 is fov=(360.0, 30.0), n_points=(1800, 16).

DepthCameraPattern#

A pinhole camera whose optical axis is the sensor’s local +X axis. Configure it by field of view or by explicit intrinsics:

gs.sensors.DepthCameraPattern(
    res=(128, 96),        # (width, height) in pixels -> return_shape (96, 128)
    fov_horizontal=90.0,  # degrees; fov_vertical is derived from the aspect ratio
    # fov_vertical=None,  # set instead to derive fov_horizontal, or set both
    # fx=None, fy=None,   # focal lengths in pixels, override the fov
    # cx=None, cy=None,   # principal point in pixels, defaults to the image center
)

GridPattern#

Parallel rays cast from a plane in a single direction, a height map under the sensor, for instance:

gs.sensors.GridPattern(
    resolution=0.1,              # grid spacing, meters (>= 1 mm)
    size=(2.0, 2.0),             # (length, width) of the grid, meters
    direction=(0.0, 0.0, -1.0),  # ray direction in the sensor frame (here: straight down)
)

Depth cameras#

gs.sensors.DepthCamera is a raycaster with a DepthCameraPattern. It exposes everything a lidar does, plus read_image(), which reshapes the per-ray distances into a depth image:

depth_cam = scene.add_sensor(
    gs.sensors.DepthCamera(
        pattern=gs.sensors.DepthCameraPattern(
            res=(96, 72),         # (width, height) in pixels
            fov_horizontal=90.0,  # degrees
        ),
        entity_idx=go2.idx,
        link_idx_local=0,
        pos_offset=(0.3, 0.0, 0.1),  # meters, in the link frame
        max_range=5.0,               # meters
        return_world_frame=True,
    ),
)

scene.build()
scene.step()
depth = depth_cam.read_image()  # shape ([n_envs,] 72, 96), meters

read_image() returns the distances field alone, reshaped to ([n_envs,] height, width); misses carry no_hit_value. The point cloud is still available through read().points. Because a depth camera shares the raycasting backend with any lidar in the scene, the two cast against the same geometry.

Running python examples/sensors/lidar_teleop.py --pattern depth teleoperates a depth camera on the Go2 (--pattern also accepts spherical and grid):

Common options#

Lidar and DepthCamera share the raycaster options below (the mounting fields are common to all attached sensors):

gs.sensors.Lidar(
    pattern=pattern,
    entity_idx=robot.idx,        # attach to this entity; omit for a world-fixed sensor
    link_idx_local=0,            # which link of the entity to attach to
    pos_offset=(0.0, 0.0, 0.15), # meters, in the link frame
    euler_offset=(0.0, 0.0, 0.0),# extrinsic x-y-z, degrees
    min_range=0.0,               # meters; hits closer than this are dropped
    max_range=20.0,              # meters
    no_hit_value=None,           # value for a miss; defaults to max_range
    return_world_frame=False,    # world frame if True, else the sensor's local frame
    draw_debug=False,            # draw ray starts and hit points in the viewer
)

Multiple environments#

Build with n_envs > 0 and every read gains a leading batch axis; the pattern layout is unchanged:

scene.build(n_envs=4)
scene.step()
result = lidar.read()
result.points.shape     # (4, 128, 64, 3)
result.distances.shape  # (4, 128, 64)