Sensors#

Sensors read information out of a scene without changing its physics. Attach a sensor to a link, step the simulation, and read back a tensor each step. For the attach-and-read model, imperfections, history, and batched reads, see the sensors guide.

Sensor types#

Create a sensor with scene.add_sensor(), passing an options object from gs.sensors. The call returns a handle whose read() gives the measured value and read_ground_truth() the noiseless one, both with the same shape. scene.add_camera() creates a visualization camera instead, which lives outside the sensor pipeline; the Camera sensor page draws the distinction.

Shapes use the batched-optional notation ([n_envs,] ...): the leading n_envs axis is present when the scene is built with multiple environments and absent otherwise.

gs.sensors.*

read() returns

Shape

RasterizerCameraOptions, RaytracerCameraOptions, BatchRendererCameraOptions

CameraReturnType(rgb), rgb uint8

([n_envs,] height, width, 3)

Contact

torch.Tensor (bool)

([n_envs,] 1)

ContactForce

torch.Tensor (float32)

([n_envs,] 3)

IMU

IMUReturnType(lin_acc, ang_vel, mag) (float32)

each ([n_envs,] 3)

Raycaster (alias Lidar), DepthCamera

RaycasterReturnType(points, distances) (float32)

points ([n_envs,] *pattern_shape, 3), distances ([n_envs,] *pattern_shape)

SurfaceDistanceProbe

torch.Tensor (float32) distances

([n_envs,] n_probes)

ContactProbe

torch.Tensor (bool)

([n_envs,] n_probes)

ContactDepthProbe

torch.Tensor (float32) penetration depth

([n_envs,] n_probes)

KinematicTaxel

KinematicTaxelReturnType(force, torque) (float32)

each ([n_envs,] n_probes, 3)

ElastomerTaxel

torch.Tensor (float32) marker displacement

([n_envs,] n_probes, 3)

ProximityTaxel

ProximityTaxelReturnType(force, torque) (float32)

each ([n_envs,] n_probes, 3)

TemperatureGrid

torch.Tensor (float32)

([n_envs,] grid_x, grid_y, grid_z)

JointTorque

torch.Tensor (float32)

([n_envs,] n_dofs)

Notes on the return types:

  • Camera sensors: read() returns CameraReturnType, which carries a single rgb field. Depth, segmentation, and surface normals come from scene.add_camera(...).render(...).

  • SurfaceDistanceProbe: read() returns the probe-to-surface distances; the corresponding nearest points are available as sensor.nearest_points, shape ([n_envs,] n_probes, 3).

  • Raycaster patterns: pattern_shape follows the ray pattern: for example (n_horizontal, n_vertical) for a spherical pattern and (height, width) for DepthCamera.

Sensor reference pages#

See also#