Ray Casting in a Voxel Block Grid
Note
This is NOT ray casting for triangle meshes. Please refer to /python_api/open3d.t.geometry.RayCastingScene.rst for that use case.
Ray casting can be performed in a voxel block grid to generate depth and color images at specific view points without extracting the entire surface. It is useful for frame-to-model tracking, and for differentiable volume rendering.
We provide optimized conventional rendering, and basic support for customized rendering that may be used in differentiable rendering. An example can be found at examples/python/t_reconstruction_system/ray_casting.py
.
Conventional rendering
From a reconstructed voxel block grid from TSDF Integration, we can efficiently render the scene given the input depth as a rough range estimate.
76# examples/python/t_reconstruction_system/ray_casting.py
77 depth_file_names = load_depth_file_names(config)
78 intrinsic = load_intrinsic(config)
79 extrinsics = load_extrinsics(config.path_trajectory, config)
80 device = o3d.core.Device(config.device)
81
82 for i, extrinsic in tqdm(enumerate(extrinsics)):
83 depth = o3d.t.io.read_image(depth_file_names[i]).to(device)
84 frustum_block_coords = vbg.compute_unique_block_coordinates(
85 depth, intrinsic, extrinsic, config.depth_scale, config.depth_max)
86
87 result = vbg.ray_cast(block_coords=frustum_block_coords,
88 intrinsic=intrinsic,
89 extrinsic=extrinsic,
90 width=depth.columns,
91 height=depth.rows,
92 render_attributes=[
The results could be directly obtained and visualized by
90# examples/python/t_reconstruction_system/ray_casting.py
91 height=depth.rows,
92 'depth', 'normal', 'color', 'index',
93 'interp_ratio'
94 ],
95 colorized_depth = o3d.t.geometry.Image(result['depth']).colorize_depth(
96 config.depth_scale, config.depth_min, config.depth_max)
97
98 # Render color via indexing
99 vbg_color = vbg.attribute('color').reshape((-1, 3))
100 nb_indices = result['index'].reshape((-1))
101 nb_interp_ratio = result['interp_ratio'].reshape((-1, 1))
102 nb_colors = vbg_color[nb_indices] * nb_interp_ratio
Customized rendering
In customized rendering, we manually perform trilinear-interpolation by accessing properties at 8 nearest neighbor voxels with respect to the found surface point per pixel:
97# examples/python/t_reconstruction_system/ray_casting.py
98 depth_max=config.depth_max,
99 weight_threshold=1,
100 range_map_down_factor=8)
101
102 fig, axs = plt.subplots(2, 2)
103
104 (2)) / 255.0
105
Since the output is rendered via indices, the rendering process could be rewritten in differentiable engines like PyTorch seamlessly via /tutorial/core/tensor.ipynb#PyTorch-I/O-with-DLPack-memory-map.