SAPIEN is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. It enables various robotic vision and interaction tasks that require detailed part-level understanding. SAPIEN is a collaborative effort between researchers at UCSD, Stanford and SFU. The dataset is a continuation of ShapeNet and PartNet.
SAPIEN is distributed via PyPI. Installation is just
pip install sapien
It requires Linux with NVIDIA, AMD, or Intel GPU to run. Verify installation with
python -m sapien.exapmle.hello_world
Next, follow our tutorial at: https://sapien-sim.github.io/docs/.
To use SAPIEN on a GPU server without display, the only system dependencies required are libegl1 and libxext6. If using NVIDIA docker environment, enable graphics, utility, and compute by setting the environment variable in the Dockerfile.
ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute
To use SAPIEN on a GPU server with virtual display, additionally install xvfb, x11vnc, and any window manager such as fluxbox or xfce. Add display capabilities for NVIDIA docker.
ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute,display
Assuming fluxbox, start a VNC server by
x11vnc -create -env FD_PROG=/usr/bin/fluxbox -env X11VNC_FINDDISPLAY_ALWAYS_FAILS=1 -env X11VNC_CREATE_GEOM=${99:-1920x1080x16} -gone 'pkill Xvfb' -nopw
# Note: you should use a strong password and/or only allow local access
Now you can connect to the server at port 5900. SAPIEN should be fully functional, test with
python -m sapien.exapmle.hello_world
color and material parameters for visual shapes are unified to material.VulkanRenderer to SapienRenderer (VulkanRenderer is still an alias)SapienRendererKuafuRenderer, use the rt shader in SapienRenderer insteadget_global_posecamera.set_parent and camera.set_local_pose.scene.add_camera and scene.remove_cameraadd_mounted_camera can be replaced with add_camera followed by camera.set_parent and camera.set_local_pose. add_mounted_camera is still provided but fovx should not longer be provided.find_camera_by_mount.camera.near, camera.far, camera.set_fovx, camera.set_fovy, camera.set_focal_lengths, camera.set_principal_point, camera.skew, and the all-in-one method camera.set_perspective_parameters.actor.get_visual_bodies() and visual_body.get_render_shapes(), users typically do shape.scale and shape.pose. These are no longer valid. It is required to check visual_body.type. When type is mesh, shape.scale is replaced with visual_body.scale and shape.pose is replaced by visual_body.local_pose. These changes are made to match add_visual_shape functions when building the actor.scene.renderer_scene.add_xxx_light with scene.add_xxx_lightscene.remove_mounted_camera with scene.remove_camerafovx from scene.add_mounted_camera.sapien.core.renderer for detailsactor.get_collision_shapesCollisionShape.set_collision_groupsContact.renderer.create_material()add_xxx_shape is replaced with add_xxx_collision.scene.renderer_sceneSAPIEN Website: https://sapien.ucsd.edu/. SAPIEN Documentation: https://sapien-sim.github.io/docs/.
Make sure all submodules are initialized git submodule update --init --recursive.
To build SAPIEN, simply run ./scripts/docker_build_wheels.sh. It is not recommended to build outside of our provided docker.
For reference, the Dockerfile is provided here. Note that PhysX needs to be compiled with clang-9 into static libraries before building the Docker image.
It can be tricky to setup all dependencies outside of a Docker environment. You need to install all dependencies according to the Docker environment. If all dependencies set up correctly, run python setup.py bdist_wheel to build the wheel.
If you use SAPIEN and its assets, please cite the following works:
@InProceedings{Xiang_2020_SAPIEN,
author = {Xiang, Fanbo and Qin, Yuzhe and Mo, Kaichun and Xia, Yikuan and Zhu, Hao and Liu, Fangchen and Liu, Minghua and Jiang, Hanxiao and Yuan, Yifu and Wang, He and Yi, Li and Chang, Angel X. and Guibas, Leonidas J. and Su, Hao},
title = {{SAPIEN}: A SimulAted Part-based Interactive ENvironment},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}}
@InProceedings{Mo_2019_CVPR,
author = {Mo, Kaichun and Zhu, Shilin and Chang, Angel X. and Yi, Li and Tripathi, Subarna and Guibas, Leonidas J. and Su, Hao},
title = {{PartNet}: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level {3D} Object Understanding},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}
@article{chang2015shapenet,
title={Shapenet: An information-rich 3d model repository},
author={Chang, Angel X and Funkhouser, Thomas and Guibas, Leonidas and Hanrahan, Pat and Huang, Qixing and Li, Zimo and Savarese, Silvio and Savva, Manolis and Song, Shuran and Su, Hao and others},
journal={arXiv preprint arXiv:1512.03012},
year={2015}
}
If you use SAPIEN Realistic Depth generated by SAPIEN's simulated depth sensor, please cite the following work:
@ARTICLE{10027470,
author={Zhang, Xiaoshuai and Chen, Rui and Li, Ang and Xiang, Fanbo and Qin, Yuzhe and Gu, Jiayuan and Ling, Zhan and Liu, Minghua and Zeng, Peiyu and Han, Songfang and Huang, Zhiao and Mu, Tongzhou and Xu, Jing and Su, Hao},
journal={IEEE Transactions on Robotics},
title={Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation},
year={2023},
volume={},
number={},
pages={1-19},
doi={10.1109/TRO.2023.3235591}}