I’m developing a straightforward method to collect first-person (ego) and third-person (exo) data for robotic training. I’ve been using the HO-cap dataset to establish a baseline, and here are some updates I’ve made:

* added in MANO parameters * added TSDF fused mesh from realsense camera to better visualize dataset and potentially set the stage for a gsplat integration * Built a consistent interface between different ego+exo datasets (in this case assembly101 and hocap)

From here the plan is to include the egocentric perspective now that the exocentric part is done!

The full MVP masterplan laid out

1. Build a strong opensource baseline on a permissively licensed dataset - [x] Exocentric data <- we are here - [ ] Egocentric data <- up next - [ ] Pose estimation algorithm - [ ] Mano fitting algorithm 2. Build an opensource data collection pipeline using accessible and cheap hardware - [ ] Make data collection app using quest3 passthrough API for egocentric data - [ ] Use 1+ iPhones with Final Cut Pro and Final Cut Camera for exocentric data - [ ] code to synchronize the two and generate data 3. Integrate collected data to SO-100 arms and perform imitation learning with it - [ ] Use collected dataset w/egomimic algorithm to validate collection apparatus with a simulated robot - [ ] deploy on SO-100 robot

You can take a look at the code here - https://github.com/rerun-io/pi0-lerobot?tab=readme-ov-file#h...