We introduce Omnipush, a dataset with high variety of planar pushing behavior. The dataset contains 250 pushes for each of 250 objects, all recorded with RGB-D and high precision state tracking.
The objects are constructed to explore key factors that affect pushing --the shape of the object and its mass distribution-- which have not been broadly explored in previous datasets and allow to study generalization in model learning.

Paper: Omnipush: accurate, diverse, real-world dataset of pushing dynamics with RGBD images. (PDF).
Maria Bauza, Ferran Alet, Yen-Chen Lin, Tomás Lozano-Pérez, Leslie P. Kaelbling, Phillip Isola, and Alberto Rodriguez.
Contact: Maria Bauza (bauza@mit.edu)

Dataset description

We split the datasets in 7 types

• 0_weight, 1_weight, 2_weight: 250 pushes for 250 obecjts. The name of the dataset indicates the number of extra weights added to the objects it contains. In this case, for each object there are 250 pushes.
• plywood, old, old_plywood: 250 pushes for out-of-distribution objects and surface. These datasets contain pushes that were recorded with out-of-distribution objects (old and old_plywood) and/or on an out-of-distrubtionon surface (plywood and old_plywood). For each object considered, there are 250 pushes.
• bias: 2500 pushes for 5 objects. For 5 objects we collected a larger dataset with 2500 additional pushes for each.

Data formats

• bag: raw data from each push with object and pusher poses and RGB-D images.
• hdf5, json: raw data except for the RGB-D images. This means that they include object poses as well as tip poses.
• json_simple: initial and final poses of the object and pusher.
• color_img: initial and final RGB images of each push.

Datasets used for benchmarking and other useful data

• Model prediction data: normalized data of the full dataset of initial and final object and pusher poses. Used for the model prediction benchmark. The non-normalized version of the data is in the folder processed_data .
• Video prediction dataset: contains the dataset used in the video prediction benchmark.
• CAD_models, top-down_view: CAD model and top-down view of each object in the dataset.
• Example on how to download the folder with all CAD models: on a terminal, wget -r ftp://omnipush.mit.edu/omnipush/CAD_models
• The downloading process might take a while, please contact us if you have any issues.

Content of the model prediction dataset

• 6 files per object: 2 (input,output) * 3 (3,5,7 input dimensions).

Data capturing setup

About the surfaces

The surface used for most experiments is ABS. You can get the same material from McMaster (US vendor): ABS.
In the datasets plywood and old_plywood we considered a surface made of plywood.

About the objects

We have used 250 new objects to record this dataset. In the picture below, you can see some examples of the shapes considered for these objects.

Each object is built by magnetically attaching different sides to a central piece. Additonally, the sides can carry extra weight which allows us to alter the mass distribution.

Object Name Convention

We provide CAD_models and top-down_views of each object in the dataset.

Code

We provide code for the meta-learning baseline on prediction and code for video prediction.

We also provide a rendering script in Python to visualize different shapes: visualization code.
And processing code to exemplify how to process the bag files and visualize both the RGB images and the ground truth poses from state tracking. Below there is an example of the videos that can be generated using the code:

Mass distribution effect on dynamics

This dataset allows to study the effect of the mass ditribution on the motion of objects.

Results on benchmarks

Meta-learning benchmark

Sharing reflections on the data collection

As it is well-known in the robotics community, collecting accurate robotic data is hard and demanding. Below we detail some of our experiences and conclusions after collecting the Omnipush dataset:

1. It takes time: collecting the final version of this dataset took 12 hours a day for 2 weeks, i.e., more than 150 hours. Planning as accurately as possible how much time it will take can prevent you from aiming at something that is unfeasible. Try to be realistic, unexpected things happen. In our case, data collection for each object takes around 15min, so figuring out how to be efficient doing other things while also collecting data became really important.
2. Check everything: To make the system as autonomous as possible we had, for instance, to check continuously that all sensors worked (RGBD camera, tracking system, robot), that the frequency was good and that the memory space was enough. Collect a few times a small amount of data before collecting a large amount of data, even subtle errors can force you to recollect it all.
Similarly, make sure to avoid the possibility of any robot collision or object falling outside the workspace even when sensors are unaccurate.
3. Automate as much as possible: Minimal human intervention is key beyond expected. Human intervention can add unexpected errors including setting the wrong object or moving calibrated cameras. For the Omnipush dataset, we made sure that we had properly built all objects by checking the first collected image of that object. We realized that 5 objects had been wrongly assembled and had to recollect data for them.
4. Process some data beforehand: collecting all the dataset before using any of the data is a bad idea. Some questions that can be answered almost from the beginning: Does the recording frequency match the desired one? How frequent are the errors in the data? How many data files do you expect to discard because of errors? Include those into your estimates of how long it will take to collect the data.
5. Hardware improvements save time: it is worth to think carefully on what could go wrong or will need human intervention, and try to find ways to change the setup so that you can prevent or improve them. In our system, we can relocate the object at the center of the workspace every time it gets close to its edges by adding a circular spot that the pusher can engage to move the object. Also, hardware can break, have replacements.