package dgp.proto

Get desktop application:
View/edit binary Protocol Buffers messages

A collection of agent files.

Used in: Agents

• repeated string filenames = 1

  Relative file paths for the json files serialized from AgentGroup message. An AgentGroup message is serialized and stored under the following naming convention: <scene_dir>/agents_<scene_hash>.json where <agents_hash> is the deterministic deterministic SHA1 hash hexdigest of a agentgroup and served as the version of the scene, see dgp.utils.protobuf.generate_uid_from_pbobject for details.

AgentGroup encapsulates all Agents in a Scene.

• string name = 1

  Unique name for the group of Agents in a Scene. Usually, group name is set as the hash of algorithm or label that generate the agents. For group without unique names, generate a sha1 hash of all the AgentSnapshot keys combined.

• string description = 2

  Optional, short description of the group (~200 characters).

• string log = 3

  Log identifier Refer to dgp.identifiers.proto for more information.

• string agents_slices_file = 4

  List of AgentSlices corresponding to this Scene, each Slice contains AgentSnapshots from one Sample in the Scene.

• string agent_tracks_file = 5

  List of AgentTracks in this Scene.

• map<string, google.protobuf.Any> metadata = 6

  Optional metadata

• optional google.protobuf.Timestamp creation_date = 7

  Date and timestamp of AgentGroup creation.

• map<uint32, string> feature_ontologies = 8

  Task-specific feature ontologies. Ontologies are stored under <scene_dir>/feature_ontology/<ontology_sha1>.json Maps dgp.proto.FeatureType (Agent_3D, EGO_INTENTION, etc) to the filename containing the ontology for the specific FeatureType/Task.

• map<uint32, string> agent_ontologies = 9

  Task-specific agent ontologies. Agent ontologies are stored under <scene_dir>/ontology/<ontology_sha1>.json Maps dgp.proto.AnnotationType (bounding_box_3d, etc) to the filename containing the ontology for the specific AgentType/Task.

AgentSnapshot that takes values from one of agent types.

Used in: AgentTrack, AgentsSlice

• oneof agent_snapshot_oneof

  • AgentSnapshot2D agent_snapshot_2D = 1

  • AgentSnapshot3D agent_snapshot_3D = 2

• optional DatumId slice_id = 3

  Unique slice identifier (See dgp.idenfiers.proto) Copy from Sample.id to ensure one-one match between a AgentSlice and Sample in a Scene.

• optional DatumId raw_sensor_datum_id = 4

  Unique slice identifier (See dgp.idenfiers.proto). Copy from Datum.id if the agent snapshot is associate with certain datum.

Used in: AgentSnapshot

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]) For datasets supporting semantic segmentation and detection, num_classes corresponds to the number of segmentation classes. For datasets only supporting detection, this is the number of thing classes.

• optional BoundingBox2D box = 2

  2D box

• uint32 area = 3

  Other fields useful for downstream metric computation.

• bool iscrowd = 4

  Refer to crowd box in coco dataset.

• uint32 instance_id = 5

  Instance identifier for this agent. This needs to be unique in an image.

• repeated string features = 6

  List of features, which are the values of the fields associated with certain FeatureType in dgp.features.proto.

• uint32 feature_type = 7

  The type of feature ontology defined in FeatureType in features.proto.

Used in: AgentSnapshot

• uint32 class_id = 1

  Class identifier. Should be in range [0, num_classes - 1].

• optional BoundingBox3D box = 2

  3D box, the pose of box is in sensor coordinate.

• uint32 instance_id = 3

  Instance identifier for this agent. This needs to be unique to a scene.

• repeated string features = 4

  List of features, which are the values of the fields associated with certain FeatureType in dgp.features.proto.

• uint32 feature_type = 5

  The type of feature ontology defined in FeatureType in features.proto.

AgentTrack is the track of a single Agent in the Scene.

Used in: AgentTracks

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1])

• uint32 instance_id = 2

  Agent identifier

• repeated AgentSnapshot agent_snapshots = 3

  List of AgentSnapshots encapsulated in the AgentTrack.

• optional google.protobuf.Any metadata = 4

  Compiled proto is available to both the writer and reader of the dataset.

• repeated AgentTrack agent_tracks = 1

• map<string, google.protobuf.Any> metadata = 2

  Sample-specific metadata

Road actors in the Scene (Pedestrians, vehicles, etc).

• optional DatasetMetadata metadata = 1

  Dataset-specific metadata

• map<int32, AgentFiles> agents_splits = 2

  List of SceneAgents recorded for each split (train/val/test) in this dataset. Note: This corresponds to a particular task in the dataset. Maps <DatasetSplit, SceneAgents>

AgentsSlice encapsulates all Agents in a Sample.

Used in: AgentsSlices

• optional DatumId slice_id = 1

  Unique Slice identifier (See dgp.idenfiers.proto) Copy from unique Sample identifier (See dgp.sample.proto) to ensure one-one match between a AgentSlice and Sample. in a Scene.

• repeated AgentSnapshot agent_snapshots = 2

  List of AgentSnapshots encapsulated in the AgentsSlice.

• repeated AgentsSlice agents_slices = 1

• map<string, google.protobuf.Any> metadata = 2

  Sample-specific metadata

Generic Annotation Enumeration. Annotations will be available under the corresponding enum name. The annotation files will reside under the corresponding directory names specified in the comments on the right. For e.g., BOUNDING_BOX_2D annotations reside under bounding_box_2d/*.json

• BOUNDING_BOX_2D = 0

  2D bounding box annotations on the image

  bounding_box_2d

• BOUNDING_BOX_3D = 1

  3D bounding box annotations on the image or point cloud. Note: Typically, the annotations are provided in the point cloud reference frame, and optionally also added as annotations in the camera's reference frame.

  bounding_box_3d

• SEMANTIC_SEGMENTATION_2D = 2

  Semantic segmentation on the image.

  semantic_segmentation_2d

• SEMANTIC_SEGMENTATION_3D = 3

  Semantic segmentation on the point cloud.

  semantic_segmentation_3d

• INSTANCE_SEGMENTATION_2D = 4

  Instance segmentation on the image.

  instance_segmentation_2d

• INSTANCE_SEGMENTATION_3D = 5

  Instance segmentation on the point cloud.

  instance_segmentation_3d

• DEPTH = 6

  Depth map on image.

  depth

• SURFACE_NORMALS_2D = 13

  3D Surface normals on the image.

  surface_normals_2d

• SURFACE_NORMALS_3D = 7

  3D Surface normals on the point cloud.

  surface_normals_3d

• MOTION_VECTORS_2D = 8

  2D Motion vectors on the image. (i.e. 2D optical flow)

  motion_vectors_2d

• MOTION_VECTORS_3D = 9

  3D Motion vectors on the point cloud. (i.e. 3D flow on the point cloud)

  motion_vectors_2d

• KEY_POINT_2D = 10

  2D Key Point on the image.

  key_point_2d

• KEY_LINE_2D = 11

  2D Key Line on the image.

  key_line_2d

• POLYGON_2D = 12

  2D Polygon on the image.

  polygon_2d

• AGENT_BEHAVIOR = 14

  Agent behavior

  agent_behavior

• CLASSIFICATION = 15

  Classification. (i.e. imagenet classification label)

  classification

• KEY_POINT_3D = 16

  3D Key Point in sensor space

  key_point_3d

• KEY_LINE_3D = 17

  3D Key Line in sensor space

  key_line_3d

• POLYGON_3D = 18

  3D Polygon in sensor space

  polygon_3d

The geometry of the edges of zone, represented as a chain of polylines to form a polygon. Each edge in a boundary is associated in iteration order, where the last waypoint of `explicit` edge n has an `implicit` connection to the first waypoint of `explicit` edge n+1, including wrapping from the last `explicit` edge to the first edge in a boundary. This means, when considering implicit edges, a boundary could be considered a closed polygon. A zone has potentially multiple boundaries, because there may be multiple discrete areas of interest in a single zone (e.g. an island in the middle of an intersection). Multiple boundaries in the same zone are fully independent from a data structure point of view.

Used in: Zone

• repeated LaneDivider edges = 1

  List of edges of boundary polygon.

2D bounding box

Used in: AgentSnapshot2D, BoundingBox2DAnnotation

• int32 x = 1

  (x, y) is top left corner (in absolute pixel coordinates)

• int32 y = 2

• uint32 w = 3

  (w, h) is width and height (in absolute pixel coordinates)

• uint32 h = 4

2D bounding box annotation for a single instance.

Used in: BoundingBox2DAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]) For datasets supporting semantic segmentation and detection, num_classes corresponds to the number of segmentation classes. For datasets only supporting detection, this is the number of thing classes.

• optional BoundingBox2D box = 2

  2D box

• uint32 area = 3

  Other fields useful for downstream metric computation.

• bool iscrowd = 4

  Refer to crowd box in coco dataset.

• uint32 instance_id = 5

  Instance identifier

• map<string, string> attributes = 6

  A map of attribute names to their values. This can be used to stored `agent_behavior` states (i.e., parked car, pedestrian intent).

List of BoundingBox2DAnnotation

• repeated BoundingBox2DAnnotation annotations = 1

3D bounding box

3D Bounding box. Produced by first constructing a box of dimensions width, length, height centered at the origin, then rotating it by the specified quaternion and finally translating the box to the center coordinate.

Used in: AgentSnapshot3D, BoundingBox3DAnnotation

• optional Pose pose = 1

  6DoF pose of bounding 3D. (Translation is the center point of the box)

• double width = 2

  Dimensions of box.

• double length = 3

• double height = 4

• uint32 occlusion = 5

  Integer (0,1,2,3) indicating occlusion state: 0 = fully visible, 1 = partly occluded 2 = largely occluded, 3 = unknown

• double truncation = 6

  From 0 (non-truncated) to 1 (truncated), where truncated refers to the object leaving image boundaries.

3D bounding box annotation.

Used in: BoundingBox3DAnnotations

• uint32 class_id = 1

  Class identifier. Should be in range [0, num_classes - 1], where num_classes is the total number of classes in your ontology.

• optional BoundingBox3D box = 2

  3D box.

• uint32 instance_id = 3

  Instance identifier for this annotation. This needs to be unique to a scene.

• map<string, string> attributes = 4

  A map of attribute names to their values.

• uint32 num_points = 5

  Number of LIDAR points in the Bounding Box

List of BoundingBox3DAnnotation.

• repeated BoundingBox3DAnnotation annotations = 1

Used in: SampleCalibration

• double fx = 1

• double fy = 2

• double cx = 3

• double cy = 4

• double skew = 5

• double fov = 6

  Camera distortion parameters, copied over from Parallel Domain These include the default opencv distorition model parameters (Brown Conrady) See https://en.wikipedia.org/wiki/Distortion_(optics) and opencv calibrateCamera for details Additionally the Double Sphere Camera Model (https://arxiv.org/pdf/1807.08957.pdf) by Usenko, Demmel, and Cremers is an excellent reference

• int32 fisheye = 7

• double k1 = 8

• double k2 = 9

• double k3 = 10

• double k4 = 11

• double k5 = 12

• double k6 = 13

• double p1 = 14

• double p2 = 15

• double s1 = 16

  Additional parameters for thin prism model

• double s2 = 17

• double s3 = 18

• double s4 = 19

• double taux = 20

  Additional parameters for tilted sensor model

• double tauy = 21

• double alpha = 22

  Additional parameters for UCM and EUCM camera models

• double beta = 23

• double w = 24

  Additional parameters for FOV camera model

• double xi = 25

  Parameters for Double Sphere model

• int32 equirectangular = 26

  Flag to indicate camera model is Equirectangular

Classification annotation.

Used in: ClassificationAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• map<string, string> attributes = 2

  A map of attribute names to their values.

List of ClassificationAnnotation.

• repeated ClassificationAnnotation annotations = 1

Artifacts for a dataset registered in the TRI-ML DGP format.

• optional DatasetMetadata metadata = 1

  Dataset-specific metadata Note: The metadata in the dataset artifact is maintained for traceability purposes.

• optional RemoteArtifact artifact = 2

  DGP-compliant dataset artifact

• repeated string derived_from = 3

  sha1sum of all parent DatasetArtifacts JSON files this DatasetArtifacts is derived from. We are implicitly creating a linked list of DatasetArtifacts where the first item in the list has `derived_from` field empty and all subsequent items have it populated.

Dataset metadata specific to a particular SceneDataset.

Used in: Agents, DatasetArtifacts, SceneDataset

• string name = 1

  Dataset name

• string version = 2

  Dataset semantic version TODO: Establish semantic versioning.

• string creation_date = 3

  Date of dataset creation: yyyy-mm-dd TODO: Change this to datetime

• string creator = 4

  Dataset creator (for e.g., unique email address: john.doe@domain.com)

• optional RemotePath bucket_path = 5

  Remote path to DGP-compliant dataset (s3://<bucket>/)

• optional RemotePath raw_path = 6

  Remote path to the raw dataset (s3://<bucket>/bdd100k/)

• string description = 7

  Short description of the dataset (~200 characters)

• DatasetMetadata.DatasetOrigin origin = 8

• repeated int32 available_annotation_types = 9

  List of annotation types supported by the dataset See dgp.proto.AnnotationType in dgp/proto/annotations.proto

• optional DatasetStatistics statistics = 10

  (Optional) Dataset-specific statistics This is usually of the TRAIN set.

• float frame_per_second = 11

  (Optional) Sampling rate of the dataset This is used to synchronize different fps from different datasets.

• optional google.protobuf.Any metadata = 12

  Container for generic metadata. Metadata of "Any" type is useful in cases where the metadata type is specific to a paricular dataset type. For example, this field can contain: - a bare string which contains structured data like json, yaml, etc. - dataset specific metadata with some proto definition (assuming the compiled proto is available to both the writer and reader of the dataset.

Dataset origin identifier (i.e. whether the dataset originated internally or from the public domain)

Used in: DatasetMetadata

• PUBLIC = 0

• INTERNAL = 1

Dataset Split Enum

• TRAIN = 0

• VAL = 1

• TEST = 2

• TRAIN_OVERFIT = 3

Statistics for a dataset or a scene

Used in: DatasetMetadata, Scene

• optional DatasetStatistics.ImageStatistics image_statistics = 1

  Image and point cloud statistics

• optional DatasetStatistics.PointCloudStatistics pointcloud_statistics = 2

Statistics for Image dataset

Used in: DatasetStatistics

• int64 count = 1

• repeated float mean = 2

• repeated float stddev = 3

Statistics for Point cloud dataset

Used in: DatasetStatistics

• int64 count = 1

• repeated float mean = 2

• repeated float stddev = 3

Datum encapsulates a single data element and associated metadata to uniquely identify it in a series of logged data.

Used in: Scene

• optional DatumId id = 1

  Datum identifier Unique identifier to a datum in a specific log

• string key = 2

  Datum identifier key key = sha1(datum_id) = sha1(SerializeToString(DatumId(...))

• optional DatumValue datum = 3

  Datum envelope

• string next_key = 4

  Next datum hash key. The sha1 hash to the datum_id that follows this one. The hash corresponds to the hash of the DatumValue object. If empty, this datum is the last item in the linked list.

• string prev_key = 5

  Previous datum hash key. The datum hash to the datum_id that precedes this one. The hash corresponds to the hash of the DatumValue object. If empty, this datum is the first item in the linked list.

For certain scenarios, the sample_name will also include the directory/session name to make the sample name unique to the dataset. For internal datasets, this would be the relevant log information for traceability purposes.

Used in: AgentSnapshot, AgentsSlice, Datum, LocalEarthTransform, Sample

• string log = 1

  Log identifier For TRI / internal datasets: log = <robot_name>/<session_name>/<log_id>.<ext> Public datasets: For e.g. The KITTI raw dataset contains camera 02 images under 2011_09_26/2011_09_26_drive_0001_sync/image_02/data/0000000042.png The corresponding unique log_id would be: log = 2011_09_26/2011_09_26_drive_0001_sync

• string name = 2

  Datum name (i.e. camera_0, lidar_0) Alternatively, this serves as a datum identifier for public datasets, or untraceable datasets For public datasets where the datasets are derived from non-sequential logs, the name might be a sufficient identifier for a datum. For e.g., filename=rgb/1aa6dd36-aa7f0108.jpg, name=1aa6dd36-aa7f0108 For datums without unique names, only if the datums can not be traced, generate a sha1 hash of the datum contents.

• optional google.protobuf.Timestamp timestamp = 3

  Timestamp within the log (if available)

• uint64 index = 4

  Index within the log (if available) For TRI / internal datasets: Frame index required to find the image within the associated .mkv file. For public datasets: For e.g. The KITTI raw dataset contains camera 02 images under 2011_09_26/2011_09_26_drive_0001_sync/image_02/data/0000000042.png The corresponding unique index would be: 42

DatumValue that takes one of many data type values

Used in: Datum

• oneof datum_oneof

  • Image image = 1

  • PointCloud point_cloud = 2

  • FileDatum file_datum = 3

  • RadarPointCloud radar_point_cloud = 4

Generic Feature Type Enumeration. Features will be available under the corresponding enum name.

• AGENT_2D = 0

  Feature type of default 2D agents.

• AGENT_3D = 1

  Feature type of default 3D agents.

• EGO_INTENTION = 2

  Feature type of ego intent ontology.

• CORRIDOR = 3

  Feature type of corridor map element.

• INTERSECTION = 4

  Feature type of intersection.

• PARKED_CAR = 5

  Feature type of parked car ontology.

The feature value type specified in dgp.ontology.FeatureOntologyItem.

• NUMERIC = 0

  Feature value type of numeric values.

• FILE = 1

  Feature value type of file.

Generic (datum, annotation) pair where the underlying content is stored externally in files.

Used in: DatumValue

• optional SelfDescribingFile datum = 1

  The file containing data (e.g. extracted features)

• map<int32, SelfDescribingFile> annotations = 2

  The file containing annotations for the above datum. Typically this field should not contain manual annotation data, since any data requiring large numbers of manual labels should have those labels described with an explicit (required) schema. The key integer is defined in annotations.proto's AnnotationType

Basic image container for DGP

Used in: DatumValue

• string filename = 1

  Relative file path for the image Supported image formats: (png/jpg/jpeg)

• int32 height = 2

  Image shape

• int32 width = 3

• int32 channels = 4

• map<int32, string> annotations = 5

  Annotations for an Image Maps dgp.proto.AnnotationType (segmentation, instances) to the filename containing the image annotation. Supported image annotation formats: (png/json)

• map<string, google.protobuf.Any> metadata = 6

  Optional metadata (i.e. miscellaneous enriched annotations)

• optional Pose pose = 7

  Ego-pose of this datum with respect to the first sample in the scene.

Instance segmentation 3D annotation.

Used in: InstanceSegmentation3DAnnotations

• uint32 instance_id = 1

  Instance identifier (should be in [0, num_instances - 1]), where num_instances is the total number of instances in your point cloud.

• uint32 class_id = 2

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• optional InstanceSegmentationPoint3D point = 3

  3D point.

• map<string, string> attributes = 4

  A map of attribute names to their values.

• uint32 point_index = 5

  Reference to the index of the original point in datum point cloud.

List of InstanceSegmentation3DAnnotation.

• repeated InstanceSegmentation3DAnnotation annotations = 1

3D point.

Used in: InstanceSegmentation3DAnnotation

• float x = 1

  (x, y, z) point (in 3D Cartesian coordinates).

• float y = 2

• float z = 3

2D line annotation.

Used in: KeyLine2DAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• repeated KeyPoint2D vertices = 2

  2D line.

• map<string, string> attributes = 3

  A map of attribute names to their values. Add only key/value pairs that are stored in a project document accessible to project contributors.

• string key = 4

  An identifier key. Used to link with other annotations, which specify this key in their instance to link to corresponding KeyLine2DAnnotation.

List of KeyLine2DAnnotation.

• repeated KeyLine2DAnnotation annotations = 1

3D line annotation.

Used in: KeyLine3DAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• repeated KeyPoint3D vertices = 2

  3D line.

• map<string, string> attributes = 3

  A map of attribute names to their values. Add only key/value pairs that are stored in a project document accessible to project contributors.

• string key = 4

  An identifier key. Used to link with other annotations, which specify this key in their instance to link to corresponding KeyLine3DAnnotation.

List of KeyLine3DAnnotation.

• repeated KeyLine3DAnnotation annotations = 1

2D point.

Used in: KeyLine2DAnnotation, KeyPoint2DAnnotation

• int32 x = 1

  (x, y) point (in absolute pixel coordinates).

• int32 y = 2

2D point annotation.

Used in: KeyPoint2DAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• optional KeyPoint2D point = 2

  2D point.

• map<string, string> attributes = 3

  A map of attribute names to their values.

• string key = 4

  An identifier key. Used to link with other annotations (example: BoundingBox2D) which specify this key in their instance to link to corresponding KeyPoint2DAnnotation.

List of KeyPoint2DAnnotation.

• repeated KeyPoint2DAnnotation annotations = 1

3D point.

Used in: KeyLine3DAnnotation, KeyPoint3DAnnotation

• float x = 1

  (x, y, z) point (in 3D Cartesian coordinates).

• float y = 2

• float z = 3

3D point annotation.

Used in: KeyPoint3DAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• optional KeyPoint3D point = 2

  3D point.

• map<string, string> attributes = 3

  A map of attribute names to their values. Add only key/value pairs that are stored in a project document accessible to project contributors.

• string key = 4

  An identifier key. Used to link with other annotations, which specify this key in their instance to link to corresponding KeyPoint3DAnnotation.

List of KeyPoint3DAnnotation.

• repeated KeyPoint3DAnnotation annotations = 1

Defines lane information - centerline, boundaries, etc. A lane is defined as https://en.wikipedia.org/wiki/Lane.

Used in: SceneMap

• string id = 1

  Identifier for the lane.

• optional LaneDivider center_line = 2

  Center line of the lane. It is geometrically the centers between left and right lane boundaries.

• optional LaneDivider left_lane_boundary = 3

  Left lane boundary.

• optional LaneDivider right_lane_boundary = 4

  Right lane boundary.

Basic container for line divider on road, contains sequence of points to form a polyline.

Used in: Boundary, Lane

• repeated Point3D way_points = 1

  List of points to form a continuous line. The points are ordered along lane direction.

• LaneDivider.LaneDividerType type = 2

  Type of LaneDivider.

• LaneDivider.LaneDividerColor color = 3

  Color of LaneDivider.

• LaneDivider.LaneDividerMaterial material = 4

  Material of LaneDivider.

Enum of color of the lane divider.

Used in: LaneDivider

• COLOR_UNKNOWN = 0

• COLOR_YELLOW = 1

• COLOR_WHITE = 2

• COLOR_RED = 3

Enum of material of the lane divider.

Used in: LaneDivider

• MATERIAL_UNKNOWN = 0

• MATERIAL_PAINT = 1

Enum of types of the LaneDivider.

Used in: LaneDivider

• TYPE_UNKNOWN = 0

• TYPE_SINGLE_SOLID = 1

• TYPE_SINGLE_DASHED = 2

• TYPE_SINGLE_WHITE_DASHED = 3

• TYPE_DOUBLE_SOLID = 4

• TYPE_DOUBLE_DASHED = 5

• TYPE_DOUBLE_SOLID_FAR_DASHED_NEAR = 6

• TYPE_DOUBLE_DASHED_FAR_SOLID_NEAR = 7

• TYPE_CURB = 8

Transform from a Earth-centered Earth-fixed coordinate to Local coordinate.

Used in: SceneMap

• optional DatumId sample_id = 1

  Unique sample identifier (See dgp.idenfiers.proto). Copy from sample ID to ensure one-one match between a localmapview and sample in a scene.

• optional Pose pose = 2

  Pose of this sample in Earth-centered Earth-fixed coordinate system.

Dataset ontology

• map<string, int64> name_to_id = 1

  Dictionary maintaining the class name to unique class id mapping

• map<int64, string> id_to_name = 2

  Inverted mapping of name_to_id

• map<int64, Ontology.Color> colormap = 3

• map<int64, bool> isthing = 4

• map<int64, string> supercategory = 5

• map<int64, int64> segmentation_ids = 6

• int64 ignore_id = 7

Various look-up tables

Used in: Ontology

• int32 r = 1

• int32 g = 2

• int32 b = 3

Sample-level metadata to encode attribute information in key-value pairs.

• map<string, string> attributes = 1

Scene-level metadata.

• string location = 1

  Location name in PD location library The location name string will be of the form xx_yy, where xx = a two letter region code (eg. SF=San Francisco, SJ=San Jose, ...) yy = a descriptive level name (eg. 6thAndMission_medium) eg. SF_6thAndMission_medium

• RegionType region_type = 9

  Region type enumeration defining geographic region

• SceneType scene_type = 10

  Scene type enumeration defining scene land usage

• string time_of_day = 2

  Time of day environment from PD time_of_day library The time_of_day name string will be of the form LS_sky_xxx_yyy_zzz_www, where xxx = descriptive time of day [dawn, noon, afternoon, dusk, evening, night] yyy = descriptive sky conditions [clear, mostlySunny, partlyCloudy, overcast] zzz = approximate time of day in 24HR clock (eg. 1830, 1950, 0440, ...) www = internal code eg. LS_sky_afternoon_mostlySunny_1444_HDS035

• float sun_elevation = 11

  Sun elevation angle, in degrees above the horizon For night scenarios, this angle will be negative

• float sun_azimuth = 12

  Sun azimuth angle, in degrees anti-clockwise from Rast

• float cloud_cover = 13

  Cloud coverage ratio. Measures the proportion of the sky hemisphere that is classified as cloudy [0.0 -> 1.0]

• float fog_intensity = 3

  Fog intensity [0.0 = clear day, 1.0 = whiteout]

• float rain_intensity = 4

  Rain intensity [0.0 = no rain, 1.0 = torrential rain]

• float wetness = 5

  Road surface wetness [0.0 = completely dry, 1.0 = flooded]

• float street_lights = 6

  Streetlight intensity [0.0 = off, 1.0 = completely on]

• uint32 batch_id = 7

  Batch ID and version for delivery tracking

• uint32 version = 8

• map<string, string> attributes = 14

  Additional metadata

Coordinates on the map.

Used in: LaneDivider

• float x = 1

• float y = 2

• float z = 3

Basic point cloud container for DGP

Used in: DatumValue

• string filename = 1

  Relative file path for the point cloud Supported point cloud formats: compressed numpy (npz) format, polygon file format (ply).

• map<int32, string> annotations = 2

  PointCloud annotations Maps dgp.proto.AnnotationType (segmentation, instances) to the filename containing the point cloud annotation. Point cloud annotation schemas are defined in 'annotations.proto'.

• map<string, google.protobuf.Any> metadata = 3

  Optional metadata (i.e. miscellaneous enriched annotations) TODO: Support only specific metadata types

• repeated PointCloud.ChannelType point_format = 4

  Structure of values in the pointcloud file. Each entry in this array specifies the field meaning. A simple pointcloud may simply contain X, Y, and Z channels. This would mean that the numpy data representation of this would be an [N, 3] shape structured array.

• repeated string point_fields = 5

  Alternatively, use point fields instead of strongly-typed ChannelType For example, radar data can be represented as D-dimensional tensors/matrices with point_fields describing the columns for the point cloud data

• optional Pose pose = 6

  Ego-pose of this datum with respect to the first sample in the scene.

Used to describe the values in the pointcloud. The ChannelType refers to the dtypes of the point cloud stored under the 'filename' key.

Used in: PointCloud

• X = 0

  X, Y, and Z coordinates of point.

• Y = 1

• Z = 2

• INTENSITY = 3

  Intensity of return at point.

• R = 4

  RGB for color.

• G = 5

• B = 6

• RING = 7

  The ring index of this point. This is commonly used for spinning lidar.

• NORMAL_X = 8

• NORMAL_Y = 9

• NORMAL_Z = 10

• TIMESTAMP = 13

  Timestamp of this point sample, stored as UTC in microseconds

2D polygon annotation.

Used in: Polygon2DAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• repeated PolygonPoint2D vertices = 2

  2D polygon. Points should be put into this field with counter-clockwise order.

• map<string, string> attributes = 3

  A map of attribute names to their values.

List of Polygon2DAnnotation.

• repeated Polygon2DAnnotation annotations = 1

3D polygon annotation.

Used in: Polygon3DAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• repeated PolygonPoint3D vertices = 2

  3D polygon. Points should be put into this field with counter-clockwise order.

• map<string, string> attributes = 3

  A map of attribute names to their values. Add only key/value pairs that are stored in a project document accessible to project contributors.

List of Polygon3DAnnotation.

• repeated Polygon3DAnnotation annotations = 1

2D point for polygon annotation which allows negative coordinate for truncated rotated bounding box at the edge of an image because this kind of negative coordinate is meaningful.

Used in: Polygon2DAnnotation

• int32 x = 1

  (x, y) point (in absolute pixel coordinates).

• int32 y = 2

Used in: Polygon3DAnnotation

• float x = 1

  (x, y, z) point (in 3D Cartesian coordinates).

• float y = 2

• float z = 3

6-DOF Pose

Used in: BoundingBox3D, Image, LocalEarthTransform, PointCloud, RadarPointCloud, SampleCalibration

• optional Vector3 translation = 1

• optional Quaternion rotation = 2

• string reference_coordinate_system = 3

  Indicate the name of the coordinate system the above transform is from: i.e. let x_ref be a vector in reference_coordinate_system and x_pose a vector in this Pose's coordinate system (e.g. sensor's), then x_ref = rotation . x_pose + translation TODO: Coordinate systems Naming Convention

3D probabilistic line annotation.

Used in: ProbabilisticKeyLine3DAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• repeated ProbabilisticKeyPoint3D vertices = 2

  3D line.

• map<string, string> attributes = 3

  A map of attribute names to their values. Add only key/value pairs that are stored in a project document accessible to project contributors.

• string key = 4

  An identifier key. Used to link with other annotations, which specify this key in their instance to link to corresponding KeyLine3DAnnotation.

List of ProbabilisticKeyLine3DAnnotation.

• repeated ProbabilisticKeyLine3DAnnotation annotations = 1

3D point with uncertainty information.

Used in: ProbabilisticKeyLine3DAnnotation, ProbabilisticKeyPoint3DAnnotation

• float x = 1

  (x, y, z) point (in 3D Cartesian coordinates).

• float y = 2

• float z = 3

• float var_x = 4

  The correlation terms in the 3x3 covariance matrix.

• float cov_xy = 5

• float cov_xz = 6

• float var_y = 7

• float cov_yz = 8

• float var_z = 9

3D probabilistic point annotation.

Used in: ProbabilisticKeyPoint3DAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• optional ProbabilisticKeyPoint3D point = 2

  3D point.

• map<string, string> attributes = 3

  A map of attribute names to their values. Add only key/value pairs that are stored in a project document accessible to project contributors.

• string key = 4

  An identifier key. Used to link with other annotations, which specify this key in their instance to link to corresponding ProbabilisticKeyPoint3DAnnotation.

List of ProbabilisticKeyPoint3DAnnotation.

• repeated ProbabilisticKeyPoint3DAnnotation annotations = 1

Quaternion used to represent a 3D rotation Note, that rotation quaternions are unit-quaternions

Used in: Pose

• double qx = 1

• double qy = 2

• double qz = 3

• double qw = 4

Basic point cloud container for DGP

Used in: DatumValue

• string filename = 1

  Relative file path for the radar point cloud Supported point cloud formats: compressed numpy (npz) format.

• map<int32, string> annotations = 2

  RadarPointCloud annotations Maps dgp.proto.AnnotationType (segmentation, instances) to the filename containing the point cloud annotation. Radar Point cloud annotations are written in to compressed numpy arrays (npz) under the 'data' key.

• map<string, google.protobuf.Any> metadata = 3

  Optional metadata (i.e. miscellaneous enriched annotations) TODO: Support only specific metadata types

• repeated RadarPointCloud.ChannelType point_format = 4

  Structure of values in the pointcloud file. Each entry in this array specifies the field meaning. A simple pointcloud may simply contain X, Y, and Z channels. This would mean that the numpy data representation of this would be an [N, 3] shape structured array.

• repeated string point_fields = 5

  Alternatively, use point fields instead of strongly-typed ChannelType For example, radar data can be represented as D-dimensional tensors/matrices with point_fields describing the columns for the point cloud data

• optional Pose pose = 6

  Ego-pose of this datum with respect to the first sample in the scene.

Used to describe the values in the radar pointcloud. The ChannelType refers to the dtypes of the radar point cloud stored under the 'data' key.

Used in: RadarPointCloud

• X = 0

  X, Y, and Z coordinates of point.

• Y = 1

• Z = 2

• V_X = 3

  Radar specific channels Point Velocity

• V_Y = 4

• V_Z = 5

• RCS_DBSM = 6

  Radar cross section

• EXISTENCE_PROBABILITY = 7

• SENSOR_ID = 8

• COV_XX = 9

  Support full covariance matrix for points converted to rectangular this will be symmetric so not all channels are needed

• COV_XY = 10

• COV_XZ = 11

• COV_YX = 12

• COV_YY = 13

• COV_YZ = 14

• COV_ZX = 15

• COV_ZY = 16

• COV_ZZ = 17

• RADIAL_DISTANCE = 18

  Support for raw radar points in spherical coordinates of sensor frame

• AZIMUTH_ANGLE = 19

• ELEVATION_ANGLE = 20

• VELOCITY_XS = 21

• RADIAL_DISTANCE_VARIANCE = 22

• AZIMUTH_ANGLE_VARIANCE = 23

• ELEVATION_ANGLE_VARIANCE = 24

• VELOCITY_VARIANCE = 25

• ACCEL_XS = 26

  Extra channels to support radar target message

• COUNT_ALIVE = 27

• REFLECTED_POWER_DB = 28

• R = 29

  RGB for color.

• G = 30

• B = 31

• CLASS_ID = 32

  Class/Instance ID for point cloud semantic segmentation

• INSTANCE_ID = 33

• TIMESTAMP = 34

  Timestamp of this point sample, stored as UTC in microseconds

Region type enumeration Used to specify rough geographic region for corresponding Scene

Used in: ParallelDomainSceneMetadata

• SYNTHETIC_REGION = 0

• NORTHERN_CALIFORNIA = 1

• MICHIGAN = 2

Artifact that is hosted in a remote location as defined by RemotePath

Used in: DatasetArtifacts

• optional RemotePath url = 1

  Remote path of the artifact.

• string sha1 = 2

  sha1sum of the artifact contents. In the case where the artifact is a full directory, the sha1 is generated by computing the sha1sum on the compressed directory. i.e. via `tar -cf - <dir> | sha1sum`.

• bool isdir = 3

  Whether the remote artifact is a directory or a file. In the case of directory, the sha1 is generated by computing the sha1sum on the compressed directory. i.e. via `tar -cf - <dir> | sha1sum`.

A remote path that specifies the url of a file or directory For e.g. HTTP or S3 url (http://, s3://)

Used in: DatasetMetadata, RemoteArtifact

• string value = 1

Dataset sample that refers to data associated to a particular timestamp. A dataset sample is a container that encapsulates time-synchronized sensor data (images, point clouds, GPS/IMU etc), along with their associated annotations. Since sensor data are recorded in an asynchronous manner, the time-synchronization is assumed to be approximate. Samples within a scene are expected to have the same number of datums as their previous and next samples. In addition, we assume that the first sample in a scene will be the first datum in its chain (i.e. the datum(s) will all have prev_key set to null).

Used in: Scene

• optional DatumId id = 1

  Unique sample identifier (See dgp.idenfiers.proto) For samples, the "name" field is ignored. Typically, list of datum items in a sample are synchronized to the first item in the datum list.

• repeated string datum_keys = 2

  List of datum keys encapsulated in the sample

• string calibration_key = 3

  Calibration identifier (sha1 of the calibration object)

• map<string, google.protobuf.Any> metadata = 4

  Sample-specific metadata Optional, miscellaneous enriched annotations

Calibration for a sample

• repeated string names = 1

  Sensor names for each of the corresponding datums

• repeated CameraIntrinsics intrinsics = 2

  Intrinsics of the corresponding datum

• repeated Pose extrinsics = 3

  Extrinsincs of the corresponding datum with respect to the first datum [T_0, T_1, T_2] -> [T_0.inv() * T_0, T_0.inv() * T_1, T_0.inv() * T_2]

A single scene consisting of sequential frames in a robot session.

Used in: Scenes

• string name = 1

  Unique name for a scene For public datasets where the scenes are uniquely identified, the scene name is directly used. For scenes without without unique names, generate a sha1 hash of all the datum keys combined.

• string description = 2

  Optional, short description of the scene (~200 characters)

• string log = 3

  Log identifier Refer to dgp.identifiers.proto for more information.

• repeated Sample samples = 4

  List of samples corresponding to this scene.

• map<string, google.protobuf.Any> metadata = 5

  Optional metadata (i.e. miscellaneous enriched scene-level annotations).

• repeated Datum data = 6

  List of Datum's contained in this Scene.

• optional google.protobuf.Timestamp creation_date = 7

  Date and timestamp of scene creation.

• map<int32, string> ontologies = 8

  Task-specific ontologies Ontologies are stored under <scene_dir>/ontology/<ontology_sha1>.json Maps dgp.proto.AnnotationType (segmentation, instances, etc) to the filename containing the ontology for the specific AnnotationType/Task.

• optional DatasetStatistics statistics = 9

  Scene statistics SceneDataset statistics will be aggregated on-the-fly.

Scene Dataset Scenes are stored in the DGP under the following structure: <scene_dataset_name> ├── <scene_name> │   ├── bounding_box_2d │   │   └── ... │   ├── bounding_box_3d │   │   └── ... │   ├── calibration │   │   └── <calibration_hash>.json │   │   └── ... │   ├── ontology │   │   └── <ontology_hash>.json │   │   └── ... │   ├── point_cloud │   │   └── ... │   ├── rgb │   │   └── ... │   ├── agent_tracks │   │   └── agent_track_<hash>.json │   │   └── ... │   ├── agents_slices │   │   └── agent_slice<hash>.json │   │   └── ... | └── ... │   └── scene_<scene_hash>.json │   └── agent_<agent_hash>.json │   └── map_<agent_hash>.json └── ... └── scene_dataset_v1.0.json └── agents_v1.0.json └── map_elements_v1.0.json

• optional DatasetMetadata metadata = 1

  Dataset-specific metadata

• map<int32, SceneFiles> scene_splits = 2

  List of scenes recorded for each split (train/val/test) in this dataset. Note: This corresponds to a particular task in the dataset. Maps <DatasetSplit, Scenes>

A collection of scenes

Used in: SceneDataset

• repeated string filenames = 1

  Relative file paths for the json files serialized from Scene message. A Scene message is serialized and stored under the following naming convention: <scene_dir>/scene_<scene_hash>.json where <scene_hash> is the deterministic deterministic SHA1 hash hexdigest of a scene and served as the version of the scene, see dgp.utils.protobuf.generate_uid_from_pbobject for details.

Represents a map -- either discrete segments or a pointer to an external data object (e.g. for rasterized maps).

• repeated Lane lanes = 1

  List of Lanes in the SceneMap.

• repeated Zone zones = 2

  List of zones, such as crosswalk regions, intersections.

• repeated LocalEarthTransform transforms = 3

  List transformations from earth coord to vehicle coord.

• optional google.protobuf.Any map_data = 4

  A general encoding of map information.

Scene type enumeration Used to specify a rough land usage classification for the corresponding Scene

Used in: ParallelDomainSceneMetadata

• SYNTHETIC_SCENE = 0

• URBAN = 1

• SUBURBAN = 2

• HIGHWAY = 3

List of scenes

• repeated Scene scenes = 1

Describes a file external to the serialized protobuf with an optional schema field that can be used to describe the file.

Used in: FileDatum

• string filename = 1

  Relative file path (relative to the location where this scene description is stored. This datum is expected to obey the datum schema below, if supplied.

• optional google.protobuf.Any schema = 2

  An optional schema which describes the file at `filename`. Both the reader and the writer of this SelfDescribingFile need to have access to the datatype stored under `schema`, though for simple cases this may be a simple / integral type.

Semantic segmentation 3D annotation.

Used in: SemanticSegmentation3DAnnotations

• uint32 class_id = 1

  Class identifier (should be in [0, num_classes - 1]), where num_classes is the total number of classes in your ontology.

• optional SemanticSegmentationPoint3D point = 2

  3D point.

• map<string, string> attributes = 3

  A map of attribute names to their values.

• uint32 point_index = 4

  Reference to the index of the original point in datum point cloud.

List of SemanticSegmentation3DAnnotation.

• repeated SemanticSegmentation3DAnnotation annotations = 1

3D point.

Used in: SemanticSegmentation3DAnnotation

• float x = 1

  (x, y, z) point (in 3D Cartesian coordinates).

• float y = 2

• float z = 3

3-DOF Vector represented via double

Used in: Pose

• double x = 1

• double y = 2

• double z = 3

Defines zone information. A Zone consists of a list of boundaries, each boundary having a list of edges, where an edge is a 3D polyline, `explicit edges` are those actually in the list of polylines, and `implicit edges` are those that connect implicit edges with straight lines. Examples of zones are cross-walk intersection, etc.

Used in: SceneMap

• string id = 1

  Identifier for the zone.

• string type = 2

  Type of Zone, like cross-walk.

• repeated Boundary boundaries = 3

  List of Zone boundaries.