package objectron.proto
Get desktop application:
View/edit binary Protocol Buffers messages
message
a_r_capture_metadata.proto:233Contains a list of blend shape entries wherein each item maps a specific blend shape location to its associated coefficient.
Used in:
repeated entries = 1
message
a_r_capture_metadata.proto:234Used in:
optional string blend_shape_location = 1
Identifier for the specific facial feature.
optional float blend_shape_coefficient = 2
Indicates the current position of the feature relative to its neutral configuration, ranging from 0.0 (neutral) to 1.0 (maximum movement).
message
a_r_capture_metadata.proto:134Information about the camera position and imaging characteristics for a captured video frame.
Used in: ,
optional ARCamera.TrackingState tracking_state = 1
optional ARCamera.TrackingStateReason tracking_state_reason = 2
repeated float transform = 3
4x4 row-major matrix expressing position and orientation of the camera in world coordinate space.
optional euler_angles = 4
optional int32 image_resolution_width = 5
The width and height, in pixels, of the captured camera image.
optional int32 image_resolution_height = 6
repeated float intrinsics = 7
3x3 row-major matrix that converts between the 2D camera plane and 3D world coordinate space.
repeated float projection_matrix = 8
4x4 row-major transform matrix appropriate for rendering 3D content to match the image captured by the camera.
repeated float view_matrix = 9
4x4 row-major transform matrix appropriate for converting from world-space to camera space. Relativized for the captured_image orientation (i.e. UILandscapeOrientationRight).
message
a_r_capture_metadata.proto:170The orientation of the camera, expressed as roll, pitch, and yaw values.
Used in:
optional float roll = 1
optional float pitch = 2
optional float yaw = 3
The general quality of position tracking available when the camera captured a frame.
Used in:
UNDEFINED_TRACKING_STATE = 0
UNAVAILABLE = 1
Camera position tracking is not available.
LIMITED = 2
Tracking is available, but the quality of results is questionable.
NORMAL = 3
Camera position tracking is providing optimal results.
A possible diagnosis for limited position tracking quality as of when the frame was captured.
Used in:
UNDEFINED_TRACKING_STATE_REASON = 0
NONE = 1
The current tracking state is not limited.
INITIALIZING = 2
Not yet enough camera or motion data to provide tracking information.
EXCESSIVE_MOTION = 3
The device is moving too fast for accurate image-based position tracking.
INSUFFICIENT_FEATURES = 4
Not enough distinguishable features for image-based position tracking.
RELOCALIZING = 5
Tracking is limited due to a relocalization in progress.
message
a_r_capture_metadata.proto:246Information about the pose, topology, and expression of a detected face.
Used in:
optional geometry = 1
A coarse triangle mesh representing the topology of the detected face.
optional blend_shapes = 2
A map of named coefficients representing the detected facial expression in terms of the movement of specific facial features.
repeated float transform = 3
4x4 row-major matrix encoding the position, orientation, and scale of the anchor relative to the world coordinate space.
optional bool is_tracked = 4
Indicates whether the anchor's transform is valid. Frames that have a face anchor with this value set to NO should probably be ignored.
message
a_r_capture_metadata.proto:196Container for a 3D mesh describing face topology.
Used in:
repeated vertices = 1
optional int32 vertex_count = 2
The number of elements in the vertices list.
repeated texture_coordinates = 3
optional int32 texture_coordinate_count = 4
The number of elements in the texture_coordinates list.
repeated int32 triangle_indices = 5
Each integer value in this ordered list represents an index into the vertices and texture_coordinates lists. Each set of three indices identifies the vertices comprising a single triangle in the mesh. Each set of three indices forms a triangle, so the number of indices in the triangle_indices buffer is three times the triangle_count value.
optional int32 triangle_count = 6
The number of triangles described by the triangle_indices buffer.
message
a_r_capture_metadata.proto:211Each texture coordinate represents UV texture coordinates for the vertex at the corresponding index in the vertices buffer.
Used in:
optional float u = 1
optional float v = 2
message
a_r_capture_metadata.proto:199Each vertex represents a 3D point in the face mesh, in the face coordinate space.
Used in:
optional float x = 1
optional float y = 2
optional float z = 3
message
a_r_capture_metadata.proto:519Video image and face position tracking information.
optional double timestamp = 1
The timestamp for the frame.
optional depth_data = 2
The depth data associated with the frame. Not all frames have depth data.
optional double depth_data_timestamp = 3
The depth data object timestamp associated with the frame. May differ from the frame timestamp value. Is only set when the frame has depth_data.
optional camera = 4
Camera information associated with the frame.
optional light_estimate = 5
Light information associated with the frame.
optional face_anchor = 6
Face anchor information associated with the frame. Not all frames have an active face anchor.
repeated plane_anchor = 7
Plane anchors associated with the frame. Not all frames have a plane anchor. Plane anchors and face anchors are mutually exclusive.
optional raw_feature_points = 8
The current intermediate results of the scene analysis used to perform world tracking.
optional motion_manager_snapshot = 9
Snapshot of Core Motion CMMotionManager object containing most recent motion data associated with the frame. Since motion data capture rates can be higher than rates of AR capture, the entities of this object reflect all of the aggregated events which have occurred since the last ARFrame was recorded.
message
a_r_capture_metadata.proto:104Estimated scene lighting information associated with a captured video frame.
Used in:
optional double ambient_intensity = 1
The estimated intensity, in lumens, of ambient light throughout the scene.
optional double ambient_color_temperature = 2
The estimated color temperature, in degrees Kelvin, of ambient light throughout the scene.
repeated float spherical_harmonics_coefficients = 3
Data describing the estimated lighting environment in all directions. Second-level spherical harmonics in separate red, green, and blue data planes. Thus, this buffer contains 3 sets of 9 coefficients, or a total of 27 values.
optional primary_light_direction = 4
A vector indicating the orientation of the strongest directional light source, normalized in the world-coordinate space.
optional float primary_light_intensity = 5
The estimated intensity, in lumens, of the strongest directional light source in the scene.
message
a_r_capture_metadata.proto:118Used in:
optional float x = 1
optional float y = 2
optional float z = 3
message
a_r_capture_metadata.proto:599A subdividision of the reconstructed, real-world scene surrounding the user.
Used in:
optional string identifier = 1
The ID of the mesh.
repeated float transform = 2
4x4 row-major matrix encoding the position, orientation, and scale of the anchor relative to the world coordinate space.
optional geometry = 3
3D information about the mesh such as its shape and classifications.
message
a_r_capture_metadata.proto:615Container object for mesh data of real-world scene surrounding the user. Even though each ARFrame may have a set of ARMeshAnchors associated with it, only a single frame's worth of mesh data is written separately at the end of each recording due to concerns regarding latency and memory bloat.
optional double timestamp = 1
The timestamp for the data.
repeated mesh_anchor = 2
Set of mesh anchors containing the mesh data.
message
a_r_capture_metadata.proto:557Mesh geometry data stored in an array-based format.
Used in:
repeated vertices = 1
The vertices of the mesh.
repeated faces = 2
The faces of the mesh.
repeated normals = 3
Rays that define which direction is outside for each face. Normals contain 'rays that define which direction is outside for each face', in practice the normals count is always identical to vertices count which looks like vertices normals and not faces normals.
repeated ARMeshGeometry.MeshClassification classification = 4
Classification for each face in the mesh.
message
a_r_capture_metadata.proto:564Used in:
repeated int32 vertex_indices = 1
/ Indices of vertices defining the face from correspondent array of parent / message. A typical face is triangular.
Type of objects
Used in:
NONE = 0
WALL = 1
FLOOR = 2
CEILING = 3
TABLE = 4
SEAT = 5
WINDOW = 6
DOOR = 7
message
a_r_capture_metadata.proto:558Used in:
optional float x = 1
optional float y = 2
optional float z = 3
message
a_r_capture_metadata.proto:311Information about the position and orientation of a real-world flat surface.
Used in:
optional string identifier = 1
The ID of the plane.
repeated float transform = 2
4x4 row-major matrix encoding the position, orientation, and scale of the anchor relative to the world coordinate space.
optional ARPlaneAnchor.Alignment alignment = 3
The general orientation of the detected plane with respect to gravity.
optional geometry = 4
A coarse triangle mesh representing the general shape of the detected plane.
optional center = 5
The center point of the plane relative to its anchor position. Although the type of this property is a 3D vector, a plane anchor is always two-dimensional, and is always positioned in only the x and z directions relative to its transform position. (That is, the y-component of this vector is always zero.)
optional extent = 6
The estimated width and length of the detected plane.
optional bool classification_supported = 7
A Boolean value that indicates whether plane classification is available on the current device. On devices without plane classification support, all plane anchors report a classification value of NONE and a classification_status value of UNAVAILABLE.
optional ARPlaneAnchor.PlaneClassification classification = 8
A general characterization of what kind of real-world surface the plane anchor represents.
optional ARPlaneAnchor.PlaneClassificationStatus classification_status = 9
The current state of process for classifying the plane anchor. When this property's value is KNOWN, the classification property represents characterization of the real-world surface corresponding to the plane anchor.
Used in:
UNDEFINED = 0
HORIZONTAL = 1
The plane is perpendicular to gravity.
VERTICAL = 2
The plane is parallel to gravity.
Used in:
NONE = 0
WALL = 1
FLOOR = 2
CEILING = 3
TABLE = 4
SEAT = 5
The classification status for the plane.
Used in:
UNKNOWN = 0
The classification process for the plane anchor has completed but the result is inconclusive.
UNAVAILABLE = 1
No classication information can be provided (set on error or if the device does not support plane classification).
UNDETERMINED = 2
The classification process has not completed.
KNOWN = 3
The classification process for the plane anchor has completed.
message
a_r_capture_metadata.proto:322Wrapper for a 3D point / vector within the plane. See extent and center values for more information.
Used in:
optional float x = 1
optional float y = 2
optional float z = 3
message
a_r_capture_metadata.proto:264Container for a 3D mesh.
Used in:
repeated vertices = 1
A buffer of vertex positions for each point in the plane mesh.
optional int32 vertex_count = 2
The number of elements in the vertices buffer.
repeated texture_coordinates = 3
A buffer of texture coordinate values for each point in the plane mesh.
optional int32 texture_coordinate_count = 4
The number of elements in the texture_coordinates buffer.
repeated int32 triangle_indices = 5
Each integer value in this ordered list represents an index into the vertices and texture_coordinates lists. Each set of three indices identifies the vertices comprising a single triangle in the mesh. Each set of three indices forms a triangle, so the number of indices in the triangle_indices buffer is three times the triangle_count value.
optional int32 triangle_count = 6
Each set of three indices forms a triangle, so the number of indices in the triangle_indices buffer is three times the triangle_count value.
repeated boundary_vertices = 7
Each value in this buffer represents the position of a vertex along the boundary polygon of the estimated plane. The owning plane anchor's transform matrix defines the coordinate system for these points.
optional int32 boundary_vertex_count = 8
The number of elements in the boundary_vertices buffer.
message
a_r_capture_metadata.proto:273Each texture coordinate represents UV texture coordinates for the vertex at the corresponding index in the vertices buffer.
Used in:
optional float u = 1
optional float v = 2
message
a_r_capture_metadata.proto:265Used in:
optional float x = 1
optional float y = 2
optional float z = 3
message
a_r_capture_metadata.proto:393A collection of points in the world coordinate space.
Used in:
optional int32 count = 1
The number of points in the cloud.
repeated point = 2
The list of detected points.
repeated int64 identifier = 3
A list of unique identifiers corresponding to detected feature points. Each identifier in this list corresponds to the point at the same index in the points array.
message
a_r_capture_metadata.proto:394Used in:
optional float x = 1
optional float y = 2
optional float z = 3
message
a_r_capture_metadata.proto:8Info about the camera characteristics used to capture images and depth data.
Used in:
repeated float intrinsic_matrix = 1
3x3 row-major matrix relating a camera's internal properties to an ideal pinhole-camera model.
optional float intrinsic_matrix_reference_dimension_width = 2
The image dimensions to which the intrinsic_matrix values are relative.
optional float intrinsic_matrix_reference_dimension_height = 3
repeated float extrinsic_matrix = 4
3x4 row-major matrix relating a camera's position and orientation to a world or scene coordinate system. Consists of a unitless 3x3 rotation matrix (R) on the left and a translation (t) 3x1 vector on the right. The translation vector's units are millimeters. For example: |r1,1 r2,1 r3,1 | t1| [R | t] = |r1,2 r2,2 r3,2 | t2| |r1,3 r2,3 r3,3 | t3| is stored as [r11, r21, r31, t1, r12, r22, r32, t2, ...]
optional float pixel_size = 5
The size, in millimeters, of one image pixel.
repeated float lens_distortion_lookup_values = 6
A list of floating-point values describing radial distortions imparted by the camera lens, for use in rectifying camera images.
repeated float inverse_lens_distortion_lookup_values = 7
A list of floating-point values describing radial distortions for use in reapplying camera geometry to a rectified image.
optional float lens_distortion_center_x = 8
The offset of the distortion center of the camera lens from the top-left corner of the image.
optional float lens_distortion_center_y = 9
message
a_r_capture_metadata.proto:47Container for depth data information.
Used in:
optional bytes depth_data_map = 1
PNG representation of the grayscale depth data map. See discussion about depth_data_map_original_minimum_value, below, for information about how to interpret the pixel values.
optional string depth_data_type = 2
Pixel format type of the original captured depth data.
optional AVDepthData.Accuracy depth_data_accuracy = 3
optional bool depth_data_filtered = 4
Indicates whether the depth_data_map contains temporally smoothed data.
optional AVDepthData.Quality depth_data_quality = 5
optional camera_calibration_data = 6
Associated calibration data for the depth_data_map.
optional float depth_data_map_original_minimum_value = 7
The original range of values expressed by the depth_data_map, before grayscale normalization. For example, if the minimum and maximum values indicate a range of [0.5, 2.2], and the depth_data_type value indicates it was a depth map, then white pixels (255, 255, 255) will map to 0.5 and black pixels (0, 0, 0) will map to 2.2 with the grayscale range linearly interpolated inbetween. Conversely, if the depth_data_type value indicates it was a disparity map, then white pixels will map to 2.2 and black pixels will map to 0.5.
optional float depth_data_map_original_maximum_value = 8
optional int32 depth_data_map_width = 9
The width of the depth buffer map.
optional int32 depth_data_map_height = 10
The height of the depth buffer map.
optional bytes depth_data_map_raw_values = 11
The row-major flattened array of the depth buffer map pixels. This will be either a float32 or float16 byte array, depending on 'depth_data_type'.
Indicates the general accuracy of the depth_data_map.
Used in:
UNDEFINED_ACCURACY = 0
RELATIVE = 1
Values in the depth map are usable for foreground/background separation but are not absolutely accurate in the physical world.
ABSOLUTE = 2
Values in the depth map are absolutely accurate in the physical world.
Quality of the depth_data_map.
Used in:
UNDEFINED_QUALITY = 0
HIGH = 1
LOW = 2
message
annotation_data.proto:30Used in:
int32 id = 1
optional point_3d = 2
optional point_2d = 3
message
a_r_capture_metadata.proto:473A sample of raw accelerometer data.
Used in:
optional double timestamp = 1
The accelerometer data object timestamp. May differ from the frame timestamp value since the data may be collected at higher rate.
optional acceleration = 2
Raw acceleration measured by the accelerometer which effectively is a sum of gravity and user_acceleration of CMDeviceMotion object.
message
a_r_capture_metadata.proto:420Represents calibrated magnetic field data and accuracy estimate of it.
Used in:
optional field = 1
Vector of magnetic field estimate.
optional CMCalibratedMagneticField.CalibrationAccuracy calibration_accuracy = 2
Calibration accuracy of a magnetic field estimate.
Indicates the calibration accuracy of a magnetic field estimate.
Used in:
UNCALIBRATED = 0
LOW = 1
MEDIUM = 2
HIGH = 3
message
a_r_capture_metadata.proto:442A sample of device motion data. Encapsulates measurements of the attitude, rotation rate, magnetic field, and acceleration of the device. Core Media applies different algorithms of bias-reduction and stabilization to rotation rate, magnetic field and acceleration values. For raw values check correspondent fields in CMMotionManagerSnapshot object.
Used in:
optional double timestamp = 1
The device motion data object timestamp. May differ from the frame timestamp value since the data may be collected at higher rate.
optional attitude_quaternion = 2
The quaternion representing the device’s orientation relative to a known frame of reference at a point in time.
optional gravity = 3
The gravity acceleration vector expressed in the device's reference frame.
optional user_acceleration = 4
The acceleration that the user is giving to the device.
optional magnetic_field = 5
Returns the magnetic field vector filtered with respect to the device bias.
optional rotation_rate = 6
The rotation rate of the device adjusted by bias-removing Core Motion algorithms.
message
a_r_capture_metadata.proto:443Used in:
optional double x = 1
optional double y = 2
optional double z = 3
optional double w = 4
message
a_r_capture_metadata.proto:484A sample of raw gyroscope data.
Used in:
optional double timestamp = 1
The gyroscope data object timestamp. May differ from the frame timestamp value since the data may be collected at higher rate.
optional rotation_rate = 2
Raw rotation rate as measured by the gyroscope.
message
a_r_capture_metadata.proto:494A sample of raw magnetometer data.
Used in:
optional double timestamp = 1
The magnetometer data object timestamp. May differ from the frame timestamp value since the data may be collected at higher rate.
optional magnetic_field = 2
Raw magnetic field measured by the magnetometer.
message
a_r_capture_metadata.proto:504Contains most recent snapshots of device motion data
Used in:
repeated device_motion = 1
Most recent samples of device motion data.
repeated accelerometer_data = 2
Most recent samples of raw accelerometer data.
repeated gyro_data = 3
Most recent samples of raw gyroscope data.
repeated magnetometer_data = 4
Most recent samples of raw magnetometer data.
message
a_r_capture_metadata.proto:413A 3D vector
Used in: , , , ,
optional double x = 1
optional double y = 2
optional double z = 3
message
object.proto:82The edge connecting two keypoints together
Used in:
int32 source = 1
keypoint id of the edge's source
int32 sink = 2
keypoint id of the edge's sink
message
annotation_data.proto:50Used in:
int32 frame_id = 1
Unique frame id, corresponds to images.
repeated annotations = 2
List of the annotated objects in this frame. Depending on how many object are observable in this frame, we might have non or as much as sequence.objects_size() annotations.
optional camera = 3
Information about the camera transformation (in the world coordinate) and imaging characteristics for a captured video frame.
double timestamp = 4
The timestamp for the frame.
repeated float plane_center = 5
Plane center and normal in camera frame.
repeated float plane_normal = 6
message
object.proto:8Used in: ,
float x = 1
The position of the keypoint in the local coordinate system of the rigid object.
float y = 2
float z = 3
float confidence_radius = 4
Sphere around the keypoint, indicating annotator's confidence of the position in meters.
string name = 5
The name of the keypoint (e.g. legs, head, etc.). Does not have to be unique.
bool hidden = 6
Indicates whether the keypoint is hidden or not.
message
annotation_data.proto:12Projection of a 3D point on an image, and its metric depth.
Used in:
float x = 1
x-y position of the 2d keypoint in the image coordinate system. u,v \in [0, 1], where top left corner is (0, 0) and the bottom-right corner is (1, 1).
float y = 2
float depth = 3
The depth of the point in the camera coordinate system (in meters).
message
object.proto:27Used in:
int32 id = 1
Unique object id through a sequence. There might be multiple objects of the same label in this sequence.
string category = 2
Describes what category an object is. E.g. object class, attribute, instance or person identity. This provides additional context for the object type.
Object.Type type = 3
repeated float rotation = 4
3x3 row-major rotation matrix describing the orientation of the rigid object's frame of reference in the world-coordinate system.
repeated float translation = 5
3x1 vector describing the translation of the rigid object's frame of reference in the world-coordinate system in meters.
repeated float scale = 6
3x1 vector describing the scale of the rigid object's frame of reference in the world-coordinate system in meters.
repeated keypoints = 7
List of all the key points associated with this object in the object coordinate system. The first keypoint is always the object's frame of reference, e.g. the centroid of the box. E.g. bounding box with its center as frame of reference, the 9 keypoints : {0., 0., 0.}, {-.5, -.5, -.5}, {-.5, -.5, +.5}, {-.5, +.5, -.5}, {-.5, +.5, +.5}, {+.5, -.5, -.5}, {+.5, -.5, +.5}, {+.5, +.5, -.5}, {+.5, +.5, +.5} To get the bounding box in the world-coordinate system, we first scale the box then transform the scaled box. For example, bounding box in the world coordinate system is rotation * scale * keypoints + translation
Object.Method method = 8
Enum to reflect how this object is created.
Used in:
UNKNOWN_METHOD = 0
ANNOTATION = 1
Created by data annotation.
AUGMENTATION = 2
Created by data augmentation.
Used in:
UNDEFINED_TYPE = 0
BOUNDING_BOX = 1
SKELETON = 2
MESH = 3
message
annotation_data.proto:36Used in:
int32 object_id = 1
Reference to the object identifier in ObjectInstance.
repeated keypoints = 2
For each objects, list all the annotated keypoints here. E.g. for bounding-boxes, we have 8 keypoints, hands = 21 keypoints, etc. These normalized points are the projection of the Object's 3D keypoint on the current frame's camera poses.
float visibility = 3
Visibiity of this annotation in a frame.
message
annotation_data.proto:24The 3D point in the camera coordinate system, the scales are in meters.
Used in:
float x = 1
float y = 2
float z = 3
message
annotation_data.proto:72The sequence protocol contains the annotation data for the entire video clip.
repeated objects = 1
List of all the annotated 3D objects in this sequence in the world Coordinate system. Given the camera poses of each frame (also in the world-coordinate) these objects bounding boxes can be projected to each frame to get the per-frame annotation (i.e. image_annotation below).
repeated frame_annotations = 2
List of annotated data per each frame in sequence + frame information.
message
object.proto:92The skeleton template for different objects (e.g. humans, chairs, hands, etc) The annotation tool reads the skeleton template dictionary.
Used in:
int32 reference_keypoint = 1
The origin keypoint in the object coordinate system. (i.e. Point 0, 0, 0)
string category = 2
The skeleton's category (e.g. human, chair, hand.). Should be unique in the dictionary.
repeated keypoints = 3
Initialization value for all the keypoints in the skeleton in the object's local coordinate system. Pursuit will transform these points using object's transformation to get the keypoint in the world-coordinate.
repeated edges = 4
List of edges connecting keypoints
message
object.proto:112The list of all the modeled skeletons in our library. These models can be objects (chairs, desks, etc), humans (full pose, hands, faces, etc), or box. We can have multiple skeletons in the same file.
repeated object = 1