package research.compvideo.arcapture

Get desktop application:
View/edit binary Protocol Buffers messages

Contains a list of blend shape entries wherein each item maps a specific blend shape location to its associated coefficient.

Used in: ARFaceAnchor

• repeated ARBlendShapeMap.MapEntry entries = 1

Used in: ARBlendShapeMap

• optional string blend_shape_location = 1

  Identifier for the specific facial feature.

• optional float blend_shape_coefficient = 2

  Indicates the current position of the feature relative to its neutral configuration, ranging from 0.0 (neutral) to 1.0 (maximum movement).

Information about the camera position and imaging characteristics for a captured video frame.

Used in: ARFrame, xeno.pursuit.proto.FrameAnnotation

• optional ARCamera.TrackingState tracking_state = 1

• optional ARCamera.TrackingStateReason tracking_state_reason = 2

• repeated float transform = 3

  4x4 row-major matrix expressing position and orientation of the camera in world coordinate space.

• optional ARCamera.EulerAngles euler_angles = 4

• optional int32 image_resolution_width = 5

  The width and height, in pixels, of the captured camera image.

• optional int32 image_resolution_height = 6

• repeated float intrinsics = 7

  3x3 row-major matrix that converts between the 2D camera plane and 3D world coordinate space.

• repeated float projection_matrix = 8

  4x4 row-major transform matrix appropriate for rendering 3D content to match the image captured by the camera.

• repeated float view_matrix = 9

  4x4 row-major transform matrix appropriate for converting from world-space to camera space. Relativized for the captured_image orientation (i.e. UILandscapeOrientationRight).

The orientation of the camera, expressed as roll, pitch, and yaw values.

Used in: ARCamera

• optional float roll = 1

• optional float pitch = 2

• optional float yaw = 3

The general quality of position tracking available when the camera captured a frame.

Used in: ARCamera

• UNDEFINED_TRACKING_STATE = 0

• UNAVAILABLE = 1

  Camera position tracking is not available.

• LIMITED = 2

  Tracking is available, but the quality of results is questionable.

• NORMAL = 3

  Camera position tracking is providing optimal results.

A possible diagnosis for limited position tracking quality as of when the frame was captured.

Used in: ARCamera

• UNDEFINED_TRACKING_STATE_REASON = 0

• NONE = 1

  The current tracking state is not limited.

• INITIALIZING = 2

  Not yet enough camera or motion data to provide tracking information.

• EXCESSIVE_MOTION = 3

  The device is moving too fast for accurate image-based position tracking.

• INSUFFICIENT_FEATURES = 4

  Not enough distinguishable features for image-based position tracking.

• RELOCALIZING = 5

  Tracking is limited due to a relocalization in progress.

Information about the pose, topology, and expression of a detected face.

Used in: ARFrame

• optional ARFaceGeometry geometry = 1

  A coarse triangle mesh representing the topology of the detected face.

• optional ARBlendShapeMap blend_shapes = 2

  A map of named coefficients representing the detected facial expression in terms of the movement of specific facial features.

• repeated float transform = 3

  4x4 row-major matrix encoding the position, orientation, and scale of the anchor relative to the world coordinate space.

• optional bool is_tracked = 4

  Indicates whether the anchor's transform is valid. Frames that have a face anchor with this value set to NO should probably be ignored.

Container for a 3D mesh describing face topology.

Used in: ARFaceAnchor

• repeated ARFaceGeometry.Vertex vertices = 1

• optional int32 vertex_count = 2

  The number of elements in the vertices list.

• repeated ARFaceGeometry.TextureCoordinate texture_coordinates = 3

• optional int32 texture_coordinate_count = 4

  The number of elements in the texture_coordinates list.

• repeated int32 triangle_indices = 5

  Each integer value in this ordered list represents an index into the vertices and texture_coordinates lists. Each set of three indices identifies the vertices comprising a single triangle in the mesh. Each set of three indices forms a triangle, so the number of indices in the triangle_indices buffer is three times the triangle_count value.

• optional int32 triangle_count = 6

  The number of triangles described by the triangle_indices buffer.

Each texture coordinate represents UV texture coordinates for the vertex at the corresponding index in the vertices buffer.

Used in: ARFaceGeometry

• optional float u = 1

• optional float v = 2

Each vertex represents a 3D point in the face mesh, in the face coordinate space.

Used in: ARFaceGeometry

• optional float x = 1

• optional float y = 2

• optional float z = 3

Video image and face position tracking information.

• optional double timestamp = 1

  The timestamp for the frame.

• optional AVDepthData depth_data = 2

  The depth data associated with the frame. Not all frames have depth data.

• optional double depth_data_timestamp = 3

  The depth data object timestamp associated with the frame. May differ from the frame timestamp value. Is only set when the frame has depth_data.

• optional ARCamera camera = 4

  Camera information associated with the frame.

• optional ARLightEstimate light_estimate = 5

  Light information associated with the frame.

• optional ARFaceAnchor face_anchor = 6

  Face anchor information associated with the frame. Not all frames have an active face anchor.

• repeated ARPlaneAnchor plane_anchor = 7

  Plane anchors associated with the frame. Not all frames have a plane anchor. Plane anchors and face anchors are mutually exclusive.

• optional ARPointCloud raw_feature_points = 8

  The current intermediate results of the scene analysis used to perform world tracking.

• optional CMMotionManagerSnapshot motion_manager_snapshot = 9

  Snapshot of Core Motion CMMotionManager object containing most recent motion data associated with the frame. Since motion data capture rates can be higher than rates of AR capture, the entities of this object reflect all of the aggregated events which have occurred since the last ARFrame was recorded.

Estimated scene lighting information associated with a captured video frame.

Used in: ARFrame

• optional double ambient_intensity = 1

  The estimated intensity, in lumens, of ambient light throughout the scene.

• optional double ambient_color_temperature = 2

  The estimated color temperature, in degrees Kelvin, of ambient light throughout the scene.

• repeated float spherical_harmonics_coefficients = 3

  Data describing the estimated lighting environment in all directions. Second-level spherical harmonics in separate red, green, and blue data planes. Thus, this buffer contains 3 sets of 9 coefficients, or a total of 27 values.

• optional ARLightEstimate.DirectionVector primary_light_direction = 4

  A vector indicating the orientation of the strongest directional light source, normalized in the world-coordinate space.

• optional float primary_light_intensity = 5

  The estimated intensity, in lumens, of the strongest directional light source in the scene.

Used in: ARLightEstimate

• optional float x = 1

• optional float y = 2

• optional float z = 3

A subdividision of the reconstructed, real-world scene surrounding the user.

Used in: ARMeshData

• optional string identifier = 1

  The ID of the mesh.

• repeated float transform = 2

  4x4 row-major matrix encoding the position, orientation, and scale of the anchor relative to the world coordinate space.

• optional ARMeshGeometry geometry = 3

  3D information about the mesh such as its shape and classifications.

Container object for mesh data of real-world scene surrounding the user. Even though each ARFrame may have a set of ARMeshAnchors associated with it, only a single frame's worth of mesh data is written separately at the end of each recording due to concerns regarding latency and memory bloat.

• optional double timestamp = 1

  The timestamp for the data.

• repeated ARMeshAnchor mesh_anchor = 2

  Set of mesh anchors containing the mesh data.

Mesh geometry data stored in an array-based format.

Used in: ARMeshAnchor

• repeated ARMeshGeometry.Vertex vertices = 1

  The vertices of the mesh.

• repeated ARMeshGeometry.Face faces = 2

  The faces of the mesh.

• repeated ARMeshGeometry.Vertex normals = 3

  Rays that define which direction is outside for each face. Normals contain 'rays that define which direction is outside for each face', in practice the normals count is always identical to vertices count which looks like vertices normals and not faces normals.

• repeated ARMeshGeometry.MeshClassification classification = 4

  Classification for each face in the mesh.

Used in: ARMeshGeometry

• repeated int32 vertex_indices = 1

  / Indices of vertices defining the face from correspondent array of parent / message. A typical face is triangular.

Type of objects

Used in: ARMeshGeometry

• NONE = 0

• WALL = 1

• FLOOR = 2

• CEILING = 3

• TABLE = 4

• SEAT = 5

• WINDOW = 6

• DOOR = 7

Used in: ARMeshGeometry

• optional float x = 1

• optional float y = 2

• optional float z = 3

Information about the position and orientation of a real-world flat surface.

Used in: ARFrame

• optional string identifier = 1

  The ID of the plane.

• repeated float transform = 2

  4x4 row-major matrix encoding the position, orientation, and scale of the anchor relative to the world coordinate space.

• optional ARPlaneAnchor.Alignment alignment = 3

  The general orientation of the detected plane with respect to gravity.

• optional ARPlaneGeometry geometry = 4

  A coarse triangle mesh representing the general shape of the detected plane.

• optional ARPlaneAnchor.PlaneVector center = 5

  The center point of the plane relative to its anchor position. Although the type of this property is a 3D vector, a plane anchor is always two-dimensional, and is always positioned in only the x and z directions relative to its transform position. (That is, the y-component of this vector is always zero.)

• optional ARPlaneAnchor.PlaneVector extent = 6

  The estimated width and length of the detected plane.

• optional bool classification_supported = 7

  A Boolean value that indicates whether plane classification is available on the current device. On devices without plane classification support, all plane anchors report a classification value of NONE and a classification_status value of UNAVAILABLE.

• optional ARPlaneAnchor.PlaneClassification classification = 8

  A general characterization of what kind of real-world surface the plane anchor represents.

• optional ARPlaneAnchor.PlaneClassificationStatus classification_status = 9

  The current state of process for classifying the plane anchor. When this property's value is KNOWN, the classification property represents characterization of the real-world surface corresponding to the plane anchor.

Used in: ARPlaneAnchor

• UNDEFINED = 0

• HORIZONTAL = 1

  The plane is perpendicular to gravity.

• VERTICAL = 2

  The plane is parallel to gravity.

Used in: ARPlaneAnchor

• NONE = 0

• WALL = 1

• FLOOR = 2

• CEILING = 3

• TABLE = 4

• SEAT = 5

The classification status for the plane.

Used in: ARPlaneAnchor

• UNKNOWN = 0

  The classfication process for the plane anchor has completed but the result is inconclusive.

• UNAVAILABLE = 1

  No classication information can be provided (set on error or if the device does not support plane classification).

• UNDETERMINED = 2

  The classification process has not completed.

• KNOWN = 3

  The classfication process for the plane anchor has completed.

Wrapper for a 3D point / vector within the plane. See extent and center values for more information.

Used in: ARPlaneAnchor

• optional float x = 1

• optional float y = 2

• optional float z = 3

Container for a 3D mesh.

Used in: ARPlaneAnchor

• repeated ARPlaneGeometry.Vertex vertices = 1

  A buffer of vertex positions for each point in the plane mesh.

• optional int32 vertex_count = 2

  The number of elements in the vertices buffer.

• repeated ARPlaneGeometry.TextureCoordinate texture_coordinates = 3

  A buffer of texture coordinate values for each point in the plane mesh.

• optional int32 texture_coordinate_count = 4

  The number of elements in the texture_coordinates buffer.

• repeated int32 triangle_indices = 5

  Each integer value in this ordered list represents an index into the vertices and texture_coordinates lists. Each set of three indices identifies the vertices comprising a single triangle in the mesh. Each set of three indices forms a triangle, so the number of indices in the triangle_indices buffer is three times the triangle_count value.

• optional int32 triangle_count = 6

  Each set of three indices forms a triangle, so the number of indices in the triangle_indices buffer is three times the triangle_count value.

• repeated ARPlaneGeometry.Vertex boundary_vertices = 7

  Each value in this buffer represents the position of a vertex along the boundary polygon of the estimated plane. The owning plane anchor's transform matrix defines the coordinate system for these points.

• optional int32 boundary_vertex_count = 8

  The number of elements in the boundary_vertices buffer.

Each texture coordinate represents UV texture coordinates for the vertex at the corresponding index in the vertices buffer.

Used in: ARPlaneGeometry

• optional float u = 1

• optional float v = 2

Used in: ARPlaneGeometry

• optional float x = 1

• optional float y = 2

• optional float z = 3

A collection of points in the world coordinate space.

Used in: ARFrame

• optional int32 count = 1

  The number of points in the cloud.

• repeated ARPointCloud.Point point = 2

  The list of detected points.

• repeated int64 identifier = 3

  A list of unique identifiers corresponding to detected feature points. Each identifier in this list corresponds to the point at the same index in the points array.

Used in: ARPointCloud

• optional float x = 1

• optional float y = 2

• optional float z = 3

Info about the camera characteristics used to capture images and depth data.

Used in: AVDepthData

• repeated float intrinsic_matrix = 1

  3x3 row-major matrix relating a camera's internal properties to an ideal pinhole-camera model.

• optional float intrinsic_matrix_reference_dimension_width = 2

  The image dimensions to which the intrinsic_matrix values are relative.

• optional float intrinsic_matrix_reference_dimension_height = 3

• repeated float extrinsic_matrix = 4

  3x4 row-major matrix relating a camera's position and orientation to a world or scene coordinate system. Consists of a unitless 3x3 rotation matrix (R) on the left and a translation (t) 3x1 vector on the right. The translation vector's units are millimeters. For example: |r1,1 r2,1 r3,1 | t1| [R | t] = |r1,2 r2,2 r3,2 | t2| |r1,3 r2,3 r3,3 | t3| is stored as [r11, r21, r31, t1, r12, r22, r32, t2, ...]

• optional float pixel_size = 5

  The size, in millimeters, of one image pixel.

• repeated float lens_distortion_lookup_values = 6

  A list of floating-point values describing radial distortions imparted by the camera lens, for use in rectifying camera images.

• repeated float inverse_lens_distortion_lookup_values = 7

  A list of floating-point values describing radial distortions for use in reapplying camera geometry to a rectified image.

• optional float lens_distortion_center_x = 8

  The offset of the distortion center of the camera lens from the top-left corner of the image.

• optional float lens_distortion_center_y = 9

Container for depth data information.

Used in: ARFrame

• optional bytes depth_data_map = 1

  PNG representation of the grayscale depth data map. See discussion about depth_data_map_original_minimum_value, below, for information about how to interpret the pixel values.

• optional string depth_data_type = 2

  Pixel format type of the original captured depth data.

• optional AVDepthData.Accuracy depth_data_accuracy = 3

• optional bool depth_data_filtered = 4

  Indicates whether the depth_data_map contains temporally smoothed data.

• optional AVDepthData.Quality depth_data_quality = 5

• optional AVCameraCalibrationData camera_calibration_data = 6

  Associated calibration data for the depth_data_map.

• optional float depth_data_map_original_minimum_value = 7

  The original range of values expressed by the depth_data_map, before grayscale normalization. For example, if the minimum and maximum values indicate a range of [0.5, 2.2], and the depth_data_type value indicates it was a depth map, then white pixels (255, 255, 255) will map to 0.5 and black pixels (0, 0, 0) will map to 2.2 with the grayscale range linearly interpolated inbetween. Conversely, if the depth_data_type value indicates it was a disparity map, then white pixels will map to 2.2 and black pixels will map to 0.5.

• optional float depth_data_map_original_maximum_value = 8

• optional int32 depth_data_map_width = 9

  The width of the depth buffer map.

• optional int32 depth_data_map_height = 10

  The height of the depth buffer map.

• optional bytes depth_data_map_raw_values = 11

  The row-major flattened array of the depth buffer map pixels. This will be either a float32 or float16 byte array, depending on 'depth_data_type'.

Indicates the general accuracy of the depth_data_map.

Used in: AVDepthData

• UNDEFINED_ACCURACY = 0

• RELATIVE = 1

  Values in the depth map are usable for foreground/background separation but are not absolutely accurate in the physical world.

• ABSOLUTE = 2

  Values in the depth map are absolutely accurate in the physical world.

Quality of the depth_data_map.

Used in: AVDepthData

• UNDEFINED_QUALITY = 0

• HIGH = 1

• LOW = 2

A sample of raw accelerometer data.

Used in: CMMotionManagerSnapshot

• optional double timestamp = 1

  The accelerometer data object timestamp. May differ from the frame timestamp value since the data may be collected at higher rate.

• optional CMVector acceleration = 2

  Raw acceleration measured by the accelerometer which effectively is a sum of gravity and user_acceleration of CMDeviceMotion object.

Represents calibrated magnetic field data and accuracy estimate of it.

Used in: CMDeviceMotion

• optional CMVector field = 1

  Vector of magnetic field estimate.

• optional CMCalibratedMagneticField.CalibrationAccuracy calibration_accuracy = 2

  Calibration accuracy of a magnetic field estimate.

Indicates the calibration accuracy of a magnetic field estimate.

Used in: CMCalibratedMagneticField

• UNCALIBRATED = 0

• LOW = 1

• MEDIUM = 2

• HIGH = 3

A sample of device motion data. Encapsulates measurements of the attitude, rotation rate, magnetic field, and acceleration of the device. Core Media applies different algorithms of bias-reduction and stabilization to rotation rate, magnetic field and acceleration values. For raw values check correspondent fields in CMMotionManagerSnapshot object.

Used in: CMMotionManagerSnapshot

• optional double timestamp = 1

  The device motion data object timestamp. May differ from the frame timestamp value since the data may be collected at higher rate.

• optional CMDeviceMotion.Quaternion attitude_quaternion = 2

  The quaternion representing the device’s orientation relative to a known frame of reference at a point in time.

• optional CMVector gravity = 3

  The gravity acceleration vector expressed in the device's reference frame.

• optional CMVector user_acceleration = 4

  The acceleration that the user is giving to the device.

• optional CMCalibratedMagneticField magnetic_field = 5

  Returns the magnetic field vector filtered with respect to the device bias.

• optional CMVector rotation_rate = 6

  The rotation rate of the device adjusted by bias-removing Core Motion algoriths.

Used in: CMDeviceMotion

• optional double x = 1

• optional double y = 2

• optional double z = 3

• optional double w = 4

A sample of raw gyroscope data.

Used in: CMMotionManagerSnapshot

• optional double timestamp = 1

  The gyroscope data object timestamp. May differ from the frame timestamp value since the data may be collected at higher rate.

• optional CMVector rotation_rate = 2

  Raw rotation rate as measured by the gyroscope.

A sample of raw magnetometer data.

Used in: CMMotionManagerSnapshot

• optional double timestamp = 1

  The magnetometer data object timestamp. May differ from the frame timestamp value since the data may be collected at higher rate.

• optional CMVector magnetic_field = 2

  Raw magnetic field measured by the magnetometer.

Contains most recent snapshots of device motion data

Used in: ARFrame

• repeated CMDeviceMotion device_motion = 1

  Most recent samples of device motion data.

• repeated CMAccelerometerData accelerometer_data = 2

  Most recent samples of raw accelerometer data.

• repeated CMGyroData gyro_data = 3

  Most recent samples of raw gyroscope data.

• repeated CMMagnetometerData magnetometer_data = 4

  Most recent samples of raw magnetometer data.

A 3D vector

Used in: CMAccelerometerData, CMCalibratedMagneticField, CMDeviceMotion, CMGyroData, CMMagnetometerData

• optional double x = 1

• optional double y = 2

• optional double z = 3