package android_env.control

Get desktop application:
View/edit binary Protocol Buffers messages

Represents an input action to be executed on the target Android device.

Used in: InjectActionRequest

• ActionType action_type = 1

  The type of action to perform. Always required.

• optional Point touch_position = 2

  The screen position for touch actions. REQUIRED if action_type is TOUCH_DOWN or TOUCH_MOVE. Omitted otherwise.

• optional int32 keycode = 3

  The Android keycode integer (e.g. 4 for BACK, 3 for HOME). REQUIRED if action_type is KEY_EVENT. Omitted otherwise.

• optional int32 pointer_id = 4

  Identifies the pointer (finger) for multi-touch support. Optional. Defaults to 0 (primary pointer) if unset. Value range: [0, 9].

• optional string text = 5

  Text to insert into target text field. REQUIRED if action_type is SET_TEXT. Omitted otherwise.

• optional int32 selection_start = 6

  Selection start index for text field cursor/highlight. REQUIRED if action_type is SET_SELECTION. Omitted otherwise.

• optional int32 selection_end = 7

  Selection end index for text field cursor/highlight. REQUIRED if action_type is SET_SELECTION. Omitted otherwise.

Enumerates the types of interactive input actions that can be injected.

Used in: Action

• ACTION_TYPE_UNSPECIFIED = 0

• ACTION_TYPE_TOUCH_DOWN = 1

  Pressing down on the screen at a specific coordinate.

• ACTION_TYPE_TOUCH_UP = 2

  Lifting up from the screen.

• ACTION_TYPE_TOUCH_MOVE = 3

  Dragging/moving a pointer across the screen.

• ACTION_TYPE_KEY_EVENT = 4

  Injecting a physical hardware keycode event (e.g. BACK, HOME).

• ACTION_TYPE_SET_TEXT = 5

  Native text insertion via Accessibility ACTION_SET_TEXT.

• ACTION_TYPE_SET_SELECTION = 6

  Setting text selection or cursor position via Accessibility ACTION_SET_SELECTION.

• ACTION_TYPE_CUT = 7

  Native clipboard cut via Accessibility ACTION_CUT.

• ACTION_TYPE_COPY = 8

  Native clipboard copy via Accessibility ACTION_COPY.

• ACTION_TYPE_PASTE = 9

  Native clipboard paste via Accessibility ACTION_PASTE.

A single block of raw audio data captured from the device.

Used in: StreamedSignal

• bytes raw_bytes = 1

  Raw PCM audio bytes.

• int32 sample_rate_hz = 2

  Audio sampling rate in Hertz. Value range: [8000, 48000] (typically 44100).

• int32 channel_count = 3

  Number of audio channels. Value range: [1, 2] (1 = mono, 2 = stereo).

• optional string sample_format = 4

  PCM encoding format string (e.g., "AUDIO_FORMAT_PCM_16BIT").

Telemetry metrics for device battery state, current draw, voltage, and calculated power.

Used in: StreamedSignal

• int32 level_percent = 1

  Battery charge level percentage [0-100].

• int64 current_now_ua = 2

  Instantaneous current draw in microamperes (uA). Positive for discharge, negative for charging.

• int32 voltage_mv = 3

  Voltage in millivolts (mV).

• double power_mw = 4

  Calculated instantaneous power in milliwatts (mW).

• float temperature_celsius = 5

  Battery temperature in degrees Celsius (°C).

• string status = 6

  Battery health/status string (e.g., "CHARGING", "DISCHARGING", "FULL").

• string plugged_source = 7

  Plugged power source ("AC", "USB", "WIRELESS", "UNPLUGGED").

• int64 charge_counter_uah = 8

  Total remaining charge counter in microampere-hours (uAh).

• int32 thermal_status = 9

  Current thermal status (0 = NONE to 6 = SHUTDOWN).

• int64 energy_counter_nwh = 11

  Total remaining battery energy in nanowatt-hours (nWh).

Top-level wrapper message sent from the Python Client to the Android Server.

• oneof payload

  • UpdateSubscriptionsRequest update_subscriptions = 1

    Configure which telemetry signals the server should stream/capture.

  • InjectActionRequest inject_action = 2

    Inject a physical user action.

  • GetStateRequest get_state = 3

    Query the current state of the device synchronously.

Represents raw decoded screenshot image pixel data.

Used in: ScreenshotValue

• bytes raw_pixels = 1

  Uncompressed raw RGB pixel bytes.

• int32 width = 2

  Frame width in pixels.

• int32 height = 3

  Frame height in pixels.

Enumerates all supported sensory streams and telemetry signals that can be observed from the Android device.

Used in: StreamedSignal, UpdateSubscriptionsRequest

• DEVICE_SIGNAL_UNSPECIFIED = 0

• DEVICE_SIGNAL_SCREENSHOT = 1

  A raw image of the device display (typically PNG or JPEG).

• DEVICE_SIGNAL_ACCESSIBILITY_TREE = 2

  The current UI accessibility hierarchy tree.

• DEVICE_SIGNAL_ACCESSIBILITY_EVENTS = 3

  Stream of UI accessibility events.

• DEVICE_SIGNAL_ACTIVE_PACKAGE = 4

  The package name of the foreground application (e.g. "com.android.chrome").

• DEVICE_SIGNAL_ACTIVE_ACTIVITY = 5

  The class name of the current foreground activity.

• DEVICE_SIGNAL_SYSTEM_LOGS = 6

  Recent logcat system logs.

• DEVICE_SIGNAL_RUNNING_PROCESSES = 7

  List of currently running process names.

• DEVICE_SIGNAL_DEVICE_SETTINGS = 8

  Key-value map of system settings (e.g. screen_brightness, volume).

• DEVICE_SIGNAL_SENSOR_ACCELEROMETER = 9

  Linear acceleration force along the X, Y, and Z axes (in m/s^2).

• DEVICE_SIGNAL_SENSOR_GYROSCOPE = 10

  Rate of rotation around the X, Y, and Z axes (in rad/s).

• DEVICE_SIGNAL_SENSOR_MAGNETOMETER = 11

  Ambient magnetic field along the X, Y, and Z axes (in micro-Tesla).

• DEVICE_SIGNAL_SENSOR_BAROMETER = 12

  Atmospheric pressure (in hectopascals / hPa).

• DEVICE_SIGNAL_SENSOR_AMBIENT_TEMP = 13

  Ambient room temperature (in degrees Celsius).

• DEVICE_SIGNAL_SENSOR_LIGHT = 14

  Ambient light level (in lux).

• DEVICE_SIGNAL_SENSOR_PROXIMITY = 15

  Proximity sensor distance (in centimeters).

• DEVICE_SIGNAL_SENSOR_HEART_RATE = 16

  Heart rate from wearable sensors (in beats per minute).

• DEVICE_SIGNAL_LOCATION_GPS = 17

  Current GPS coordinates (Latitude, Longitude, Altitude).

• DEVICE_SIGNAL_AUDIO_OUTPUT = 18

  Continuous audio frames captured from the device speaker output.

• DEVICE_SIGNAL_BATTERY_POWER = 19

  Battery and power consumption metrics (current, voltage, power, temp, status).

Represents the aggregated state of all subscribed signals on the device.

Used in: ServerMessage

• repeated StreamedSignal signals = 1

  List of active signal observations currently subscribed.

• int64 snapshot_timestamp_us = 2

  Wall-clock time on the device when this state snapshot was assembled (microseconds since epoch).

• int32 orientation = 3

  Current screen rotation (0: ROTATION_0, 1: ROTATION_90, 2: ROTATION_180, 3: ROTATION_270).

Details of an error encountered on the server.

Used in: ServerMessage

• string message = 1

Wrapper for an array of floats (e.g. sensor readings).

Used in: StreamedSignal

• repeated float values = 1

Empty request to trigger a manual state fetch.

Used in: ClientMessage

(message has no fields)

GPS geolocation coordinates.

Used in: StreamedSignal

• double latitude = 1

  Latitude in degrees. Value range: [-90.0, 90.0].

• double longitude = 2

  Longitude in degrees. Value range: [-180.0, 180.0].

• double altitude = 3

  Altitude in meters above the WGS 84 reference ellipsoid.

Request to inject an input action.

Used in: ClientMessage

• optional Action action = 1

Represents a 2D coordinate point on the screen.

Used in: Action

• float x = 1

  X-coordinate normalized between 0.0 (left edge) and 1.0 (right edge).

• float y = 2

  Y-coordinate normalized between 0.0 (top edge) and 1.0 (bottom edge).

Wraps screenshot image bytes. Exactly one representation format is set.

Used in: StreamedSignal

• oneof format

  • bytes encoded_bytes = 1

    Raw compressed image bytes (typically JPEG or PNG format).

  • DecodedScreenshot decoded = 2

    Uncompressed raw RGB pixel payload.

Top-level wrapper message sent from the Android Server back to the Client.

• oneof payload

  • DeviceState state = 1

    Contains the requested device state telemetry.

  • ErrorResponse error = 2

    Error details if an operation failed on the server.

Represents a single streamed telemetry signal packaging metadata and the value.

Used in: DeviceState

• DeviceSignal type = 1

  The type of signal this message represents.

• optional StreamedSignalMetadata metadata = 2

  Metadata for this signal capture.

• oneof value

  The actual signal value.

  • ScreenshotValue screenshot = 3

  • AndroidAccessibilityForest accessibility_forest = 4

  • EventRequestList accessibility_events = 5

  • string active_package = 6

  • string active_activity = 7

  • StringList system_logs = 8

  • StringList running_processes = 9

  • StringMap device_settings = 10

  • FloatArray accelerometer = 11

  • FloatArray gyroscope = 12

  • FloatArray magnetometer = 13

  • float barometer = 14

  • float ambient_temp = 15

  • float light_level = 16

  • float proximity = 17

  • GpsCoordinates location_gps = 18

  • float heart_rate = 19

  • AudioFrame audio_output = 20

  • BatteryPowerMetrics battery_power = 21

Metadata accompanying a streamed signal value, enabling synchronization and latency tracking.

Used in: StreamedSignal

• int64 timestamp_us = 1

  The device wall-clock time when the signal was captured (microseconds since epoch).

• int64 sequence_id = 2

  Monotonically increasing sequence identifier to detect frame drops or out-of-order delivery.

Wrapper for a list of strings (e.g. log lines).

Used in: StreamedSignal

• repeated string values = 1

Wrapper for key-value maps (e.g. system settings).

Used in: StreamedSignal

• map<string, string> entries = 1

Request to modify the list of active telemetry subscriptions.

Used in: ClientMessage

• repeated DeviceSignal active_signals = 1