English · 简体中文
AndroidEnv is a Python and Kotlin platform for interacting with Android devices (both virtualized emulators and physical hardware). Built for foundation model UI agents, automated testing, and Reinforcement Learning (RL) research, AndroidEnv provides low-latency device telemetry and control over any Android application.
In AndroidEnv v2, device interaction is decoupled from Reinforcement Learning task definitions:
AndroidDevice (Direct Device Control & Telemetry): A task-free handle that connects to an on-device Kotlin Core server over WebSockets to stream screenshots, accessibility trees, system logs, and sensor data while injecting touch gestures, key events, text, and ADB commands.AndroidEnv & TaskWrapper (Reinforcement Learning): An optional dm_env layer on top of AndroidDevice that adds Markov Decision Process (MDP) semantics (step(), reset(), rewards, and episode terminations) via task definitions.stdio Model Context Protocol (MCP) server (android-env-mcp) for LLM agents.DeviceState): Subscribe on demand to the signals your agent needs:
width, height, raw_pixels).AndroidAccessibilityForest window and UI node hierarchies streamed directly from the on-device AccessibilityService.Action):
(x, y) coordinates in [0.0, 1.0] with TOUCH_DOWN, TOUCH_MOVE, and TOUCH_UP for taps, swipes, scrolls, and drags.BACK, HOME, ENTER), set text directly on focused UI nodes (ACTION_TYPE_SET_TEXT), or trigger clipboard actions (CUT, COPY, PASTE).AdbRequest commands via device.execute_adb_call().AndroidDevice with android_env.rl.environment.AndroidEnv and TaskWrapper to use custom .textproto tasks, reward parsers, and dm_env / Gymnasium wrappers.Install AndroidEnv from PyPI:
$ python3 -m pip install android-env
Or clone the repository from main and build/test with Bazel (ensure ANDROID_HOME points to a local Android SDK installation):
$ git clone https://github.com/google-deepmind/android_env/
$ cd android_env
$ bazel test //...
To build the PyPI wheel locally using Bazel:
$ bazel build //:android_env_wheel
Connect a physical Android device via USB debugging (adb devices) or create an Android Virtual Device (AVD) using Android Studio. See the Emulator guide for step-by-step instructions.
AndroidDeviceFor UI agents, foundation model evaluations, and automation scripts that do not need RL episode loops:
from android_env.components import android_device_loader
from android_env.components import config_classes
from android_env.proto import device_env_service_pb2
config = config_classes.AndroidDeviceConfig(serial='emulator-5554')
with android_device_loader.load_android_device(config) as device:
# Subscribe to screenshots and accessibility trees:
device.update_subscriptions({
device_env_service_pb2.DEVICE_SIGNAL_SCREENSHOT,
device_env_service_pb2.DEVICE_SIGNAL_ACCESSIBILITY_TREE,
})
# Read the latest device state:
state = device.get_state()
# Tap the center of the screen:
device.send_action(
device_env_service_pb2.Action(
action_type=device_env_service_pb2.ACTION_TYPE_TOUCH_DOWN,
touch_position=device_env_service_pb2.Point(x=0.5, y=0.5),
)
)
device.send_action(
device_env_service_pb2.Action(
action_type=device_env_service_pb2.ACTION_TYPE_TOUCH_UP,
)
)
Explore the device screen, inspect live accessibility trees, and run ADB commands from your browser (after pip install "android-env[playground]"):
$ android-env-playground --serial=emulator-5554
Or launch a local AVD from a source checkout with Bazel:
$ bazel run //android_env/playground:playground -- \
--config=android_env/playground/configs/emulator_config.py
To expose the device as a stdio Model Context Protocol (MCP) server for LLM clients (after pip install "android-env[mcp]"):
$ android-env-mcp --serial=emulator-5554
dm_env)To define MDP tasks with rewards and episode resets (after pip install "android-env[rl]"), see the Tasks guide, Example tasks, and detailed instructions.
This library is developed and maintained by Google DeepMind. You can find the original technical report on arXiv, as well as the introductory blog post.
If you use AndroidEnv in your research, you can cite the paper using the following BibTeX:
@article{ToyamaEtAl2021AndroidEnv,
title = "{AndroidEnv}: A Reinforcement Learning Platform for Android",
author = {Daniel Toyama and Philippe Hamel and Anita Gergely and
Gheorghe Comanici and Amelia Glaese and Zafarali Ahmed and Tyler
Jackson and Shibl Mourad and Doina Precup},
year = {2021},
eprint = {2105.13231},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
volume = {abs/2105.13231},
url = {http://arxiv.org/abs/2105.13231},
}
Disclaimer: This is not an official Google product.