Our Approach to AndroidWorld: 100% task success with an open-source model

100% on AndroidWorld: a bar reaching the top, labeled Qwen3.5 27B, across all 116 AndroidWorld tasks

Overview

We reached a 100% success rate on AndroidWorld: all 116 tasks passed. The agent runs on Qwen3.5 27B, an open-source model, trained with our own reinforcement learning technique.

AndroidWorld has 116 tasks across 20 real Android apps, from calendar and SMS to recipes, maps and music. Each task pairs a plain-language goal with a checker that inspects the device at the end. Task parameters come from a random seed, so the same task asks for different names, dates or items each run.

Our approach uses AndroidWorld's own checker as the training signal. At each step the model reads the screen as a screenshot plus its accessibility tree and returns one action: tap, type, scroll, open an app. We run it on a live Android emulator and score every episode with the task's checker: 1 for pass, 0 for fail.

That score drives a reinforcement learning loop over the benchmark. We run the tasks, send every failure back into training for more rollouts, and repeat. The failing set shrinks each round, and the final run passed all 116 tasks in 1,118 steps.

Approach

We train on the tasks the model fails, using the checker's pass/fail verdict as the only reward.

The model. We start from Qwen3.5 27B, an open-source model, and train it with our technique. It sees each screen as a screenshot plus its accessibility tree and returns one action per step.

The problem is long tasks. Short tasks are mostly solved already. Failures cluster on long ones: building a playlist from specific tracks, or copying several items from one app into another. These need a dozen or more correct actions in a row, with no feedback along the way. A mistake at step 5 only shows up at the end, when the task is already lost.

The signal is the checker. Every AndroidWorld task ships with a checker that inspects device state after the episode. That one verdict is enough to learn from.

The loop. Each failing task goes through the cycle below until it passes.

Training loop: the 116 AndroidWorld tasks go into training, where reinforcement learning sends attempts to an Android emulator and gets results back. Each task either passes or is still failing, and failing tasks go back for another rollout.

Figure 1. Failing tasks go into training, where the model acts on an Android emulator and the checker scores each attempt. Tasks that pass are done; tasks that still fail go round again.

The model attempts the task; the emulator executes each action on Android.
At the end, the checker returns 1 (pass) or 0 (fail). That is the reward.
Reinforcement learning updates the model on those rewards.
Passed tasks leave the set. Failed tasks go round again for another rollout.

The failing set shrinks each round. Run to completion, it empties.

Results

Qwen3.5 27B, trained with our technique, passed all 116 AndroidWorld tasks: a 100% success rate.

The run took 1,118 steps across all tasks. The median task finished in 7 steps and the mean was 9.6. The shortest took 2 steps and the longest, OsmAndTrack, took 52.

Steps per task	Tasks
1–5	47
6–10	36
11–20	22
21–40	10
41–60	1

The hardest tasks chain many actions across apps or items. OsmAndTrack took 52 steps, VlcCreateTwoPlaylists 37, and the four multi-recipe tasks 33 to 35. All of them pass.

What's next

We will open-source our models and the training technique behind this result.

The release will show that the 100% is not tied to one model or one seed. The technique transfers to other open-source models, including smaller ones, which perform very well after training. Those models generalize across seeds and, to a certain extent, across tasks, rather than overfitting to a single seed.

Appendix: per-task results

Qwen3.5 27B passed all 116 AndroidWorld tasks on the final run, one episode per task, 1,118 steps in total. A step is one action the model returned.

Task	Result	Steps
AudioRecorderRecordAudio	Pass	7
AudioRecorderRecordAudioWithFileName	Pass	8
BrowserDraw	Pass	16
BrowserMaze	Pass	10
BrowserMultiply	Pass	13
CameraTakePhoto	Pass	3
CameraTakeVideo	Pass	4
ClockStopWatchPausedVerify	Pass	3
ClockStopWatchRunning	Pass	4
ClockTimerEntry	Pass	4
ContactsAddContact	Pass	8
ContactsNewContactDraft	Pass	9
ExpenseAddMultiple	Pass	24
ExpenseAddMultipleFromGallery	Pass	26
ExpenseAddMultipleFromMarkor	Pass	18
ExpenseAddSingle	Pass	9
ExpenseDeleteDuplicates	Pass	8
ExpenseDeleteDuplicates2	Pass	9
ExpenseDeleteMultiple	Pass	11
ExpenseDeleteMultiple2	Pass	14
ExpenseDeleteSingle	Pass	5
FilesDeleteFile	Pass	10
FilesMoveFile	Pass	14
MarkorAddNoteHeader	Pass	12
MarkorChangeNoteContent	Pass	10
MarkorCreateFolder	Pass	5
MarkorCreateNote	Pass	7
MarkorCreateNoteAndSms	Pass	14
MarkorCreateNoteFromClipboard	Pass	9
MarkorDeleteAllNotes	Pass	7
MarkorDeleteNewestNote	Pass	5
MarkorDeleteNote	Pass	5
MarkorEditNote	Pass	6
MarkorMergeNotes	Pass	16
MarkorMoveNote	Pass	10
MarkorTranscribeReceipt	Pass	9
MarkorTranscribeVideo	Pass	18
NotesIsTodo	Pass	2
NotesMeetingAttendeeCount	Pass	6
NotesRecipeIngredientCount	Pass	6
NotesTodoItemCount	Pass	4
OpenAppTaskEval	Pass	2
OsmAndFavorite	Pass	7
OsmAndMarker	Pass	9
OsmAndTrack	Pass	52
RecipeAddMultipleRecipes	Pass	33
RecipeAddMultipleRecipesFromImage	Pass	35
RecipeAddMultipleRecipesFromMarkor	Pass	35
RecipeAddMultipleRecipesFromMarkor2	Pass	33
RecipeAddSingleRecipe	Pass	11
RecipeDeleteDuplicateRecipes	Pass	7
RecipeDeleteDuplicateRecipes2	Pass	10
RecipeDeleteDuplicateRecipes3	Pass	21
RecipeDeleteMultipleRecipes	Pass	10
RecipeDeleteMultipleRecipesWithConstraint	Pass	6
RecipeDeleteMultipleRecipesWithNoise	Pass	12
RecipeDeleteSingleRecipe	Pass	6
RecipeDeleteSingleWithRecipeWithNoise	Pass	7
RetroCreatePlaylist	Pass	18
RetroPlayingQueue	Pass	20
RetroPlaylistDuration	Pass	23
RetroSavePlaylist	Pass	22
SaveCopyOfReceiptTaskEval	Pass	10
SimpleCalendarAddOneEvent	Pass	17
SimpleCalendarAddOneEventInTwoWeeks	Pass	18
SimpleCalendarAddOneEventRelativeDay	Pass	16
SimpleCalendarAddOneEventTomorrow	Pass	17
SimpleCalendarAddRepeatingEvent	Pass	18
SimpleCalendarAnyEventsOnDate	Pass	3
SimpleCalendarDeleteEvents	Pass	12
SimpleCalendarDeleteEventsOnRelativeDay	Pass	9
SimpleCalendarDeleteOneEvent	Pass	6
SimpleCalendarEventOnDateAtTime	Pass	3
SimpleCalendarEventsInNextWeek	Pass	5
SimpleCalendarEventsInTimeRange	Pass	3
SimpleCalendarEventsOnDate	Pass	3
SimpleCalendarFirstEventAfterStartTime	Pass	3
SimpleCalendarLocationOfEvent	Pass	3
SimpleCalendarNextEvent	Pass	3
SimpleCalendarNextMeetingWithPerson	Pass	4
SimpleDrawProCreateDrawing	Pass	6
SimpleSmsReply	Pass	5
SimpleSmsReplyMostRecent	Pass	5
SimpleSmsResend	Pass	5
SimpleSmsSend	Pass	7
SimpleSmsSendClipboardContent	Pass	7
SimpleSmsSendReceivedAddress	Pass	6
SportsTrackerActivitiesCountForWeek	Pass	3
SportsTrackerActivitiesOnDate	Pass	12
SportsTrackerActivityDuration	Pass	2
SportsTrackerLongestDistanceActivity	Pass	3
SportsTrackerTotalDistanceForCategoryOverInterval	Pass	5
SportsTrackerTotalDurationForCategoryThisWeek	Pass	2
SystemBluetoothTurnOff	Pass	3
SystemBluetoothTurnOffVerify	Pass	2
SystemBluetoothTurnOn	Pass	3
SystemBluetoothTurnOnVerify	Pass	3
SystemBrightnessMax	Pass	4
SystemBrightnessMaxVerify	Pass	3
SystemBrightnessMin	Pass	4
SystemBrightnessMinVerify	Pass	3
SystemCopyToClipboard	Pass	6
SystemWifiTurnOff	Pass	4
SystemWifiTurnOffVerify	Pass	3
SystemWifiTurnOn	Pass	2
SystemWifiTurnOnVerify	Pass	2
TasksCompletedTasksForDate	Pass	4
TasksDueNextWeek	Pass	4
TasksDueOnDate	Pass	3
TasksHighPriorityTasks	Pass	2
TasksHighPriorityTasksDueOnDate	Pass	5
TasksIncompleteTasksOnDate	Pass	2
TurnOffWifiAndTurnOnBluetooth	Pass	6
TurnOnWifiAndOpenApp	Pass	8
VlcCreatePlaylist	Pass	17
VlcCreateTwoPlaylists	Pass	37