Brief summary
Maestro is an open-source UI automation tool. You write a user journey as a short YAML file and it drives the app on a phone or in a browser the way a person would. Over the past months we used it to build the UI regression suite for a mobile app that ships on Android and iOS, and for its web admin. This post covers how we structured the suite so that QA can own it, what surprised us on real devices, including what Maestro cannot do on a physical iPhone and what we use there, how it compares with Appium on the same journey and the same two phones, and where a UI test is the wrong instrument.
Introduction
The product we test is a mobile app that talks to a piece of hardware over Bluetooth, runs on two operating systems and gets new builds several times a week. Regression by hand on every build does not scale, and we did not want UI regression to depend on developer time. We needed a suite that QA writes, QA maintains and developers can read in review.
Maestro's pitch fit that. It works at what its docs call arm's length: it does not need the app's source or a test SDK, it reads the accessibility tree the OS already exposes and sends taps and text through the device. Tests are YAML. Waits are built in: an assertVisible retries for up to seven seconds before it fails, so flows do not fill up with sleeps. A few months in, on real devices, we would pick it again for the same job, with the numbers and the iPhone caveat below in front of us.
What we built
Close to sixty flows for the mobile app and about a hundred for the web admin, written and maintained by QA without developer time. The number matters less than the shape, because the shape is what let the suite grow without slowing down.
A test file reads like the ticket. It has a given, a when and a then, and it contains no locators at all:
appId: ${APP_ID}
tags: [login, smoke]
---
- runFlow: ../handlers/handle_permissions.yaml
- runFlow: ../steps/login/open_login_screen.yaml
- runFlow:
file: ../steps/login/enter_credentials.yaml
env:
EMAIL: ${TEST_EMAIL}
PASSWORD: ${TEST_PASSWORD}
- runFlow: ../steps/login/tap_submit.yaml
- runFlow: ../handlers/dismiss_save_password.yaml
- runFlow: ../steps/login/verify_signed_in.yaml
Underneath are three layers. Pages are small JavaScript files, one per screen, holding the ids and the visible texts. Steps are shared sequences: open the login screen, enter credentials, verify the home screen. Handlers deal with everything the OS throws at a run: permission prompts, the offer to save a password, a banner. A test calls steps, steps read pages, and handlers are called wherever the OS can interrupt.
The three layers of the suite and the handlers strip that absorbs the OS.
One rule came out of this: Android and iOS run the same test, and where they differ, the difference lives in the page file, never in the test. When an id changes, one line changes. A new screen gets its page and its steps while the ticket is still open, and a developer reading the test in review can tell whether it checks what the ticket asked for.
Tags decide what a run includes: a smoke set, the full regression, and the flows that need the hardware on the bench or a real mailbox. Credentials come in through environment variables. Each test carries the ticket it covers, and our test management tool mirrors the flows one to one, so the whole team can see the coverage.
What surprised us
Three things on real devices shaped the suite more than any feature of the tool did. None of them is in the quick start; all three are in the docs or in the run logs once you know what to look for, and each one changed how a flow is written, how a run is wrapped, or what runs it.
Visible means in the accessibility tree
Maestro judges visibility from the accessibility tree, not from pixels. That is why it works on any app, and it has a consequence: an element under a modal, or inside a closed drawer, is still in the tree. In one of our runs the assert on the home title completed while the screenshot Maestro saved for the same step shows a dialog over it.
The screenshot Maestro saved for that step, next to the log line it wrote for it.
The fix is in the flow. Dialogs get dismissed before the assert, and where a dialog is possible an assertNotVisible on it goes in front of the title check. Steps that change state take a screenshot, and takeScreenshot gives you one wherever you want it. When the picture and the tree disagree, we trust the picture.
The phone is part of the system under test
On both platforms Maestro installs a small companion app on the device and drives through it, and on a physical Android phone that install happens at the start of every run unless you tell it not to. When it stalls, the run stalls with it, and Maestro has no timeout of its own for that: the debug log ends at the line naming the device and the first step never starts. So every run has a timeout around it, a run that never reached its first step counts as an infrastructure failure, and someone has to own the phones.
The five things that take part in one run. QA writes the first one.
No physical iPhone in the official CLI
Maestro's CLI runs iOS flows on Xcode simulators; a physical iPhone is not supported, and the docs say so. Our iOS testing has to happen on real hardware because of Bluetooth, so on the iPhone we run the same YAML through maestro-runner, a community runner that drives the installed app through a WebDriverAgent signed with our team certificate. The first start builds that agent and takes minutes; later starts reuse it.
A community project follows iOS releases with a delay, so we keep Appium with a prebuilt WebDriverAgent as the second way to drive the phone, and we use it for exploratory work anyway. On an iPhone, plan for two tools from the start.
Maestro and Appium on the same two phones
How we measured
Speed comes up in every conversation about Maestro, so we measured it against Appium: one journey, the same five steps, both tools, both phones.
- The journey: open the app, wait for the home title, swipe from the right edge to open the sidebar, tap Log in, tap Take me back on the dialog that follows.
- Same conditions everywhere: a physical Samsung and a physical iPhone 13, app logged out, cold start, text selectors only, no handlers, the tool's helper app already on the phone, one warm-up run and three timed runs.
- We time the four UI steps, from the launched app to the last tap. Tool start-up, session creation and report writing are left out on both sides, because they are not the same thing in the two tools. On the iPhone, Maestro means the community runner from the previous section.
Speed
| Seconds, median of 3 runs | Maestro, Samsung | Appium, Samsung | Maestro runner, iPhone | Appium, iPhone |
|---|
| Home title visible | 1.3 | 0.8 | 0.3 | 0.2 |
| Swipe opens the sidebar | 1.3 | 0.5 | 1.0 | 3.0 |
| Tap Log in | 3.1 | 1.1 | 1.2 | 0.7 |
| Tap Take me back | 2.5 | 1.3 | 1.0 | 0.7 |
| Four steps together | 8.2 | 3.6 | 3.5 | 4.6 |
On Android, Appium does the four steps in less than half the time. The gap is in what each tool does after an action: Maestro waits for the screen to stop changing before it reports a step done, Appium returns as soon as the tap is sent. That waiting is Maestro's built-in tolerance to flakiness, and this is its price, one to two extra seconds per action on this app. On the iPhone the two are close: the community runner has no settle wait, and Appium loses time on the swipe.
Published comparisons mostly put Maestro ahead on speed, usually without a journey or numbers attached. On our app the order is reversed on Android, and the reason is the settle wait above. Where Maestro is clearly faster is the loop around the run: a flow is interpreted, so a change runs seconds after you save it.
Everything else
Speed is one row of the comparison. The rest, from the two projects' docs and our own use:
| Maestro | Appium |
|---|
| Setup | One binary and a Java runtime; first flow runs within minutes | Node, a server, a driver per platform, a client library in your language |
| Written in | YAML; no programming language needed | Java, Python, JavaScript or another language |
| Waits and retries | Built in; an assert retries for seven seconds | An implicit timeout you set and explicit waits you write |
| Complex logic | Limited: conditions and small JavaScript snippets | Anything the language can do |
| Physical iPhone | Not in the official CLI | Supported |
That second table is why the regression suite stays on Maestro: a QA engineer writes and fixes the flows between tickets, with nothing to compile and no server or client library to keep in step. The first table is what that costs on the phone. For a suite that runs unattended, a second or two per action does not change the decision.
The trade-offs
Limits in the docs
The limits are in the docs. Web support is in beta and Chromium only; it worked for our admin portal, and we would not plan a cross-browser strategy on it. On Android, inputText only accepts ASCII, which matters as soon as a form takes a name with diacritics or a non-Latin script. On iOS, hideKeyboard is listed as flaky and the docs suggest tapping a neutral area instead. Check the list against your own app before committing.
What belongs in a UI test
The bigger trade-off was ours. A phone test takes tens of seconds and a phone; our production flows take 53 s at the median, so the sixty mobile flows on one phone are most of an hour. An API test for the same rule takes a second and no hardware. When a ticket says the server must reject a malformed email, the phone test proves the error text is on the screen, and the API test proves the rule. We write the second one, and keep the first only where the screen itself is the requirement.
The phone test proves the error is on the screen. The API test proves the rule.
The same applies in the other direction. Our home screen shows a signal level next to each device. Maestro can read the "100%" label. It cannot know whether the device is connected. When the proof lives in hardware, in a colour or in a pixel, we do not ask a screen-tree assertion to give it.
The rule we use now:
- Maestro: journeys a person does on the phone, where the outcome is on the screen.
- API tests: rules and contracts, status codes, validation.
- Device logs or a human: physical outcomes, colour, timing.
- Nowhere: a test written only so that everything is automated.
Conclusion
Maestro gave us a UI suite that QA owns, developers can read, and that runs on both phones and the web admin from one set of files, which is what we needed.
Measured, Appium does the same steps faster on Android, and on a physical iPhone it is the better-supported of the two. We keep Maestro for the regression suite because QA can maintain it, and Appium on the bench for the iPhone and for anything that needs code. What made the suite work was the structure around the tool: pages, steps and handlers that keep the tests readable, a screenshot at every state change, a timeout around the run, a person who owns the phones, and a clear line about what belongs in a UI test.
