An automated test fails after a new product iteration. The natural reaction is to open the test code and find what broke. But on an iterative game project, I learned to ask a different question first:
Is this test broken, or is the behavior it validates no longer part of the product?
That question changed how I think about automation. A failing test is not always a maintenance task. Sometimes it is a signal that the product contract has changed—and that the test may no longer deserve to exist.
When a Broken Test Is Not the Real Problem
When an automated test stops working, there are at least two systems to investigate: the automation and the product. A selector may have changed, an interaction utility may no longer work, or the test environment may be unstable. In those cases, repairing the automation is appropriate.
But the product itself may also have changed. A gameplay interaction may have been redesigned, an asset may have been replaced, or an experimental feature may have taken a different direction. A test written against the earlier version can be technically repairable while no longer being conceptually valid.
If I immediately update the code, I may spend time preserving an expectation that the product team has already abandoned. Worse, the repaired test can create false confidence because it verifies yesterday’s behavior instead of today’s intent.
My maintenance process therefore starts outside the test code. I confirm the current expected behavior with the available product and design information. I determine whether the user journey still exists, whether its outcome is still correct, and whether the risk the test was meant to protect is still relevant.
Only then do I decide whether to repair, rewrite, replace, or remove the test. Automated tests are executable expectations; when the expectation changes, the code must not be treated as the source of truth by default.
You Cannot Automate What You Do Not Understand
Before automating a behavior, I want the team to understand what the user is trying to accomplish and what a successful outcome looks like.
Knowing how to execute a sequence of steps is not the same as understanding the behavior. A tester may be able to complete an interaction and check that the next screen appears while important questions remain unanswered:
- Which part of that flow is an actual product requirement?
- Which visual or interaction details are still experimental?
- What would represent a meaningful failure for the user?
- Which outcome is essential, and which implementation detail could change?
- What risk is the automated test intended to detect?
Without those answers, automation can encode assumptions instead of requirements.
This became especially clear to me while working with gameplay. A test can reproduce a sequence perfectly and still validate the wrong thing. Without a clear goal, it may become overly attached to one animation, screen position, or temporary flow.
Product understanding helps us choose assertions that survive appropriate change. It also helps us recognize inappropriate change when it occurs.
The question I use is:
Do we understand the behavior well enough to recognize when the test is wrong—and when the product is wrong?
If the answer is no, automation is premature. The next task is not to write more code. It is to learn more about the behavior.
A Proof of Concept Is Not a Stable Contract
On an iterative game project, some of the earliest gameplay existed to help the team discover what the final experience should become. Dynamics, assets, interactions, and flows were intentionally revised as the team learned.
That kind of iteration is not a failure of planning. A proof of concept answers questions; it does not promise that every current detail will remain.
When we automated scenario details too early, normal product changes repeatedly broke the tests. Each failure required investigating what changed, why it changed, whether the test remained relevant, and whether the new behavior needed a different check. Maintenance included rebuilding our understanding of the scenario, not only editing code.
There is an important difference between a game that has been clearly designed on paper and one that is deliberately being shaped through successive interactions.
When the dynamics, assets, gameplay, and expected outcomes are already understood, automation can begin earlier with more confidence. There is a clearer contract to encode.
When those decisions are still being explored, detailed scenario automation may create more churn than useful feedback. Manual testing is often faster because a person can adapt to the latest iteration and observe unexpected behavior without rebuilding outdated expectations. Waiting is a technical decision about timing.
What We Can Build Before the Product Stabilizes
Postponing scenario-specific tests does not mean putting automation work on hold.
While gameplay was still changing, we could build the reusable parts of the framework that did not depend on a final interaction. That included device connectivity and configuration, utilities for common interactions, the foundational test structure, and the mechanisms needed to collect execution evidence.
We could also prepare hooks, listeners, or collectors for information such as execution status, failures, duration, logs, and screenshots. The final report format did not need to be decided immediately. What mattered first was making sure the framework could capture reliable data that a future report would need.
This separation is valuable:
- Product-specific scenarios express what the current product should do.
- Framework capabilities provide the stable machinery for executing, observing, and diagnosing those scenarios.
The first category may need to wait for clearer product decisions. The second can advance much earlier.
The framework should make future change inexpensive. If device handling, common actions, evidence collection, and reporting concerns are scattered through every test, a small product change can create widespread maintenance.
Reusable utilities do not eliminate change, but they give the team controlled places to respond to it. They also allow scenario development to move faster once the behavior is ready to encode.
The real choice is not “automate or do nothing.” It is whether to encode unstable expectations now or invest in the foundation that will support valuable automation later.
The Signals That Tell Me It Is Time to Automate
Stability is important, but I do not wait for a product to become permanently finished. That moment rarely arrives.
Instead, I look for enough clarity to treat the behavior as a useful contract. The user goal is understood. The expected outcome can be explained. The team knows which details are intentional, and the scenario is likely to remain relevant through future iterations.
Then I consider value.
A stable scenario is not automatically worth automating. I consider execution frequency, the risk created by failure, and the usefulness of the feedback. Repetition can justify automation, while a less frequent scenario may still justify it if it protects a critical user or business flow.
I also consider diagnosability. A test that only reports “failed” may cost more time than it saves. The framework needs enough evidence—logs, screenshots, status, and context—to help the team understand the result.
Finally, I compare the expected value with both implementation and maintenance. A quick script is not necessarily a cheap test. If the underlying behavior changes frequently or the scenario depends on fragile details, its long-term cost can be much higher than the initial implementation suggests.
The right time to automate is when the team has sufficient product knowledge, the behavior is stable enough, and the expected feedback is worth the complete cost of ownership.
When Manual Testing Is Still the Better Decision
Manual testing and automation are not competing maturity levels. They are different ways of learning about product quality.
Manual validation is often better when behavior is changing rapidly. A tester can explore the current experience, notice confusing outcomes, and adjust as new information appears.
It can also be the better economic decision.
If a scenario takes little time to validate manually but requires significant implementation, device setup, data preparation, or ongoing maintenance, automating it may not create a return. The fact that something can be automated does not mean it should be.
Manual testing is also essential when the team needs exploration rather than repetition. Automation is strong at checking known expectations consistently. A person is better positioned to investigate uncertainty, notice behavior outside the scripted path, and ask whether the experience makes sense.
The decision can change. Manual exploration may become a strong automation candidate once behavior stabilizes and repetition increases. An automated test may deserve removal if the risk, behavior, or cost changes. A healthy strategy keeps both options available.
My Automation Decision Framework
I use five questions before committing to scenario automation:
-
Do we understand the user behavior and expected outcome? We should be able to explain the user’s goal, success, failure, and the risk the test will protect. If we cannot, we need product learning before test code.
-
Is the behavior stable enough to encode? The product does not need to be finished, but the scenario needs enough agreement to function as a meaningful contract rather than a temporary experiment.
-
Is execution frequency or product risk high enough to justify automation? Repetition can create value, and so can protection of a critical flow. A scenario with neither may remain more efficient to validate manually.
-
Is implementation and maintenance cost lower than the expected value? The comparison must include framework needs, test data, devices, diagnosis, and future updates—not only the time required to write the first version.
-
Will the behavior remain in the product long enough for the investment to pay back? A short-lived experiment may disappear before automation provides meaningful value. A durable user journey has more opportunity to repay the investment.
These questions do not produce a mathematical score. They create a structured conversation among QA, development, product, and design. They make the decision explainable and easier to revisit when conditions change.
Three Lessons for QA Leaders
First, automation maturity should not be measured by test count. A large suite can still be expensive, fragile, and poorly aligned with product risk. The more useful measure is whether automation provides trustworthy feedback that helps the team make decisions.
Second, reusable infrastructure can advance while product behavior stabilizes. Device connectivity, interaction utilities, evidence collection, and reporting foundations prepare the team without prematurely turning an experiment into a fixed contract.
Third, before repairing a broken test, review the expectation it represents. The correct response may be to fix the code, but it may also be to rewrite or remove the test because the product has moved forward.
The strongest automation strategy is not the one that automates everything as early as possible. It is the one that understands where automation creates durable value, where manual testing creates faster learning, and when the decision should change.
When has waiting to automate been the better technical decision for your team?