Quality Assurance
Part of Automation quality assurance
Measuring error rates before and after automation
Define a business error rate, keep denominators and review coverage comparable, and interpret before-and-after results carefully.
To compare error rates, define a wrong business outcome, count it against the same eligible unit before and after automation, and show how outcomes were checked.
A platform's failed-run percentage answers a different question. A run can finish with a wrong record, while a failed run may leave a successful write that needs reconciliation.
Fix the unit and error definition
Choose a unit that survives the process change: one request reaching a decision, one order reaching a destination or another agreed case. Write down the start event, end state and included case types. Define an error a reviewer can verify, such as a wrong recipient, incorrect value, unauthorised action or missing required result.
For a primary case-level rate, count a case with several defects once: eligible cases with at least one confirmed error ÷ eligible cases whose outcomes were checked. Record the number and types of defects separately. Show unresolved cases and unchecked cases apart from the denominator; neither is evidence of a correct outcome.
Before-and-after measurement workflow
- Define a wrong business outcomeName an error a reviewer can verify, such as a wrong recipient, incorrect value, unauthorised action or missing required result.
- Fix the eligible unitOne request reaching a decision, one order reaching a destination or another agreed case, with its start event, end state and included case types.
- Build a comparable baselineRetain case references, outcome evidence, error classifications, discovery points and checking status for the pre-automation period.
- Measure after launch on the same basisUse the same definitions and review method, and show eligible volume, number checked and case mix with each rate.
- Keep other quality signals visibleReport returned cases, repair work, unknown destination results and errors found after initial closure, and show run failures separately.
- Interpret with traceable casesPublish counts, denominators, periods, checking methods and exclusions, then inspect cases with and without recorded errors.
Establish a comparable baseline
For the period before automation, retain case references, outcome evidence, error classifications, discovery points and whether each case was checked. Include repairs found by the receiving team.
If only a sample can be reviewed, record how it was selected. Reviewing only complaints or visible exceptions cannot estimate the rate across all eligible cases.
Use the same definitions and review method after launch. Show the eligible volume, number checked and case mix with each rate.
If the later period contains more simple cases, a lower overall rate may reflect the mix. Break out material case types when enough cases were checked for a useful comparison.
Records to retain for every eligible case
- Case reference that survives the process change
- Outcome evidence from both source and destination
- Error classification and defect type
- Discovery point, including which team found it
- Whether the case was checked at all
- Repairs found by the receiving team
- Selection method, if only a sample was reviewed
- Unresolved and unchecked cases, held apart from the denominator
Keep other quality signals visible
Report returned cases, repair work, unknown destination results and errors found after initial closure. Record which team discovers and fixes each problem. A lower count in the originating team is incomplete evidence if the receiving team now performs the correction.
Show software run failures separately. Power Automate, for example, provides execution logs and performance metrics, including run and action counts and durations. Those execution measures do not establish the rate of incorrect business outcomes. If run status is the only evidence available, say that the business error rate has not been measured.
Interpret the change
Publish the before and after counts, denominators, periods, checking methods and exclusions with the percentages. Inspect some cases with recorded errors and some without, using source and destination evidence. Note material changes in rules, staffing, review coverage or workload.
Use the findings to choose the next check: repair a recurring error, examine case mix, widen outcome review or continue monitoring. A percentage without traceable cases cannot explain why it changed.
