Quality Assurance
Part of Automation programme reviews
Measuring realised value rather than forecast savings
Compare an automation forecast with observed work and outcomes, separating released capacity from confirmed spending effects.
Measure realised value from work and outcomes observed after launch. Keep the approved forecast as a comparison, not as the result. A claim that automation released time needs evidence of the work that disappeared, the work that remains and any new work created across the process.
Reopen the forecast without rewriting it
Retrieve the approved case, its measurement period and assumptions about eligible volume, handling time, adoption, exceptions and running costs. Keep that version intact. Record separately which assumptions still hold. This distinguishes a forecast error from a change in operating conditions.
Choose the same case unit and business endpoint used for the intended benefit. If the outcome was a completed request, count requests rather than bot runs.
Report case types separately when their work differs materially. If automation handles simple cases while staff receive harder ones, a blended handling-time average can mislead.
Steps to Measure Realised Value from Automation
- Retrieve approved forecast and assumptionsKeep original version intact; note which assumptions still hold
- Define business endpoint and case typeUse same unit as originally planned (e.g. completed requests, not bot runs)
- Separate case types by work complexityAvoid blended averages when simple and complex cases are handled differently
- Measure observed effort and outcomesInclude checking, exceptions, corrections, and support work
- Report gaps with evidenceInvestigate differences in volume, adoption, case mix, and cross-team work
Build the observed result
For the period under review, count eligible cases and the human effort in their current route. Include checking, exceptions, corrections, support and receiving-team work. Compare that effort with a baseline adjusted to the same eligible case types and volume where reliable baseline data exists.
Keep elapsed time and output quality beside effort; a shorter task can still leave cases waiting.
| Line | Treatment |
|---|---|
| Approved forecast | Original expected benefit and its assumptions |
| Observed capacity released | Comparable baseline hours for the reviewed work minus current human hours across the process |
| New operating effort | Monitoring, repair and other work only if it is absent from the current-hours figure |
| Spending effect | Confirmed reduction in expenditure or a separately evidenced expense avoided |
| Outcome and guardrails | Completed results, corrections, waiting and unresolved cases |
Do not subtract a support or review hour twice. If current human hours already include it, a second deduction understates the result. If baseline or current effort is estimated, label the calculation provisional and show the basis. Compare costs and benefits over the same period.
Explain the gap
A difference between forecast and observation is a question to investigate. Check eligible volume, adoption, case mix, exceptions and work taken on by other teams. An automation might improve the business result while missing its hours forecast. Record that outcome on its own evidence rather than converting it into an unsupported dollar figure.
Describe released hours as capacity unless a budget owner confirms a corresponding spending reduction. An avoided future expense also needs a specific planned cost and an agreed basis; show it separately from money already saved.
Finish with the observed amount or range, case population, method, missing evidence and next decision. A forecast may be updated for future planning, but the approved version should remain recognisable as the earlier expectation.
Key Metrics for Realised Value Measurement
- Eligible case volume
- Actual count of cases processed during review period
- Human effort (baseline)
- Hours recorded before automation rollout
- Current human effort
- Hours spent post-automation, including exceptions and support
- Capacity released
- Baseline minus current effort (adjusted for case mix)
