User Acceptance Testing That Holds Vendors Accountable
In this article
- What user acceptance testing is for, and what it is not
- Build the UAT test plan from the signed requirements
- Classify every finding: defect, gap or change request
- Run cycles, not a single session
- Entry and exit criteria
- Entry criteria (do not start until all are true)
- Exit criteria (acceptance can be considered when all are true)
- Turning the results into a controlled partial acceptance
- UAT mistakes that let vendors off the hook
- Key takeaways
- Frequently asked questions
- What is user acceptance testing in an ERP or MES project?
- Who should perform UAT?
- How many UAT cycles are needed?
- What is the difference between a defect and a functional gap?
- What should a UAT test plan template contain?
User acceptance testing (UAT) is the stage where the company, not the vendor, proves that a delivered system does what the signed requirements say it should do, using real scenarios and real data. It is the last point at which you hold leverage: before formal acceptance, open items are the vendor’s problem; after it, they are change requests. This article explains how to run UAT so it produces evidence a vendor cannot argue with.
The approach comes from an audit I ran on a manufacturing execution system (MES) that had been delivered to our plant. I took the signed requirements specification, turned it into a 60-point test plan, and executed it over four test cycles with the operations users. The result was 3 defects and about 20 functional gaps, each tied to a requirement ID, a test step and a screenshot. That file was the evidence base for a controlled partial acceptance: we accepted what worked, withheld acceptance of what did not, and agreed dates and conditions for the rest. Nobody shouted; the document did the talking.
What user acceptance testing is for, and what it is not
UAT answers one question: does the system, as delivered and configured, satisfy each agreed requirement from the business user’s point of view? It is not the vendor’s internal testing, it is not a training session, and it is not the time to discover what you want. Three things are commonly confused:
| Test type | Who runs it | Question answered | Baseline |
|---|---|---|---|
| System / integration testing | Vendor | Does the build work technically, including interfaces? | Functional specification |
| User acceptance testing | Company’s business users, led by the project manager | Does it meet the signed requirements in our scenarios with our data? | Requirements document with IDs |
| Training | Vendor or key users | Can users operate it? | Role-based training plan |
If the vendor proposes to “run UAT with you”, decline politely. They can support it, fix findings and retest, but the test cases, the execution and the verdict belong to the company. If you do not yet have a requirements document with IDs, fix that first; the method in how to write ERP requirements vendors cannot misread produces exactly the baseline UAT needs.
Build the UAT test plan from the signed requirements
Every test case traces to a requirement ID. This is what turns a finding from an opinion (“it should work differently”) into a contractual fact (“REQ-023 is not met”). For the MES audit, the 60 test points came from walking the specification section by section and asking, for each requirement, “what would I have to do on the screen to prove this is true, and what would I expect to see?”
A test case needs the fields below. I keep them in a spreadsheet with one row per case so that cycle results can be added as columns.
Test ID: UAT-017
Requirement: MES-RQ-3.4 (production order status sync to ERP)
Process: Plan to produce
Role: Production supervisor
Preconditions: Production order PO-1001 released in ERP;
MES connected; line 2 idle.
Steps: 1. Start PO-1001 on line 2 in MES.
2. Record 500 kg output, 12 kg scrap.
3. Close the order in MES.
Expected: ERP production order shows 500 kg completed,
12 kg scrap, status Closed, within 1 minute,
without manual entry.
Test data: Item FEED-L2-50, batch B-2409-07.
Result: Pass / Fail / Blocked
Evidence: Screenshot or report reference
Finding type: Defect / Gap / Change request / None
Severity: S1 / S2 / S3 / S4
Rules for building the plan:
- Cover every Must requirement with at least one test; cover Should requirements with the scenarios users run daily.
- Write expected results as observable facts: a status, a quantity, a document number, a report line. “Works correctly” is not an expected result.
- Include negative tests: wrong batch, over-issue, missing authorization. Systems usually pass the happy path and fail the exceptions.
- Include end-to-end chains across systems: an MES event that must appear in the ERP, an HR change that must reach payroll. Interfaces are where most gaps hide.
- Use real master data and realistic volumes. Ten test items and one customer will not expose what a month of live transactions will.
Classify every finding: defect, gap or change request
The single most valuable discipline in the MES audit was separating what we found into three types. Vendors will happily reclassify a gap as a change request if you let them; the classification has to be yours, and it has to be defensible.
- Defect: the feature exists but behaves wrongly (calculates incorrectly, crashes, loses data). The vendor fixes it at no cost.
- Functional gap: a signed requirement is not delivered at all, or only partially. The vendor delivers it at no cost, because it was in scope and priced.
- Change request: something the business now wants that was not in the signed requirements. The company decides whether to pay for it, and when.
Severity determines what blocks acceptance. The scale I use:
| Severity | Definition | Effect on acceptance |
|---|---|---|
| S1 Critical | Process cannot be completed; data loss or wrong financial posting; no workaround | Blocks acceptance of the affected scope |
| S2 Major | Requirement not met; manual workaround possible but not sustainable | Blocks acceptance unless fixed by an agreed date, with payment withheld |
| S3 Minor | Requirement met with usability or cosmetic issues; workaround acceptable | Accepted with a dated fix list |
| S4 Observation | Improvement idea; no requirement breached | Logged as a candidate change request |
Agree this scale with the vendor before UAT starts, ideally in the contract. Arguing about definitions while findings are on the table is a losing position.
Run cycles, not a single session
A single UAT “day” cannot produce reliable evidence, because the first pass always finds blockers that hide everything behind them. The four cycles I ran on the MES had different purposes:
- Cycle 1, full pass. Execute all 60 test points regardless of results. Record every failure, including ones caused by earlier failures (mark them Blocked, not Fail). This gives the vendor one consolidated list instead of a drip of emails.
- Cycle 2, retest after fixes. Re-run every failed or blocked case once the vendor delivers fixes. Also re-run a sample of passed cases adjacent to the changed areas; fixes break neighbors.
- Cycle 3, integration and regression. Run the end-to-end chains across the MES, the ERP and any other connected systems with the fixes in place, plus the full set of previously passed cases. This is where several of the functional gaps surfaced, because single-screen tests had passed while the data did not arrive where it was supposed to.
- Cycle 4, acceptance run. Execute the complete plan on a frozen build with production-like data, witnessed by the vendor, with results signed by the process owners. This is the record that goes into the acceptance decision.
Between cycles, hold a short findings review with the vendor: confirm classification and severity, agree who does what by when, and keep the findings log as the single source of truth.
Entry and exit criteria
UAT that starts too early wastes users’ time; UAT that ends on a date rather than on evidence wastes the leverage. Write both sets of criteria down.
Entry criteria (do not start until all are true)
- Vendor has completed its own system and integration testing and provided the results.
- The test environment matches the intended production configuration and contains migrated or representative master data.
- Test cases are written, mapped to requirement IDs and reviewed by process owners.
- Users have been trained on the screens they will test.
- A findings log, the severity scale and the review cadence are agreed with the vendor.
Exit criteria (acceptance can be considered when all are true)
- 100% of test cases executed in the acceptance cycle.
- No open S1 findings; no open S2 findings without a dated, contractually backed fix plan.
- All findings classified and agreed in writing as defect, gap or change request.
- End-to-end integration scenarios passed with data verified in the receiving systems.
- Process owners have signed their sections of the results.
Turning the results into a controlled partial acceptance
Full rejection is rarely realistic: the plant needs the system, and the vendor has delivered most of it. Full acceptance with a “promise to fix” list is how companies lose. The middle path, which the MES evidence allowed us to take, is a partial acceptance:
- List the scope that passed, by requirement ID, and accept it formally. Users can start working on it.
- List the open defects and gaps, with severity, owner and a fix date for each.
- Tie a defined portion of the remaining payment to closure of that list, verified by a retest the company runs, not a vendor statement.
- State explicitly that items on the gap list are in-scope deliverables, not change requests, and that warranty and support periods start from final acceptance, not partial.
- Record the change-request candidates separately, unpriced, so they do not contaminate the gap list.
The tone of this negotiation depends entirely on the quality of the evidence. Twenty gaps described as “the system doesn’t do what we expected” invite debate; twenty gaps each citing a requirement ID, a test step, an expected result and a screenshot invite a fix schedule.
UAT mistakes that let vendors off the hook
- Testing from the vendor’s scripts. Their scripts test what they built. Yours must test what you bought.
- Accepting a demo as a test. Watching a consultant click through a scenario is not users executing it on their own.
- No requirement traceability. Findings without IDs are opinions.
- Letting severity be set by the vendor. Everything becomes S3.
- Testing only the happy path. Exceptions and interfaces are where the gaps are; the MES audit found most of its gaps in cross-system flows, not on single screens.
- Signing acceptance to hit a go-live date. Go live on a partial acceptance if you must, but never sign away the open list.
- No retest after fixes. A vendor’s “fixed” email is not a test result.
Key takeaways
- UAT is the company’s test against the signed requirements, not the vendor’s test of its own build.
- Derive every test case from a requirement ID, with observable expected results and real data.
- Classify findings as defect, gap or change request, with a severity scale agreed before testing starts.
- Run multiple cycles: full pass, retest, integration and regression, then a witnessed acceptance run.
- Use written entry and exit criteria so UAT ends on evidence, not on a calendar date.
- When results are mixed, negotiate a partial acceptance with a dated gap list and payment tied to closure.
Frequently asked questions
What is user acceptance testing in an ERP or MES project?
User acceptance testing is the phase in which the company’s business users execute realistic scenarios on the delivered system to confirm it meets the signed requirements before formal acceptance. It follows the vendor’s own system and integration testing and precedes go-live. Its output is a findings log that drives fixes, acceptance decisions and, where needed, payment milestones.
Who should perform UAT?
The company’s process owners and key users, coordinated by the company’s project manager. The vendor supports the environment, fixes findings and attends reviews, but should not write the test cases or decide pass or fail. Independent execution is what gives UAT results weight in acceptance and contract discussions.
How many UAT cycles are needed?
At least three in practice: a full first pass, a retest of failures after fixes, and a final acceptance run on a frozen build. Complex projects with interfaces benefit from a separate integration and regression cycle. The MES audit described here used four cycles, which was what it took to get a stable, witnessed result.
What is the difference between a defect and a functional gap?
A defect is a delivered feature that behaves incorrectly; a functional gap is a signed requirement that was not delivered or only partially delivered. Both are the vendor’s responsibility at no extra cost. A change request, by contrast, is new scope the business asks for after signature and is priced separately.
What should a UAT test plan template contain?
For each test case: a unique test ID, the requirement ID it verifies, the process and user role, preconditions, numbered steps, test data, the expected result as an observable fact, the actual result per cycle, evidence reference, finding type and severity. A summary sheet should show coverage of Must requirements and open findings by severity.
If you are approaching acceptance on an ERP, MES or integration project and want an independent test plan built from your requirements, that is a core part of my client-side ERP implementation service, and the same approach was applied in the SAP Business One and MES integration case study. Feel free to contact me to discuss your project’s acceptance stage.