The demonstration worked. An incoming email became a draft order, with the customer details and product codes in the right places. Then someone asked what would happen if the order system stopped responding after receiving the request.
Would the workflow check whether the draft had been created, or submit the order again?
That question belongs in the pilot review. A successful demonstration shows that a process can work. Before relying on it, the business also needs to know how incomplete information and interrupted transactions will be handled.
Define the job before testing it
Consider a fictional distributor that wants to extract order details from emails and prepare drafts for staff review. This is an illustrative example, not a SAGEN customer case study.
The pilot can read an approved mailbox, look up product codes and prepare draft orders. It cannot change payment details, promise delivery dates or release shipments.
Write down the permitted data sources and actions, along with the point at which a person must approve the work. Someone reviewing a draft has different responsibilities from someone authorizing a purchase.
Use a controlled test environment with synthetic or appropriately de-identified records. There is no reason to interrupt live orders or expose customer information to demonstrate a failure.
Test 1: The request contains conflicting information
The email gives one delivery address; the attachment gives another. A product description also matches two catalog items.
The workflow should hold the draft and show the reviewer the conflicting details. It should not select an answer simply to complete the task.
Inspect the resulting record. Can the reviewer find the source of each disputed field? Is the order still awaiting approval? Has the item reached the correct review queue with a reason someone can understand?
Set explicit rules for missing or contradictory information. A model's confidence score does not resolve a disagreement between two addresses or establish which customer account is intended.
The reviewer needs a usable way to correct the record, request clarification or reject it. Check what happens after each choice, including whether the corrected item can continue without losing its history.
Test 2: The order system does not confirm the result
The workflow submits a request to create a draft, but the connection times out before a response arrives. The draft may already exist.
The next step should establish what happened. Submitting the same request again without checking could create an extra order.
Look for an operation identifier that connects the request to the destination record. Confirm that recovery checks the order system or uses its documented duplicate-protection mechanism.
Some APIs support idempotency keys: repeated requests for the same operation can be processed without producing additional effects. The details depend on the API's contract and implementation. Amazon's engineering guidance explains why this matters when designing retries. [1]
Agree a retry limit and a recovery path for cases that remain uncertain. The person handling an exception should be able to see what was attempted and what is still unknown, without having to reconstruct the transaction from scattered logs.
Test 3: The same request arrives again
A customer forwards the original email. Alternatively, a scheduled job restarts and processes an item it has already seen.
Check that a repeat does not create an extra draft. Also test a legitimate new order with the same products and quantities. The duplicate check must preserve that new work.
This requires a business rule, not just a comparison of similar text. Agree which order references, customer identifiers and revision details distinguish a repeated request from a new transaction. Decide how a revised attachment affects the earlier draft.
Inspect the records after both tests. The evidence should explain why the second item was linked to an existing order, held for review or treated as a separate request. A successful test accounts for every input rather than quietly dropping difficult cases.
Make sure someone can resolve the exception
A workflow can correctly hold an order and still leave the business waiting indefinitely. Test the handoff when the usual reviewer is unavailable.
Pending items should have an owner, a visible age and enough context for an authorized backup to act. Set an escalation interval and agree how staff will return to the manual process if necessary. A review timeout must not become permission to release an order.
Include review and correction time in the pilot results. An extracted document is not completed work if someone still needs to spend several minutes fixing it.
Recovery also needs to stay within the workflow's permissions. For example, a document that asks the system to search another customer's records must not grant that access. Restrict actions in the application and connected systems, rather than relying on instructions to the model alone. OWASP recommends limiting available tools and permissions and requiring approval for high-impact actions. [2]
Our secure enterprise system guide covers the wider architecture considerations.
Record the result before deciding to launch
Copy this worksheet into the pilot plan. Agree the expected result before each test and complete the other fields afterward. Use evidence references rather than sensitive customer details.
| Test | Expected result | Observed result | Evidence reference | Accountable owner | Launch decision |
|---|---|---|---|---|---|
| Conflicting input | Hold the draft; show the conflict to a reviewer | Fill in | Fill in | Name | Pass / Fix and retest |
| Timeout after write | Check the outcome; recover without an extra draft | Fill in | Fill in | Name | Pass / Fix and retest |
| Duplicate request | Link the repeat; preserve legitimate new work | Fill in | Fill in | Name | Pass / Fix and retest |
A blank or unverified result is not a pass. Record defects and retest evidence before closing them. Note the workflow version so relevant checks can be repeated after changes to prompts, models, permissions or integrations.
These three tests are a starting point, not a complete security assessment. Add cases appropriate to the work and its consequences. NIST's risk-management guidance supports evaluation tied to the intended context of use. [3]
Compare the full process with the existing manual approach, including reviewer time, rework, queue delays and unresolved transactions. Decide which failures block launch before looking at average completion speed. The result may support a limited release, a narrower pilot or further work before production.
SAGEN's AI Opportunity and Security Assessment provides a use-case shortlist, data and risk assessment, and recommended pilot scope. To discuss a process you are considering, bring a description of the task and the exceptions your team currently handles.
Frequently asked questions
What should a workflow pilot test before launch?
Test normal work and foreseeable failures: incomplete inputs, conflicting data, duplicate events, uncertain integration results, unavailable reviewers and unauthorized actions. Agree the expected result for each case before running it.
Should a workflow automatically retry a failed action?
Only when the failure and action support a safe retry. A timeout can leave the outcome unknown. Check the destination or use the API's documented idempotency mechanism before repeating an action that could create a duplicate effect.
Does every automated action need human approval?
No. The review requirement should reflect the action's impact, the workflow's authorization and the consequences of an error. Define high-impact approval points and test what happens when approval is missing or refused.