What can synthetic pilots prove before virtual assistant work goes live?

Analysis 1: A synthetic pilot uses fabricated identities, documents, transactions, messages, and outcomes that reproduce workflow structure without copying a live person’s confidential record

A synthetic pilot uses fabricated identities, documents, transactions, messages, and outcomes that reproduce workflow structure without copying a live person’s confidential record. Its primary value is safe instruction testing: whether the worker can find the system, recognize required fields, follow the sequence, preserve evidence, stop at a decision boundary, and produce a reviewable result. It cannot reproduce every ambiguity, emotion, urgency, system history, or consequence of live work. The record should identify the population, observation window, system, responsible owner, and unavailable evidence. This keeps a numerical result from implying broader certainty than the design supports. Before collection, publish a field dictionary and freeze the extraction parameters. Record exclusions with reasons, maintain a population control total, and reconcile transfers or deletions. This prevents a later analyst from improving the result by silently redefining which cases counted after outcomes became visible.

Analysis 2: Start from a risk and task model rather than random fake examples

Start from a risk and task model rather than random fake examples. List normal path, missing field, contradictory source, duplicate, changed identity, suspicious link, inaccessible system, expired approval, urgent deadline, out-of-scope request, sensitive disclosure, and failed downstream action. For each case define expected permitted actions, stop point, escalation owner, completion evidence, and unacceptable behaviors before the assistant sees it. Preserve source facts separately from local interpretation. A virtual assistant can prepare the comparison and exception file; authorized security, legal, finance, HR, clinical, safety, or executive owners retain decisions in their fields. For each observation, distinguish a system event from a business fact and a reviewer conclusion. Systems can timestamp an action without explaining its purpose, completeness, authorization, or downstream effect. The study should retain those layers separately and state which evidence supports each claim.

Analysis 3: Make the artifacts realistic enough to test mechanics while marking them unmistakably as synthetic

Make the artifacts realistic enough to test mechanics while marking them unmistakably as synthetic. Use reserved domains, fake account ranges, nonproduction systems, watermarks, and isolated storage. Prevent test emails, payment files, customer notices, or API calls from reaching real destinations. A convincing scenario that can accidentally trigger production creates a new risk instead of reducing one. Apply least privilege and data minimization throughout the test. Store sensitive material only in approved systems, use stable references in the study file, and document access removal when temporary review ends. Reviewers should inspect a small random sample in addition to every defined high-risk case. Risk selection finds expected failure modes; random selection can reveal ordinary defects outside the model. Document the sample frame, selection seed or method, substitutions, and records unavailable for review.

Analysis 4: Keep an independent answer key and blinded review

Keep an independent answer key and blinded review. The designer records source facts, intended traps, expected classification, required fields, and correct owner. Reviewers compare outputs at the field and action level rather than assigning only pass or fail. Serious errors such as disclosure, unauthorized approval, or production execution should remain visible and should not be averaged away by many correct low-risk fields. The record should identify the population, observation window, system, responsible owner, and unavailable evidence. This keeps a numerical result from implying broader certainty than the design supports. Reperform calculations and classifications independently for selected cases. Compare inputs, formulas, time-zone handling, rounding, status mapping, and owner decisions. Investigate disagreement rather than forcing consensus into the dataset, and report when incomplete evidence prevents a defensible result.

Analysis 5: Test ambiguity deliberately

Test ambiguity deliberately. Real queues contain incomplete names, conflicting dates, poor scans, unfamiliar terminology, changed priorities, and instructions that do not fit the case. Include situations where the correct result is to pause and ask a narrow question. If every synthetic case has one clean path, the pilot rewards confident completion and fails to test judgment boundaries. Preserve source facts separately from local interpretation. A virtual assistant can prepare the comparison and exception file; authorized security, legal, finance, HR, clinical, safety, or executive owners retain decisions in their fields. Use counts, values, medians, ranges, and age bands with their denominators. Avoid a single composite score that lets numerous low-risk items offset one serious disclosure, unauthorized action, or payment event. Qualitative exception narratives belong beside the summarized measures.

Analysis 6: Measure instruction quality separately from worker performance

Measure instruction quality separately from worker performance. When several capable reviewers misread the same field or choose different owners, revise the playbook, form, role map, or status definition. Re-run failed scenarios after changes and preserve both versions. The purpose is to improve the operating system, not to create an exam whose ambiguity is treated as a personnel defect. Apply least privilege and data minimization throughout the test. Store sensitive material only in approved systems, use stable references in the study file, and document access removal when temporary review ends. Interview operational owners with the same neutral prompts: what evidence was missing, which field was ambiguous, what decision remained theirs, and whether the prepared record supported that decision. Do not ask leading satisfaction questions or treat courtesy responses as independent outcome evidence.

Analysis 7: Follow the synthetic phase with shadow work using redacted or tightly controlled live cases where policy permits

Follow the synthetic phase with shadow work using redacted or tightly controlled live cases where policy permits. The assistant prepares the result without executing it, while an authorized owner processes the case independently. Compare source capture, questions, decisions reached by owners, and unexpected context. Shadowing narrows the gap between a laboratory case and real operations without granting full authority immediately. The record should identify the population, observation window, system, responsible owner, and unavailable evidence. This keeps a numerical result from implying broader certainty than the design supports. Run a sensitivity check by changing reasonable cutoffs, risk bands, and treatment of unresolved items. Identify conclusions that persist and those that depend on policy choices. Sensitivity analysis does not authorize choosing the convenient result; it exposes the assumptions requiring owner judgment.

Analysis 8: Set explicit launch gates

Set explicit launch gates. Representative cases must preserve provenance, protect restricted information, respect stop rules, reach correct owners, and produce results that reviewers can reproduce. High-risk failures require remediation and retest regardless of the overall score. Define initial production volume, review percentage, rollback method, incident route, and the conditions that suspend the lane. Preserve source facts separately from local interpretation. A virtual assistant can prepare the comparison and exception file; authorized security, legal, finance, HR, clinical, safety, or executive owners retain decisions in their fields. Maintain a corrective-action log with condition, evidence, risk, responsible owner, target date, execution reference, verification, and residual limitation. Mark an issue closed only when its defined test passes. A meeting, reminder, or accepted recommendation is not evidence that the operating state changed.

Analysis 9: Report the validity boundary honestly

Report the validity boundary honestly. A passed pilot supports the claim that the tested instructions and controls worked for the included scenarios at that time. It does not establish legal compliance, professional competence, fraud immunity, live-scale capacity, or future performance. Expand evidence gradually, monitor new exception types, and update the synthetic suite when systems, policies, threats, and work scope change. Apply least privilege and data minimization throughout the test. Store sensitive material only in approved systems, use stable references in the study file, and document access removal when temporary review ends. Schedule reassessment when volume, systems, owners, providers, policy, or threat conditions change. Archive the study version, sources, codebook, population receipt, exception file, and limitations under the organization’s retention rules so later reviewers can reproduce what was actually tested. The launch memo should map every tested scenario to the corresponding live control, identify risks not represented, and name the first production cases subject to complete review. Keep synthetic success, shadow success, and production authorization as separate gates. If tooling, owner availability, data sensitivity, or work scope differs from the test environment, record the delta and retest the affected control before relying on the earlier result.

Does this study report service performance?

No. It proposes a bounded method and reports no observed company, assistant, worker, or customer outcomes.

Who retains consequential decisions?

The business and its qualified, authorized owners retain decisions in legal, financial, clinical, safety, security, employment, tax, and executive fields.