Philippines staffing research · Published October 22, 2026

Measuring Inbox Triage Accuracy Without a Clean Baseline

A practical protocol for classifying messages, testing priority decisions, and reporting accuracy with honest denominators.

On this page

Methodology

Bounded method review with 4 authoritative sources, 6 analysis sections, and 0 service performance claims.

Key Stats

4: authoritative sources
6: analysis sections
0: service performance claims

Key Takeaways

Build a small taxonomy
Create a gold set
Score the triage

Build a small taxonomy

Use two dimensions: sender group and request type. Add a priority flag only for conditions that are defined in advance, such as a legal threat, a safety issue, a board deadline, or a payment failure. Keep the categories few enough that a reviewer can apply them consistently. Write a definition and two examples for each category. Include the boundary cases: a vendor asking for a meeting is not a customer complaint; a newsletter with an urgent subject line is not urgent. The taxonomy should make the easy cases obvious and the hard cases discussable.

Create a gold set

Have the executive or an experienced chief of staff classify a sample of real or realistic messages. Use that set as the reference, but review it for internal consistency. If the owner disagrees with their own earlier labels, the definition needs work. Protect the messages and remove unnecessary personal data. The gold set should be small enough to review and large enough to cover the main categories. Keep the source date and the taxonomy version. A gold set that changes every week cannot support a stable measure.

Score the triage

Ask the VA to classify each message, choose the destination, and write a one-line reason. Score category accuracy, priority accuracy, destination accuracy, and the quality of the reason. A correct label with no reason is harder to review and easier to fake. Report confusion between categories, not only overall accuracy. If legal messages are often labeled as customer complaints, the fix is a definition and a training example. If urgent messages are under-flagged, the priority rule may be too vague. Use the error pattern to repair the taxonomy.

Sampling and reporting

Report the sample size, the categories represented, the gold-set version, and the cases the reviewers could not classify. Show category-level confusion and the reason for each material error. A single accuracy percentage can hide a dangerous pattern, such as consistent under-flagging of legal or safety messages. Repeat the test with new messages after a taxonomy change. Keep the earlier results separate rather than pooling them with the new version. If the VA improves only on the familiar cases, the result may reflect memory rather than skill. Use fresh cases for the decision.

Limitations

The gold set reflects one owner's judgment and may not generalize. A small sample can miss rare but important categories. The VA may improve because the test cases become familiar. Repeat with new cases and keep the old results separate. Triage accuracy also depends on the inbox design, the sender data, and the time available. A low score may indicate a broken intake rather than a weak assistant. Report the conditions beside the score.

Decision use

Use the result to decide whether the VA can work the inbox with a sample review, needs a narrower set of sender groups, or should stay in a draft-only lane. Do not use a single accuracy percentage to make a broad claim about the person. Retest after a change in taxonomy, mailbox volume, or executive preferences. Keep the gold set and the scoring notes so the next reviewer can reproduce the measure. If the executive cannot agree with the gold labels, the issue may be a values conflict rather than a skill gap. Resolve the standard before scoring the assistant, and record the decision so the next reviewer is not asked to guess which version of the inbox rules applies.

Sources checked October 22, 2026. Method protocol only; no provider performance claim.

Sources

U.S. GAO, Assessing Data Reliability: Public guidance supporting the protocol.
NIST Privacy Framework: Public guidance supporting the protocol.
W3C WAI, Use Clear and Understandable Content: Public guidance supporting the protocol.
Google Site Reliability Engineering Workbook, Postmortem Culture: Public guidance supporting the protocol.

FAQs

Plan the next step

Translate the method into a bounded role brief with named review points.

Review virtual assistant services

Read the daily blog guides · Explore service workflows · Plan your staffing routine

Use the method to define the population, evidence, and reviewer before testing.