Pilot to production in four weeks

A pilot can look convincing while leaving the operating questions unanswered. Who checks an ambiguous result? What happens when a connector fails? Does the time saved survive the review queue? Who can turn the system off?

A four-week delivery window makes those questions concrete. It can be a useful planning constraint for one bounded workflow when access, people, and approvals are already available. It is not a delivery guarantee, and it is rarely enough for a broad transformation or a consequential autonomous decision system.

The intended outcome is modest and valuable: a limited production service with a named owner, evidence that it helps, and a tested way back to the existing process.

Before the clock starts: choose a finishable unit#

“Automate accounts payable” is too large. “Prepare an exception brief for invoices missing a purchase-order match” is a workable starting point. The assistant retrieves permitted records, identifies missing evidence, and drafts a brief. A person verifies it and decides what happens next. Payment execution stays outside the scope.

That example is illustrative throughout this article; the schedule and thresholds below are planning examples, not reported client results.

Before starting, secure approved data access, a test environment, an integration path, representative historical cases, and time from the people who do the work. Name a process owner who decides whether the output is useful and a technical owner who operates the service. Reserve the relevant security and data reviews. A four-week plan cannot absorb an unknown procurement cycle or manufacture permission to use sensitive records.

Write a one-page delivery contract: eligible cases, excluded actions, expected output, review responsibility, success measures, operating limits, and release authority. Assign ownership for both the launch decision and the service that follows it. The people responsible for monitoring, incidents, and future changes should be identified before the first release.

Week one: measure the work people actually do#

Sit with operators as they handle ordinary and difficult cases. Watch the switches between systems, the follow-up questions, and the decisions they make without writing them down. Record where the process begins and where responsibility transfers.

Measure active handling time separately from elapsed turnaround time. An assistant may reduce drafting effort while the case still waits two days for approval. Capture rework, escalation rates, backlog age, and the quality of the completed outcome. Preserve a manual baseline for comparable case types and volumes.

Build an evaluation set from authorized historical cases, including missing records, conflicting information, duplicates, and unusual document formats. Have domain reviewers agree on what a correct brief contains and what constitutes a critical error. Split development examples from a held-out test set before tuning prompts or retrieval. Keep related cases together so duplicates do not leak across the split.

Use development cases to improve the system and a separate test set to assess the candidate you intend to release. If every evaluation example has already influenced the prompt or implementation, the result tells you too little about performance on unfamiliar cases.

The week-one gate is an approved scope and scorecard, a recorded baseline, and a reviewed case set. If experts cannot agree on correctness, resolve that disagreement before asking a model to reproduce it.

Week two: build the whole path, then evaluate it#

Connect the smallest useful flow from input to reviewed output. Include authentication, retrieval, evidence references, the reviewer’s screen, and the handoff back into the process. A polished answer in a separate chat window does not establish that the workflow works.

Keep business rules outside free-form generation where possible. Validate required fields, eligible case status, and record identifiers in ordinary application logic. Treat retrieved documents as evidence, not instructions that can grant new permissions. Give the service only the access its bounded task requires.

Freeze a release candidate, including its model, prompt, retrieval settings, and connector versions. Run it against the held-out cases. Review correctness, evidence support, useful abstentions, and performance by case category. If a failed test informs another round of tuning, use fresh held-out cases for the next final assessment.

For this illustrative invoice workflow, a team might require at least 95% of briefs to meet its completeness rubric, every factual assertion to have a verifiable record reference, and no critical failures in the agreed test suite. Those are example gates to negotiate with the process owner. Zero observed critical failures is a release condition, not proof that the real-world failure rate is zero. Record the sample size and uncertainty.

Test expired credentials, empty search results, timeouts, and repeated submissions as well. The gate requires both acceptable task performance and demonstrated controls: failures reach a person, permissions hold, and retries cannot create duplicate work items.

Week three: observe live work without changing its outcome#

Run the candidate beside the existing process. It receives authorized live inputs and produces a separate output, but its result does not trigger business actions. Operators continue working normally; reviewers compare the assistant’s evidence and proposed brief with the completed case.

This is shadow mode: the candidate sees live work, but the existing process remains responsible for the outcome. Disable every side effect, including notifications, record updates, and follow-up tasks. A shadow run should not quietly create extra work in another system.

Look for problems the historical set missed: stale records, sudden bursts of cases, changed field names, and reviewers who cannot quickly verify a citation. Track the full cost per completed eligible case, including model calls, retrieval, retries, and human review. Shadow mode estimates potential workflow value; it does not yet demonstrate realized time savings.

Agree on an observation window and minimum case count before starting. For example, a team might require five business days and 100 eligible cases, whichever takes longer. These are illustrative planning limits; low volumes or rare exceptions may require more time than the calendar allows.

Rehearse disabling the assistant, routing new work to the manual queue, and reconciling outstanding cases. The gate is acceptable live-input behavior, no unresolved critical issue, and a recovery rehearsal completed by the actual operator.

Week four: release a small service people can support#

Enable a limited cohort: one team, one queue, or a capped share of eligible cases. Keep human approval before consequential actions. Show reviewers the supporting evidence, uncertainty, and an easy way to reject or edit the draft.

Compare outcomes with the baseline and, where feasible, a contemporaneous manual cohort of comparable cases. Measure handling time including review, rework, missed exceptions, queue age, and cost. A faster draft is useful only if the completed process improves.

Give the limited release a defined cohort, an evaluation period, and explicit stop conditions. Expand only after reviewing the results. A small launch creates room to discover operational problems while keeping the affected workload manageable.

Release approval belongs to named people: the process owner accepts usefulness, the technical owner accepts operational readiness, and the relevant control owners accept the scoped data and access arrangements. Hand over a runbook with alerts, escalation contacts, version history, manual routing, and incident response steps.

Define rollback precisely. Disabling new runs does not undo an external change already made. Track each work item and its approval history so the owner can reconcile partial actions. For an illustrative limited release, any unauthorized access, cross-customer data exposure, or duplicate external write triggers immediate suspension and investigation. Quality or latency breaches trigger the agreed pause or rollback procedure.

The calendar does not overrule the evidence#

At the final review, choose among continuing the limited service, expanding it, extending observation, narrowing scope, or stopping. Stop when review consumes the expected savings, the evidence cannot be verified, critical failures remain unresolved, or nobody can own the service after delivery.

Four weeks is useful because it forces a decision about one operational slice. A credible result includes the people, evidence, controls, and recovery path needed to keep that slice working after the delivery team leaves.

Related reading#

• Why we watch the work before we automate it
• MCP in the enterprise: a field guide