Artificial Intelligence & Automation · 40 checks
AI production readiness checklist
Use this before an AI system goes in front of users. It covers the work that separates a prototype from something a team can depend on. Nothing here is specific to a vendor, a model or a framework, and you are free to copy it into your own repository and adapt it.
Most of the items below are unglamorous and most get scoped last. That is the reason production launches slip. Working through them before the launch date is set is considerably cheaper than discovering them afterwards.
Interface contract
Every team downstream is blocked until these are fixed. Settle them before tuning anything.
- Input schema is defined and validated at the boundary, with malformed input rejected rather than coerced.
- Output schema is defined, including the shape returned when the system declines to answer.
- Latency budget is agreed with whoever consumes the output, and the modeling approach fits inside it.
- Error taxonomy exists: the caller can distinguish "no answer", "low confidence", "upstream failure" and "invalid request".
- The contract is versioned, and there is a stated policy for breaking changes.
Failure behavior
Production systems are judged on what they do when they are wrong. Design this before the happy path.
- Behavior on low confidence is defined: fall back, escalate to a person, or decline and say so.
- Behavior when an upstream dependency times out or errors is defined and tested.
- Fallback path exists and has been exercised, not just written down.
- Degradation is graceful: one component misbehaving reduces capability rather than stopping the workflow.
- Rate limits and cost ceilings are enforced, so a loop or a traffic spike cannot run unbounded.
- There is a documented answer to "what does a user see when this fails".
Evaluation
If you cannot rerun the evaluation, you cannot safely change anything afterwards.
- An evaluation set exists, is version controlled, and lives alongside the code.
- It covers the awkward cases, not only the representative ones.
- Every case that has caused an incident has been added to it.
- It runs automatically when the model, prompt, retrieval or pipeline changes.
- A pass threshold is agreed in advance, so the result is a decision rather than a discussion.
- Results are recorded over time, so regressions are visible as a trend.
Observability
Conventional monitoring tells you the service responded. It does not tell you the response was sensible.
- Input and output distributions are logged, so drift is visible against a baseline.
- Confidence scores are recorded where the method produces them.
- Fallback rate and human escalation rate are tracked. These usually move before anyone complains.
- Latency is tracked at the percentiles that matter, not only the mean.
- Cost per request is visible and attributable.
- Alerting fires on rate of change, not only on absolute thresholds.
- Logs are retained long enough to investigate an incident reported a week later.
Data and privacy
Usually the item that delays a launch when it is discovered late.
- It is documented what data leaves your infrastructure, and where it goes.
- Personal data handling has a lawful basis, and retention is defined.
- Prompts and outputs containing personal or confidential data are excluded from logs, or redacted before they are written.
- Vendor terms have been read on whether submitted data is used for training.
- Deletion requests can be honored, including anything captured in logs or evaluation sets.
Rollout
Each stage exists to surface a different class of problem.
- Internal use against real inputs, with no external consequence.
- Limited cohort with human review on every output.
- Wider cohort with review on flagged cases only.
- General availability with monitoring and an unambiguous rollback path.
- Rollback has been tested, not assumed.
- Feature flag or equivalent allows disabling without a deploy.
Ownership
An unowned system becomes an unmonitored one, and an unmonitored system fails silently.
- A named person owns it, not a team inbox.
- There is a runbook covering the three most likely failures.
- Someone other than the original author has run the deploy and the rollback.
- Retraining or prompt revision has a defined trigger and a defined approver.
- A review date is in the calendar, so nobody has to remember to look.
If most of this is already in place, the remaining work is probably smaller than it feels. If most of it is not, the gap between the prototype and the launch date is larger than the plan currently says, and it is better to know that now.
Published by Talavishiraaj Groups. Copy it, adapt it, put it in your own repository. No attribution required.
Related
If working through this has surfaced more gaps than you expected, that is usually the point at which it is worth a conversation.
Start a conversation