# Shipping Reliable AI: companion kit 1.0.0

An offline learning lab by Qentra. Book author: qentra.tech.

Start without installing anything: https://qentra.tech/books/shipping-reliable-ai/resources

This kit uses **fictional, template-generated data and scripted responses**. It does not call an AI model, access a customer system, or establish production quality. A green result means only that the bundled response passed the bundled teaching checks. Never paste customer records or API keys into public demos.

## First success, without a terminal

1. Open the public companion. Choose **Generate practice data**. You get 40 fictional cases across five check families.
2. Select **Run baseline**. All 40 intentionally fail. Open a case and read the input, actual response, rule and reason. A red result here is expected, not a setup error.
3. Select **Run improved version**. All 40 pass the same teaching rules. Compare the two reports; no rule was loosened.
4. Select **Introduce a regression**. Eight refund cases fail again. This demonstrates why an overall score alone is insufficient.
5. Download the dataset as JSONL (one JSON object per line), CSV (spreadsheet-friendly), and the result report as JSON. Downloads stay on your device. Reloading clears the lab state.

The page's **browser terminal** shows real results computed by the bundled JavaScript checks. It is not an operating-system shell or a Python interpreter. It cannot open your device terminal or run arbitrary pasted code. The Python path below runs the equivalent teaching workflow locally; parity is checked against the browser engine.

## Local path: Python 3.12

The reference environment is Python **3.12.14**. There are no third-party runtime packages. Python installation is optional for the browser route.

1. Download and unzip `supportly-companion-v1.0.0.zip`. Open the extracted folder containing `README.md` and `scripts/`.
2. Open a terminal **in that folder**. Windows: type `powershell` in File Explorer's address bar. macOS: Terminal, type `cd `, drag the extracted folder into the terminal, then Return. Linux: file manager → Open in Terminal. A terminal is a text window that runs commands on your computer.
3. Check Python: Windows `py -3.12 --version`; macOS/Linux `python3 --version`. If Python is missing, use the browser lab or install it from https://www.python.org/downloads/. Do not install an unrelated package called `python` with pip.
4. Run one line at a time. Windows: replace the leading `python3` below with `py -3.12`. Use a new, empty extracted folder to avoid overwriting your own reports.

```sh
python3 scripts/generate_data.py --output data/dataset.jsonl --count 40 --seed 42
python3 scripts/run_evals.py --dataset data/dataset.jsonl --variant baseline --report reports/baseline.json
python3 scripts/run_evals.py --dataset data/dataset.jsonl --variant improved --report reports/improved.json
python3 scripts/compare_reports.py reports/baseline.json reports/improved.json --output reports/comparison.json
python3 -m unittest discover -s tests -v
python3 scripts/metrics_demo.py
```

Expected: data generation succeeds; baseline prints **0/40 passed** and exits 1 intentionally; improved prints **40/40 passed**, exit 0; comparison prints **0 new failures / 40 new passes**, exit 0. Do not chain baseline to the next step with `&&`, because its expected exit 1 stops the chain. The unit tests verify known failure paths as well as passes.

Then demonstrate a release-blocking change:

```sh
python3 scripts/run_evals.py --dataset data/dataset.jsonl --variant regressed --report reports/regressed.json
python3 scripts/compare_reports.py reports/improved.json reports/regressed.json --output reports/regression.json
```

Both commands exit 1: eight refund cases are now failing. That is the intended regression, not a broken Python environment.

## If something goes wrong

| Symptom | Meaning | Recovery |
|---|---|---|
| `python3` / `py` not found | Python isn't installed or on PATH | Use the browser route or install official Python; reopen the terminal. |
| Can't open `scripts/...` | Terminal is in the wrong directory | Open the extracted folder containing README and scripts, then reopen the terminal there. |
| Exit 1 on baseline/regressed | The teaching checks found expected failures | Read reasons and run improved; keep the checks unchanged. |
| Exit 2 / JSON parse error | Missing, malformed or unsupported data | Regenerate the fictional dataset. JSONL must have one complete object on every physical line. |
| Reports cannot be compared | Dataset or check versions differ | Rerun both variants against the same dataset and kit version. |
| Copy/download blocked by browser | Browser permission or download policy | Select code manually; try the direct ZIP link. The page reports failure rather than claiming success. |

## What is here

- `scripts/generate_data.py`: deterministic labelled fixture generation, bounded count and seed.
- `scripts/run_evals.py`: refund-window, escalation, email-pattern, retrieval and same-order refund-trace checks.
- `scripts/compare_reports.py`: paired case diffs; rejects mismatched datasets/checks.
- `scripts/metrics_demo.py`: synthetic confusion matrix, Cohen's kappa and retrieval edge cases. These labels are not real expert annotations and no live judge is calibrated.
- `scripts/supportly.py`: small check functions; regression tests cover duplicate retrieval IDs, empty relevance labels, cumulative refunds, non-finite amounts and mismatched orders.
- `.github/workflows/companion.yml`: example CI gate for a repository with this kit at its root. GitHub-hosted CI was not executed by downloading the kit; review permissions before enabling it.
- `tests/browser-parity.mjs`: optional Node.js parity check (`node tests/browser-parity.mjs python3`); Node is not needed for the Python lesson.
- `browser/demo-engine.mjs`: browser implementation used for parity checks. The web page runs only this fixed demo, not arbitrary user scripts.

The richer input/expected/grading schema in chapter 4 is an application-design example, not this fixed kit schema. Use the generator for this runner; adapting your own schema requires a separately tested adapter.

The refund trace illustrates a narrow invariant, not complete authorization, tenant isolation, idempotency or a refund service. The email pattern does not detect all sensitive information. The six-label judge example is a calculator exercise, not evidence of reliability. No checkout, payment or real refund is performed.

## Move from practice to a real application

Read chapters 4–9 and 13. With permission, curate redacted domain-expert cases, define policy expectations, replace scripted responses with your adapter, add trusted authorization controls, and evaluate the adapter separately. Keep a genuine holdout separate from practice fixtures. Review each high-impact failure rather than hiding it in an average.

Code and synthetic fixtures: MIT (`LICENSE`). The book manuscript and brand assets are not included in that license. Version 1.0.0; issue contact support@qentra.tech. Send reproduction steps and fictional examples, never credentials or customer data.