17
Right
Sample pilot scorecard
25 questions your team actually asks. Graded.
This is what you get at the end of a pilot. The company is fictional and every result below is made up to show the format. Real scorecards use your questions, your answer-owners and your records.
ILLUSTRATIVE SAMPLE · Sample Co. (fictional home services firm) · not a client result
4
Partly
2
Wrong
2
Not in records
17 of 23 graded questions right. The pass line agreed in week one for this sample was 16 of 23, so this sample pilot passes. The two "not in the records" items become a short to-do: write them down once.
| # | Question | Answer-owner | Result | Why |
|---|---|---|---|---|
| 01 | What is our warranty on a water heater install? | Office manager | Right | Matched the warranty sheet in the shared drive and linked it. |
| 02 | Which supplier gives us net-30 terms? | Owner | Right | Found in the vendor agreement folder. |
| 03 | What did we quote the Garza job in March? | Sales lead | Right | Pulled the quote PDF from the CRM attachment. |
| 04 | How do we handle a callback inside 30 days? | Service manager | Partly | Gave the old policy. The new one was only in an email thread that was not connected. |
| 05 | Who is our contact at the city permit office? | Office manager | Right | From the contacts sheet. |
| 06 | What is the after-hours rate? | Owner | Right | From the 2026 price list. |
| 07 | Which techs are certified for gas line work? | Service manager | Wrong | Listed a tech who left in June. The roster file was out of date. Fix: one roster, one owner. |
| 08 | What did we promise the Lee family about the reschedule? | Dispatcher | Partly | Found the text, missed the follow-up call note in the CRM. |
| 09 | How many leads came in from the website last month? | Owner | Right | From the CRM report, with the link. |
| 10 | What is our standard payment schedule for jobs over $10k? | Owner | Right | From the terms page of the proposal template. |
| 11 | Where is the safety checklist for attic work? | Service manager | Right | Linked the checklist. |
| 12 | What is the return policy at our main parts supplier? | Purchasing | Not in records | Never written down. Marked not in the records, not counted as a miss. |
| 13 | Which customers are due for annual maintenance this month? | Dispatcher | Right | From the service agreement list. |
| 14 | What did the insurance adjuster ask for on the Patel claim? | Office manager | Right | From the email thread, linked. |
| 15 | How do we price a same-day emergency call? | Owner | Partly | Right base rate, missed the weekend add-on in a separate memo. |
| 16 | Who approved the discount on the Moore job? | Sales lead | Right | From the CRM note. |
| 17 | What is the onboarding checklist for a new tech? | Office manager | Right | Linked the checklist. |
| 18 | What is our vehicle maintenance schedule? | Owner | Not in records | Lives in the fleet manager's head. Marked not in the records. |
| 19 | Which permits are open right now? | Office manager | Right | From the permit tracker sheet. |
| 20 | What did we tell the HOA about noise hours? | Owner | Right | From the email, linked. |
| 21 | What is the markup on special-order parts? | Purchasing | Wrong | Read a draft price sheet instead of the final one. Fix: archive drafts. |
| 22 | How do we book a crane for a rooftop unit? | Service manager | Right | From the vendor notes. |
| 23 | What is the deposit on a full system replacement? | Owner | Right | From the proposal template. |
| 24 | Which reviews have we not replied to? | Marketing | Partly | Found the review site export, but it was two weeks old. |
| 25 | What is our policy on customer-supplied parts? | Owner | Right | From the policy doc. |
Why it matters
The misses are the useful part.
- Most wrong answers trace back to a stale or duplicate file, not the AI. The scorecard tells you which file.
- "Not in the records" shows knowledge that lives only in someone's head.
- You decide keep, change or stop based on a number you agreed to up front.
Twenty minutes to see if the audit fits.
Tell me which workflow eats the most of your team's week. I'll tell you honestly whether an audit is worth it.