AI Interview Scorecard and Questions: A How-To Guide

Most small-business hiring goes sideways before the first interview, because nobody wrote down what "good" actually means. You have a fuzzy picture in your head, you interview on gut feel, and six months later you are surprised the person is underperforming against a standard you never said out loud. A hiring scorecard fixes that. It is a one-page document that defines a role by the results it has to produce, the few skills that actually predict those results, and a set of interview questions that score every candidate against the same bar. This guide walks through how to build one, and how an AI assistant can do the heavy lifting while you stay the one deciding.

Start with the outcome, not the job title

The biggest mistake is writing a duties list. "Answer the phones, manage the calendar, handle email" describes activity, not success. Instead, ask one question: if this person nailed the job, what would be true in six to twelve months that is not true today?

Write four to six outcomes, each one a result you can measure or clearly recognize. Rank them. The top one is the result that, if missed, makes the hire a failure no matter what else they did well.

For a real estate brokerage owner drowning in scheduling, a strong outcome looks like: "By month 3, I spend under 2 hours a week on scheduling and inbound calls, down from about 15." Compare that to "manage scheduling." One is a finish line you can check. The other is a vibe. Numbers and dates are what separate the two.

Cut the must-have list down to four to six skills

Once you have outcomes, ask of each one: what must a person reliably be able to do for this to actually happen? You will generate a long list. Now cut it hard.

A real must-have is something that screens people out, and that not every candidate already has. If everyone you would ever consider already has it, it is not helping you choose, so drop it. Keep four to six. More than that and you have written two jobs into one. Separate must-haves (which reject a candidate) from nice-to-haves (which break ties). For the office manager above, the short list might be operational follow-through, calm communication with upset customers, scheduling judgment, and self-direction. Every skill should trace back to at least one outcome. If it maps to none, it does not belong on the page.

Write questions that test for those skills

Decorative questions ("Where do you see yourself in five years?") tell you nothing. Tie every question to a specific skill, and ask for a real past example rather than a hypothetical. "Tell me about a time you owned a process where things were slipping through the cracks" is harder to fake than "What would you do if things were slipping?" Past behavior predicts future behavior; made-up answers fall apart under a follow-up.

For each question, add a probe and rating anchors. A probe is the follow-up that separates a rehearsed story from a real one: "What did you personally do?" or "What was the result, in numbers?" Anchors are a short 1-to-5 scale describing what a weak, adequate, and strong answer sounds like, so two people scoring the same answer land in the same place. Add one outcome-pressure question that puts the hardest part of the actual job in front of the candidate: "I do all scheduling and phones today. How would you take that over so I am down to 2 hours a week by month 3?" You are watching how they think, not just what they have done.

Decide the bar before you meet anyone

Write your hire-or-reject rule in advance. Something like: "Hire only if follow-through scores 4 or higher, no must-have scores below 3, and the overall average is at least 3.5." Setting the threshold before the first interview is what stops you from talking yourself into a likeable candidate who cannot do the job. Then run a quick fairness pass: every question should be job-relevant, every candidate should get the same core questions and anchors, and "culture fit" should be replaced with observable behaviors like "gives direct feedback" or "follows through on commitments." This is a structuring tool, not legal advice, so have your actual hiring practices reviewed by qualified counsel.

Let an AI assistant do the draft, you do the judgment

Building this from scratch every time is slow, which is why most owners skip it. An AI assistant is good at the mechanical part: take your rough sentence, expand it into ranked outcomes, propose the four-to-six skills, and draft the questions with probes and anchors. You read it, cut what is wrong, sharpen the numbers, and approve it. Nothing gets used until you say so.

If you want a guided version that produces the full one-page scorecard and matching question set from a single description of the role, the AI interview scorecard and questions skill walks through it step by step.