How to design an AI skills test for hiring: work sample, rubric and legal limits

An AI skills test for hiring should be a work sample. Give each candidate a real task from the job, the AI tools the job uses, and a time limit of 30 to 60 minutes. Score the output and the checks they made against a fixed rubric. A quiz on AI terms tells you who has read about AI. A work sample shows who can use it to do the job.

This guide covers the design, the rubric and the fairness and legal limits. It focuses on US and EU rules. This is general information, not legal advice.

Step 1: Start from the job

List the tasks in the role where AI is used, or will be. Ask people who do the job now. Pick the two or three tasks that matter most and come up often. The test should sample those tasks, not AI skill in general.

This step is also your legal footing in the US. The Uniform Guidelines on Employee Selection Procedures accept content validity as one form of evidence. That means showing the test's content reflects important parts of the job (29 CFR 1607.5). A test built from a written list of real job tasks is easier to defend than one built from general AI knowledge.

Step 2: Build the task

• Use realistic inputs: a messy set of notes, a spreadsheet, a customer email thread. Remove any real personal or confidential data.
• Plant one problem the candidate should catch. Examples: a figure in the source that contradicts another, a request that needs data the candidate does not have, or a draft with an invented citation.
• State which AI tools the candidate may use. Ideally use the same tools as the job. If candidates use their own, allow any mainstream tool.
• Ask for two deliverables: the finished output, and a short note on what the candidate checked and changed.
• Set a time limit. Keep the whole task under 60 minutes, so the test does not become unpaid work.

For example, a marketing role might get three product notes and a request for a launch email. One note gives a wrong price. A strong candidate uses AI to draft quickly, spots the price conflict, and flags it rather than guessing. Emeri’s AI interview questions by role offer follow-up questions to ask about the work.

Step 3: Write the rubric before anyone takes the test

Use four or five criteria. Score each from 1 to 4. Write a short description of what each score looks like, so two reviewers reach the same number. A workable set:

• Output quality: does the finished work meet the standard of the job? A 4 could go out with no edits. A 1 needs rewriting.
• Accuracy and checking: did the candidate catch the planted problem and verify facts against the inputs? A 4 caught it and explained how. A 1 passed it through.
• Use of context: did the candidate give the AI tool the right inputs and constraints? A 4 supplied the relevant material and limits. A 1 used a one-line request.
• Judgment and data care: did the candidate keep sensitive details out of places they should not go, and flag what they could not resolve?
• Explanation: could a manager follow the candidate’s note and trust the result?

The AI competency levels guide gives level descriptions you can borrow for the score anchors.

Step 4: Pilot it on current staff

1. Ask five to ten people who do the job well, and a few who are newer, to take the test.
2. Score them with the rubric. Strong performers should score higher. If they do not, the task or the rubric is measuring something else.
3. Time each person. Cut the task if most need more than the limit.
4. Check the instructions. Any question a pilot taker asks should be answered in the instructions.

Fairness: check for adverse impact

In the US, the Uniform Guidelines use a rule of thumb for adverse impact. Compare the selection rate of each race, sex or ethnic group with the rate of the group that does best. A rate below four-fifths of the highest is generally treated as evidence of adverse impact (29 CFR 1607.4). Track pass rates by group once you have enough candidates. If a gap appears, review the task and rubric for content that is not tied to the job.

Access to AI tools varies. A candidate who has never had a paid AI account may be slower at the start. Allow a few minutes to get familiar with the tool, or supply it, so the test measures skill rather than past access.

Accommodations for disability

The Americans with Disabilities Act bars selection tests that screen out people with disabilities unless the test is job-related and consistent with business necessity. It also requires tests to be given so the results reflect the skill being measured, not a sensory, manual or speaking impairment (42 U.S.C. 12112). Tell candidates in advance what the test involves and how to ask for an accommodation, such as extra time or a screen reader.

If AI scores the test

Some employers use an AI tool to grade candidate work. That can bring extra rules.

• New York City: Local Law 144 bars employers from using an automated employment decision tool unless it had a bias audit within the past year. Employers must publish a summary of the audit and notify candidates. The city began enforcing the law on July 5, 2023 (NYC Department of Consumer and Worker Protection).
• European Union: the AI Act lists AI systems used to evaluate candidates as high-risk (AI Act Service Desk). The obligations for those systems apply from December 2, 2027 (AI Act Service Desk).

A human reviewer scoring against a rubric avoids most of these questions. If you do use AI to help grade, keep a human in charge of the decision and record how scores were produced.

Mistakes to avoid

• Testing vocabulary. Knowing what a token is does not show that a person checks AI output.
• Rewarding clever prompts. The prompt is a means. Score the output and the checks.
• Long take-home projects. They cost candidates time, and they can favor people with free evenings.
• Changing the task between candidates. Comparable scores need the same task and rubric.

Using an external assessment

Some employers add a standard assessment so they can compare candidates across roles. Emeri’s certification scores people on real work tasks and reports a band from E1 to E4. The employers page explains how to read a band, and the methodology explains how grading works. A role-specific work sample still tells you the most about fit for the job. For checking a credential a candidate already holds, see how to verify a candidate’s AI certification.