AI Fluency Assessment for Hiring In-House Counsel

To assess AI fluency when hiring in-house counsel, give candidates a realistic legal task and ask them to use AI to complete it. Review how they prompt, check sources, refine the output, and explain their recommendation.

GC AI is the legal AI software built for in-house counsel and used by over 2,200 legal departments, including Columbia Sportswear, TIME, Lockheed Martin, Riot Games, and Wayfair.

We’re hearing legal leaders rethink what they look for in a new hire: curiosity about AI, the ability to use it in their work, and judgment about its output. On CZ and Friends, our CEO and three-time general counsel Cecilia Ziniti interviews legal and business leaders about how they build teams and use AI.

For a counsel role, that might mean reviewing a vendor agreement and explaining whether Procurement can move forward. The fictional exercise below gives you a company brief, agreement excerpts, candidate instructions, and a rubric for discussing the work.

Define What the AI Fluency Assessment Should Show

Tina Hwang, Head of Legal at ClassDojo, described what she looks for when hiring:

I really look for that baseline curiosity

She described looking for lawyers willing to try an AI workflow and adapt how they work. Hwang is also a limited partner in The Council, an early GC AI investor.

Choose a task the person would need to perform when entering the role. In this example, counsel will review a vendor agreement, advise Procurement, and coordinate with Finance and security colleagues.

Assess the AI workflow as well as the final advice. Look at what the candidate asks AI to do, the company context they provide, how they refine an unhelpful response, and which conclusions they check or resolve themselves.

The U.S. Office of Personnel Management’s guidance on work samples supports choosing tasks that resemble the job and assess competencies expected at entry. It also calls for trained evaluators.

Before using this exercise in hiring, have HR and employment counsel review its relevance to the role, how you will administer it, accommodations, and the evaluation method. The EEOC’s guidance on employment tests and selection procedures addresses job-relatedness, validation, disability accommodations, and discriminatory effects. Use the proposed materials as a starting point for that review.

Give candidates the same packet, instructions, deliverables, and baseline time allowance, with appropriate accommodations. Arrange approved access to GC AI and provide a common orientation. Keep this document-based assignment within the supplied materials, and ask candidates to flag questions that would require further legal research.

Lawyers who want more practice can use GC AI Academy’s AI courses for legal professionals.

• Our Intro to AI for In-House Legal assumes no prior AI knowledge.
• Advanced AI for In-House Legal suits lawyers with some experience or introductory training.

Give candidates the same preparation guidance, then assess how they handle the assignment.

Build a Synthetic Vendor Review Packet

Prepare two source files: “Synthetic Company Brief” and “Synthetic Vendor Agreement Excerpts.” Give candidates both files, the assignment, tool instructions, and rating criteria. Keep the calibration prompt, illustrative findings, and example recommendation in a separate evaluator packet.

These invented materials support a limited triage assignment. They do not form a complete contract. Ask candidates to identify issues in the packet and flag what is missing before reaching a conclusion about the complete agreement or its enforceability.

Synthetic Company Brief

Juniper Vale is considering Cinderline Analytics for an internal agreement-intake service. Legal and Procurement expect 40 users over an initial 12-month term.

Supply these company positions:

• Budget: Finance has authorized a maximum first-year spend of $180,000, including setup. Additional spending requires Finance approval.
• Data use: The proposed service would process agreements and business contact information. Juniper Vale’s position permits use only to provide and maintain its service. Any broader use requires separate Legal approval.
• Security: The security lead must review the vendor’s safeguards before any company data enters the service.
• Liability: Legal’s opening position is a general cap of at least 12 months of fees paid or payable. Legal must separately assess the treatment of confidentiality and security claims.
• Exit: Procurement needs a usable export before access ends, with a defined format and known cost.
• Ownership: Procurement owns the vendor relationship, Finance owns budget exceptions, and Legal owns contract recommendations. The packet grants the candidate no approval or signing authority.

For the exercise, authorize only the supplied synthetic files in the designated environment. Real employer, client, and personal materials are outside the permitted source set.

Synthetic Agreement Excerpts

Clause	Vendor’s Proposed Language
1. Fees	“Annual subscription fees are $174,000. A one-time implementation fee of $20,000 is due upon execution. Fees are payable annually in advance.”
2. Renewal	“The initial subscription term is 12 months from the effective date. The subscription renews for successive 12-month terms unless either party gives notice at least 60 days before expiration. Renewal fees will be based on Provider’s then-current list price.”
3. Customer Data	“Provider may use Customer Data, including uploaded agreements, business contact information, and support tickets, to deliver, maintain, and improve the services. Provider may retain Customer Data for 90 days after termination.”
4. Security	“Provider will maintain the safeguards described in Schedule A.” Schedule A is absent from the packet.
5. Liability	“General liability is limited to fees paid in the three months preceding the event giving rise to the claim. Liability for confidentiality or security breaches is limited to fees paid in the preceding 12 months.”
6. Exit	“Customer may request export of Customer Data for 60 days after termination, at Provider’s then-current rates.”

The missing schedule, undefined improvement use, and unspecified export terms leave the candidate with questions to resolve. Watch whether those gaps change the advice.

Give Candidates an AI-Assisted Legal Assignment

Ariana Goodell, Senior Counsel for Privacy and Policy at Verkada, described using AI to learn the technical concepts behind the products she advises on:

If you want to do lawyering, there's a baseline AI literacy you're going to have to have.

Tell candidates to use AI to help review the packet, draft proposed changes, and prepare their advice to Procurement. Let them decide which parts to delegate to AI and ask them to explain those choices. Their recommendation should state whether negotiations can continue and what must happen before signing or uploading company data.

Require five deliverables:

1. A short recommendation stating the proposed next step and its conditions.
2. An issue table with no more than six prioritized findings.
3. Proposed language for two provisions.
4. A brief message to Procurement explaining the required actions and owners.
5. A verification record showing material AI output the candidate retained, corrected, or rejected, with the reason.

Each issue-table row should identify the source quotation, clause reference, company position, risk, proposed edit, missing fact, and decision owner. Require the candidate to connect each recommendation to both the proposed contract and the company’s instructions.

Ask candidates to retain the prompts and responses they relied on. Agree in advance how you will review their process: prompt and response history, a live walkthrough, or an optional recording.

Ask the candidate to walk through an initial prompt, a response they relied on, and the source check that led them to retain, revise, or reject it. If they changed the prompt, ask what was missing and whether the new response helped. Let them explain what they handled through direct reading or their own reasoning, too.

Try the packet internally before setting the time allowance. Record where reviewers needed clarification and whether they could complete the deliverables in the intended time.

Use GC AI for the AI Fluency Test

You can use our 14-day trial to try the exercise with your team. Run the synthetic packet yourself, then arrange individual access for each candidate and give everyone the same orientation and practice task. Confirm access before scheduling the assessment.

With our Files feature, candidates can upload and organize the two source files, then attach them to the analysis. Have them confirm that GC AI has both the agreement excerpts and the company brief before they start.

Candidates should write their own instructions to GC AI. An evaluator can use the following example privately when calibrating the assignment:

Review the attached synthetic vendor agreement against the separate company brief. Advise Juniper Vale’s Procurement lead on the next step. Prioritize up to six issues. For each, identify the clause and exact supporting language, the company position, the practical risk, a proposed change, missing information, and the person who needs to decide or act. Keep contract terms, company instructions, and your inferences distinct. Flag missing documents and unresolved interpretations. Draft proposed wording for two provisions.

Watch how the candidate handles the first response. If it lists contract terms without applying the company brief, a useful follow-up would ask AI to compare each term with the relevant company position and explain the consequence. The candidate then checks whether the revision addressed the gap. Give credit for an accurate first response that they verify and retain, too.

Check the Passage and the Conclusion

Our Exact Quote feature opens a citation in the original document and highlights the supporting quotation. Candidates can use it to locate the text, then read the surrounding language and assess whether it supports the analysis.

In this packet, the liability clause refers to fees paid in the preceding three months, while the fee clause requires annual payment in advance. The candidate needs to read those provisions together. Treating the cap as automatically equal to one quarter of the annual subscription would introduce an assumption.

Ask the candidate what the citation establishes, what remains uncertain, and how their proposed edit addresses it. The ABA’s Formal Opinion 512 reinforces the importance of independently checking AI-assisted work and protecting confidential information.

Review a Recommendation You Can Trace to Evidence

Anirma Gupta, a strategic legal advisor and former Chief Legal Officer at Unity, described how AI changes the lawyer’s contribution:

the real value add for us lawyers is at the judgment layer

She connected that judgment to understanding the business, its risk profile, and the decisions it needs to make. In this exercise, the candidate has to turn the findings into advice that respects Juniper Vale’s budget, data-use limits, and approval responsibilities.

Use the following authored example to calibrate evaluators, and keep it outside the candidate packet. Its proposed negotiating positions still require the designated owners’ review.

Source Quote and Clause	Company Position	Risk	Proposed Edit	Missing Fact	Decision Owner
Clause 1: “$174,000” plus “$20,000”	First-year budget is $180,000, including setup.	The stated charges total $194,000, exceeding authorization by $14,000.	Negotiate total first-year charges within budget or seek approval for the excess.	Whether the vendor will reduce either charge.	Procurement negotiates; Finance decides any budget exception.
Clause 3: “deliver, maintain, and improve”	Use is limited to providing and maintaining Juniper Vale’s service.	“Improve” leaves the scope of additional use unresolved.	Limit permitted use to providing and maintaining the service for Juniper Vale; require separate written agreement for broader use.	What improvement activities the vendor intends.	Legal.
Clause 4: “safeguards described in Schedule A”	Security review precedes company data use.	The reviewer cannot assess the referenced safeguards from this packet.	Obtain and review Schedule A; incorporate the agreed version.	Schedule A and any security follow-up information.	Procurement obtains it; the security lead reviews it.
Clause 5: “fees paid in the three months”	General cap of at least 12 months of fees paid or payable.	The short lookback and payment timing create uncertainty about the cap.	Propose a defined 12-month fee basis and separately review confidentiality and security treatment.	Vendor response and the contemplated exposure.	Legal.
Clause 6: “then-current rates”	Usable export before access ends, with known format and cost.	The company lacks defined export arrangements and pricing.	Specify format, delivery timing, access arrangements, and charges.	Vendor export capabilities and Procurement’s operational requirements.	Procurement and Legal.

A corresponding recommendation could be:

Continue negotiations, subject to resolving the identified terms. Obtain Schedule A for security review before any company data enters the service. Procurement should negotiate the first-year charges within the authorized budget or obtain Finance approval. Legal should resolve data-use scope and liability wording, while Procurement confirms workable export arrangements.

Ask the candidate how they handled renewal pricing and the notice window. Do those terms need an immediate change or an operational follow-up? The six-row limit makes prioritization visible and gives you a reason to ask about an omitted issue.

Compare the proposed language with the recommendation. If the candidate recommends customer-specific data use, the edit should carry that restriction through to the contract.

A useful verification record names the conclusion, source, and reason for retaining or changing it. If an AI draft assigns a $43,500 liability cap, a candidate might record: “Replaced the calculation with an unresolved interpretation because annual advance payment affects the fees-paid lookback.” Correct output deserves the same explanation. A generic “checked citations” entry gives you little evidence of the candidate’s reasoning.

Score Five Dimensions With Observable Anchors

Use this proposed rubric to discuss the submitted work and shared interview answers. Train evaluators on the packet and example evidence before administering the exercise.

N/O means not observed: the exercise did not produce enough evidence to assess that dimension. Zero means observed failure: the candidate demonstrated the behavior described in that anchor. Record the supporting passage or action for every rating. Incomplete verification evidence alone does not establish a false conclusion.

Dimension	0	1	2	3
Framing	Addresses the wrong decision or omits the requested recommendation.	Requests a general review with little direction.	Gives AI the audience, decision, source materials, and required outputs.	Also sets useful scope limits, priorities, and questions for unresolved facts.
Context	Treats a vendor term as company approval or contradicts a supplied position.	Uses company context inconsistently.	Supplies the relevant budget, data, security, and exit instructions and checks that the output applies them.	Explains the consequence of a proposed exception and identifies who can authorize it.
Source Verification	Carries a materially false or unsupported source claim into the final deliverable.	Shows partial checking but cannot explain how one material conclusion was verified.	Checks material quotations and compares conclusions with the sources.	Examines material clause interactions and missing documents, records their effect, and corrects output where needed.
Legal Judgment	Recommends signing or uploading company data without resolving the applicable conditions in the brief.	Lists issues with limited prioritization or practical next steps.	Proposes coherent edits, a conditional recommendation, and appropriate escalation.	Explains trade-offs, aligns all deliverables, and identifies what new evidence would change the advice.
Confidentiality	Uses material outside the authorized source set or an unauthorized environment.	Recognizes the rules but needs a reminder to apply them.	The submitted record and observed work show use of the approved materials and environment.	Explains what they would need to confirm about permitted inputs, access, and data handling before using company documents.

Avoid a total-score cutoff. These anchors have no established relationship to hiring success. Review the evidence for each dimension alongside the role requirements and other selection information.

Use the Interview to Examine Unresolved Decisions

Ask the same core follow-ups of each candidate. OPM’s structured-interview guidance supports consistent questions and rating standards.

Useful questions for this packet include:

• Before using a company agreement in an AI tool, what would you need to confirm about permitted inputs, access, and data handling?
• Which recommendation did you retain or change after checking the source, and why?
• How does annual advance payment affect your reading of the liability cap?
• What would you ask the vendor about its use of data to improve the service?
• What must Schedule A establish before the security lead can finish reviewing it?
• If Finance approves the additional $14,000, which issues remain open?
• Which proposed edit would you negotiate first, and what would justify a fallback?

When you change a fact, ask the candidate to explain what they would update in the prompt and which conclusions they would recheck. Ask where AI saved effort and where verification took more work. Record how they explain their choices and recognize the limits of the evidence. Keep interface navigation separate from the quality of their legal reasoning.

Human evaluators assign the ratings and make the hiring decision. GC AI provides the environment for the proposed document review. The hiring team reviews the candidate’s work manually.

Calibrate the Exercise in GC AI

Run the synthetic packet through your intended GC AI setup before introducing it to candidates. Review the source checks, proposed language, Procurement message, and evaluation notes. Confirm that the assignment gives evaluators enough evidence to discuss the skills the role requires.

After each exercise, have the evaluator manually copy the reviewed issue table, source references, verification decisions, and ratings into a candidate-specific record in the hiring team’s approved system. Follow HR’s access and retention rules. Keep the synthetic reference packet and its version separate, with calibration notes for the next administration.

To evaluate whether GC AI fits this work sample, bring the fictional agreement and company brief to a demo. Ask the team to show how a candidate can check a cited finding against the source text.