The silence that follows is expensive. Benchmarks prove competence in the abstract; your risks live in the specifics. Edge cases, specialized terminology, and unique constraints that define your work rarely appear in anyone else’s test suite. The gap between benchmark scores and your reality isn’t a few percentage points, it’s damaged client trust, regulatory scrutiny, and sleepless nights for the executives who signed off on the deployment.

This gap between promise and performance isn’t a technical glitch. It’s a governance challenge. And it reveals something profound about how we’ve been thinking about AI leadership entirely wrong.

The Blindspot in Every AI Playbook

Pick up any executive guide to AI transformation, recent executive AI guides from IBM, McKinsey, and the Big Four consultancies, and you’ll find sophisticated frameworks for governance, detailed roadmaps for implementation, and compelling visions of AI-powered futures. These books and reports get 90% of the story right. They correctly identify that leaders must move from being passive consumers of AI to active creators of AI value. They emphasize governance, skills development, and strategic alignment.

But they systematically omit the single most important mechanism for achieving these goals: how leaders translate their deep domain expertise, their understanding of what quality means in their specific context, into measurable, enforceable standards for AI systems.

This isn’t a minor oversight. It’s the difference between governance theater and actual control. Between hoping your AI behaves and knowing it will perform.

The authors of these guides aren’t ignorant. These guides tend to focus on high-level strategy and often treat evaluation as a technical implementation detail. But this reveals a fundamental misunderstanding of what evaluation actually is. It’s not quality assurance. It’s not testing. It’s the very act of encoding what your organization values into a form that can be measured, managed, and improved.

When a law firm defines what constitutes a properly researched legal memo, when an insurance company articulates what empathetic claim handling looks like, when a bank specifies acceptable risk thresholds, these aren’t technical specifications. They’re strategic decisions that define competitive advantage. And in the AI era, these decisions must be translated into what I call “evaluation-as-policy.”

The Non-Delegable Duty of Defining “Good”

Here’s what the playbooks miss: in an AI-transformed enterprise, defining what constitutes acceptable performance isn’t something leaders can delegate to their technical teams. It’s not something they can outsource to vendors. It’s a fundamental leadership responsibility as non-negotiable as setting strategy or managing risk.

Think about how you currently ensure quality in human work. You don’t just hire smart people and hope for the best. You provide clear expectations. You review work products. You give specific feedback. You know what good looks like because you’ve spent years developing that expertise.

The same expertise that allows you to recognize a well-crafted legal argument, a compelling marketing campaign, or a thorough risk assessment is exactly what’s needed to create meaningful AI evaluations. The only difference is that instead of reviewing work after the fact, you’re encoding your standards upfront in a form that can be systematically applied.

This is where the concept of “golden data” becomes critical. Golden data isn’t just training data or test data. It’s the carefully curated collection of examples that embody your organization’s definition of excellence. Each example is a concrete instantiation of your standards, your values, your risk tolerance.

Creating golden data isn’t a technical task, it’s a leadership function. When your general counsel reviews AI-generated legal summaries and annotates what’s acceptable and what’s not, she’s not doing QA. She’s encoding the firm’s legal standards into a strategic asset. When your head of customer service identifies model responses that perfectly capture your brand voice, he’s not just providing feedback. He’s building competitive advantage.

From Abstract Principles to Executable Standards

The challenge, of course, is that most leaders don’t know how to bridge the gap between their expertise and the technical requirements of AI evaluation. They can articulate what they want—“accurate legal citations,” “empathetic customer responses,” “comprehensive risk assessments”—but they don’t know how to make these concepts measurable and enforceable.

This is the murky void that exists in most organizations today. Everyone agrees that evaluation is important. Few understand how to actually do it. Even fewer realize that the solution doesn’t require technical expertise, it requires clear thinking about what matters to your business.

Let me make this concrete. Evaluation, at its core, follows a simple three-column pattern: input (what goes into the AI), output (what the AI produces), and expected output (what you wanted it to produce). This isn’t complicated. It’s exactly how you’d evaluate human work, just structured more systematically.

The power comes from how you assess the relationship between your system's actual output and the expected output. Sometimes you need exact matches—a legal citation must be precisely correct. Sometimes you need fuzzy matching—a customer service response should cover the right points even if the wording varies. And sometimes you need nuanced judgment—does this financial advice demonstrate appropriate fiduciary duty?

This is where the concept of LLM-as-a-Judge becomes transformative. Instead of trying to codify every possible variant of acceptable output, you can articulate your standards in natural language—the same way you’d instruct a human employee—and use a language model to assess whether outputs meet those standards.

If you can write a memo explaining what makes a good quarterly report, you can create evaluation criteria for AI-generated reports. If you can train a junior attorney on proper legal research, you can define standards for AI legal research. The skill you need isn’t programming. It’s the ability to articulate what you already know.

The Strategic Asset Nobody’s Talking About

Here’s what should keep executives up at night: while you’re treating evaluation as a technical afterthought, your competitors might be building it as a strategic asset. Because your evaluation criteria and golden datasets aren’t just test files. They’re the usable codification of your organizational knowledge, competitive insights, and strategic priorities.

Consider what goes into a sophisticated evaluation suite for a law firm’s AI systems. It contains examples of how to spot obscure jurisdictional issues that only experienced partners would catch. It embodies the firm’s approach to risk assessment that differentiates it from competitors. It captures the nuanced judgment calls that define the firm’s reputation.

This isn’t a generic capability that any firm could replicate. It’s proprietary intellectual property as valuable as any other strategic asset. Some evaluations—basic accuracy, general fairness—can and should be shared across industries. But your core evaluations, the ones that capture what makes your organization unique, are trade secrets.

The organizations that recognize this are doing something radical: they’re treating evaluation development as a C-suite responsibility. They’re running cross-functional workshops where legal, risk, product, and customer service leaders collaborate to define golden datasets. They’re version-controlling these assets like critical code. They’re measuring and reporting on evaluation coverage like any other strategic metric.

Making It Real: From Theory to Practice

At this point, you might be thinking, “This sounds important but impossibly complex.” Let me show you how wrong that assumption is. You can start meaningfully evaluating your AI systems this week with just a spreadsheet and clear thinking.

To see this principle in action, you can try it yourself in under two minutes using our open-source platform, Lake Merritt. Follow the first exercise in the Quick Start guide, a “60-Second Sanity Check.” You’ll simply create a spreadsheet with three columns: the input (the question you ask the AI), the output (the AI's actual response), and the expected_output (your definition of a perfect answer). When you run the evaluation, you’ll see how an “LLM-as-a-Judge” programmatically assesses the quality of the actual output against your ideal expected_output. Fiddle with it, change the content in the expected_output column and see how it impacts the evaluation scores. This simple, hands-on exercise will give you the concrete intuition needed to apply this process to your own business context.

Begin with what I call a “10-row quick start.” Take ten representative cases from a real use case in your business. For each input, develop your own idea of what outputs you expect and why, and then have domain experts define their ideal outputs. Settle on an initial set of expected outputs. This is your initial golden dataset. Now run your AI system against these inputs and compare its outputs to your golden standard.

The results will be immediately illuminating. You’ll see patterns in where the AI struggles. You’ll identify edge cases you hadn’t considered. Most importantly, you’ll begin developing intuition for what kinds of standards are easy to meet and which require more sophistication.

As you develop confidence, you can scale this approach. The ten rows become a hundred, then a thousand. The simple comparisons evolve into sophisticated rubrics. The ad-hoc checks become systematic “evaluation packs”, version-controlled, repeatable test suites that can be run automatically before any AI system updates are deployed.

There’s an even more powerful approach that allows your leadership to encode their expertise more rapidly: learning from reality. This method allows your executives to shift from being authors to being editors, which is often a more efficient use of their time. Instead of trying to define perfect outputs upfront, have your key leaders and their most trusted senior experts (the same people who define your strategy) annotate actual AI outputs. They can mark what’s good, what’s problematic, and what’s unacceptable. These leadership-validated annotations then become core foundations for your evaluation system, ensuring it recognizes quality the same way you would.

To make this concrete: for a legal summary AI system, instead of asking your general counsel to write ten perfect legal summaries from scratch, you can present her with ten AI-generated summaries and have her annotate them, correcting a citation here, flagging a risk there. Those annotations, born from senior-level judgment, become the executable standards for your evaluation system. This creates a virtuous cycle where your top experts continually refine the AI's alignment with your organization's most critical standards.

This creates a virtuous cycle. Your AI systems generate outputs. Your experts review and annotate them. These annotations become evaluation criteria. The evaluations drive improvements. The improved systems generate better outputs. And the cycle continues, with each iteration encoding more of your organization’s expertise into measurable, manageable form.

The Agent Revolution Changes Everything

So far, I’ve focused on evaluating AI outputs, the text, analysis, or recommendations that AI systems produce. But the next generation of AI isn’t just generating content. It’s taking action. AI agents are making decisions, using tools, following processes, and interacting with other systems in complex workflows.

This fundamentally changes what evaluation means. It’s no longer sufficient to check if the final answer is correct. You need to evaluate the entire process. Did the agent use the right tools? Did it follow required procedures? Did it respect security boundaries? Did it escalate appropriately when uncertain?

Consider a legal research agent. The quality of its final memo matters, but so does its process. Did it search the right databases? Did it prioritize binding precedent appropriately? Did it verify that cited cases haven’t been overturned? These behavioral evaluations require a different approach, one that captures and analyzes the full trajectory of the agent’s actions.

This is where technical concepts like OpenTelemetry traces become essential. But don’t let the jargon intimidate you. A trace is simply a record of everything the agent did, every tool it called, every decision it made, every piece of data it accessed. Evaluating these traces means you can ensure not just that the agent reached the right conclusion, but that it got there the right way.

The implications are profound. In traditional software, you could separate business logic from implementation details. In agentic AI, the process IS the product. The way an agent conducts legal research, handles customer complaints, or analyzes risk isn’t just a means to an end—it’s a direct expression of your organizational values and standards.

Proof That This Works

These aren’t theoretical frameworks or academic exercises. Organizations are using these approaches today to solve real problems and prevent real failures.

Consider a challenge at the heart of AI governance: ensuring systems behave fairly and align with your company’s values. This isn't just a legal or regulatory checkbox; it's fundamental to brand safety, customer trust, and strategic alignment. A powerful example is the BBQ (Bias Benchmark for QA), a rigorous academic framework for detecting demographic bias. Using a tool like Lake Merritt, this top-tier public benchmark can be implemented as a reusable "evaluation pack" to systematically test your systems. To underscore its industry significance, BBQ was the sole fairness and bias benchmark OpenAI chose to use in its safety testing for GPT-5. This shows how you can move beyond theory to not just flag problems, but quantify them, track them over time, and ensure that fixes actually work.

This same approach of codifying standards applies to any area where deep, nuanced domain expertise is your competitive advantage. Rather than rely on generic public benchmarks like BBQ, however, the task is to develop your own measures that support and reflect your organization's priorities and imperatives. For instance, a financial services firm can move beyond generic compliance to evaluate its unique interpretation of "fiduciary duty." Such an evaluation might progress from basic, deterministic checks—like verifying required disclosures are present—to sophisticated, judgment-based assessments of whether advice truly serves a client’s best interests in a nuanced scenario.

Crucially, these evaluations work because they are built by the domain experts who own the outcome, not by technicians. In the financial services scenario, this means the legal team defines disclosure, compliance specifies risk scenarios, and customer advocates articulate what "client’s best interests" means in practice. But the principle is universal: for a marketing AI, the brand team would define what is "on-brand"; for a medical AI, clinicians would define a "safe diagnostic summary." The technical team's role is to simply implement these expert-defined standards into a systematic, repeatable process.

The Ecosystem of Evaluation

To demonstrate that these concepts aren’t just theory, I’ve built Lake Merritt, an open-source evaluation workbench that embodies these principles. I use Lake Merritt every day to evaluate my own AI apps and services, and have also utilized it effectively as part of Civics.com's professional consulting services, ensuring that my clients' AI products operate as expected. But let me be clear: Lake Merritt isn’t the point. The methodology is the point. Lake Merritt simply proves that the methodology works.

The platform does several things that matter. It provides a web interface simple enough that a lawyer or product manager can use it without training. It supports what I call the “Hold My Beer” workflow—where you can go from a vague idea about quality to a working evaluation in minutes. It treats evaluations as code, making them versionable, shareable, and systematic. It can evaluate not just outputs but entire agent workflows through OpenTelemetry trace analysis.

While I launched Lake Merritt this week because I think it’s valuable to have an easy to use evals tool that non-technical people can get started with, this software is just one option in a rich ecosystem of evaluation tools. Arize Phoenix provides powerful observability and monitoring capabilities. Galileo offers sophisticated analytics and agent debugging tools. Open-source projects like DeepEvals and OpenAI Evals provide flexible frameworks for custom evaluations. LangWatch excels at specific use cases. Each serves different needs at different scales.

In the legal domain specifically, pioneers are emerging. Vals has published groundbreaking reports on legal AI evaluation. ScoreCard is working to standardize agent evaluations for legal use cases. Individuals like Ryan McDonough who is a true global thought leader on AI and evals in law at KPMG, and newer voices like Anna Guo and her collaborators in Singapore, are openly sharing their learnings and pushing the field forward. There are many, many others making starting to make strides.

This diversity is healthy and necessary. No single tool or approach will serve every need. What matters is that organizations develop the capability—through whatever tools make sense for them—to systematically evaluate their AI systems against their specific standards.

We’re in the advanced planning stage now of bringing this community together at an evaluation summit jointly hosted by Stanford and MIT. The goal isn’t to crown winning tools or approaches. It’s to share learnings, establish best practices, and accelerate the entire field’s development. To stay informed about that event or if you have constructive and relevant work in the custom evaluations arena, please reach out here.

Your Path Forward

If you’ve read this far, you’re probably convinced that custom evaluation matters. The question is what to do about it. Let me give you a practical path forward that you can start this week.

First, identify your highest-risk AI use case. This is where evaluation matters most and where you’ll get immediate value from better oversight. Don’t try to boil the ocean. Pick one critical application and focus there.

Second, convene your domain experts. Bring together the people who truly understand what quality means for this use case. This isn’t a technical meeting, it’s a business meeting. The question on the table is simple: “What does good look like?”

Third, create your first golden dataset. Start small, even ten examples are enough to begin. For each example, capture the input and the ideal output. Have your experts explain why each output is ideal. These explanations become the seeds of your evaluation criteria.

Fourth, test your current AI system against this golden dataset. Don’t expect perfection. Expect illumination. You’ll immediately see patterns in where your system struggles and where it excels.

Fifth, iterate and expand. Add more examples. Refine your criteria. Develop more sophisticated evaluations. Move from manual checks to automated gates. Build evaluation into your deployment pipeline so that no AI update goes live without passing your standards.

This isn’t a technical project. It’s a governance initiative. It’s how you exercise real control over AI systems that are increasingly critical to your operations. It’s how you ensure that AI serves your strategic objectives rather than undermining them.

The Executive Imperative

We’re at an inflection point in how organizations create value with AI. The experimental phase is ending. The operational phase is beginning. And in this operational phase, the organizations that thrive won’t be those with the most sophisticated models or the largest datasets. They’ll be those that can most effectively translate their human expertise into AI capabilities.

This translation happens through evaluation. Not generic benchmarks or vendor-supplied metrics, but custom evaluations that embody your specific standards, values, and priorities. These evaluations aren’t a tax on innovation, they’re an accelerator for it. They allow you to move fast because you can move with confidence. They allow you to delegate to AI because you can verify performance. They allow you to differentiate because you can systematically improve what matters most to your business.

The choice facing every executive is stark. You can continue treating AI evaluation as a technical detail, hoping that your vendors and technical teams somehow divine what quality means for your organization. Or you can recognize that in the AI era, evaluation is the executive function, the mechanism through which leadership expertise shapes organizational outcomes.

Your AI strategy without custom evaluation isn’t a strategy. It’s expensive hope. And in a world where AI increasingly mediates critical business functions, hope is not a plan.

The boards that are asking “How do we know it works for us?” aren’t being paranoid. They’re being prescient. They understand that AI governance without custom evaluation is like financial governance without custom accounting standards, theoretically possible but practically meaningless.

The good news is that building evaluation capability doesn’t require massive investment or technical transformation. It requires clarity about what matters to your business and the discipline to measure it systematically. If you can articulate expectations to humans, you can create evaluations for AI. If you can recognize quality when you see it, you can encode that recognition into systematic assessment. Literally, that recognition just needs to be articulated in language in order to be usable as criteria in programmatic evals.

In the AI era, this isn’t optional. It’s existential. The organizations that master evaluation will shape AI to serve their purposes. Those that don’t will find themselves shaped by AI systems they don’t sufficiently control.

The question isn’t whether you’ll develop custom evaluation capabilities. It’s whether you’ll develop them before or after they become urgently necessary. Before or after your first AI crisis. Before or after your competitors use superior evaluation to deliver superior AI-powered services.

The time to start is now. Not because the technology demands it, but because leadership demands it. Because in a world where AI increasingly mediates how organizations create value, the ability to define and measure what “good” looks like isn’t just a technical capability.

It’s the executive function itself.

In my testimony, I presented California's SB 813 as a potential "third way" for AI regulation—a middle path between heavy-handed restrictions and complete absence of oversight. This approach creates voluntary certification through Multi-stakeholder Regulatory Organizations (MROs) that can verify AI systems meet safety and reliability standards. Certified systems gain a rebuttable presumption of "reasonable care" in tort cases—creating a powerful incentive for responsible innovation without mandating specific technical approaches.

The economic implications of AI agent systems formed a central focus of our discussion. These autonomous AI systems are already transforming software engineering, legal services, and commercial transactions. Companies like Perplexity and Amazon are deploying AI agents that can conduct transactions and make purchases on users' behalf, while Stripe now offers tools for businesses to authorize AI agents to make direct payments.

The economic boost could reach 3-5% of GDP by 2030, yet the same technology that scales productivity can displace jobs or amplify malicious actors. During questioning I discussed authenticated delegation protocols that tie every agent action to a verifiable human or legal entity, limiting liability drift and curbing fraud, and urged pairing flexible certification with robust up-skilling programs rather than blunt “human-in-the-loop” mandates that freeze scalability.

What's particularly striking is how quickly these technologies are moving from research concepts to everyday deployment. When I first testified to this committee on generative AI, many of these capabilities seemed theoretical. Today, they're commercially available. This rapid evolution suggests we need frameworks that can adapt as quickly as the technology while providing necessary guardrails around high-risk applications.

The committee demonstrated a sophisticated understanding of the challenges, asking thoughtful questions about security implications of foreign AI models, intellectual property concerns with training data, and evolving approaches to human oversight requirements. As Senator Rothfuss noted, Wyoming has a tradition of "regulating to enable rather than restrict"—a philosophy perfectly suited to this moment of technological transformation.

I've been honored to work with the Wyoming legislature over several years as they've crafted blockchain legislation and other digital innovation frameworks. Their approach of careful listening, thoughtful questioning, and balanced policy-making continues to serve as a model for how states can navigate technological disruption. I look forward to continuing this important conversation at future hearings as we work toward frameworks that unlock AI's benefits while mitigating potential harms.

May 16, 2025 Update: Further Thoughts on AI Regulation, MROs & a Path to Interstate Co-operation

After I posted my Wyoming testimony on multistakeholder regulatory organizations (MROs), Nancy (Leyes) Myrland left an insightful LinkedIn comment that zeroed-in on three issues:

1. Will state-level guardrails still matter if Washington eventually centralises AI oversight?

2. How often would a “trustworthy” badge have to be renewed when models evolve daily?

3. Are California-only guardrails enough, or must other states join for real protection?

I made a short reply to Nancy on LinkedIn but the character limit is short and her questions invite a richer look at both California’s SB 813 and an idea I sketched for the legislature: inter-state reciprocity. So let’s go deeper!

Nancy’s questions—answered

1 | Will a future federal regulator make state action moot?

Not at all. SB 813 obliges every MRO to spell out “an approach to interfacing effectively with federal and non-California state authorities” . In American law we repeatedly see innovations flow bottom-up: Blue-Sky securities rules, driver-licence compacts, the Uniform Commercial Code. States are nimble laboratories; Congress often scales what they prove. A running California MRO framework gives Washington a tested chassis to bolt onto.

2 | How often does “trustworthy” recertification happen?

• Model-level triggers. Each MRO plan must define technical thresholds for updates requiring renewed certification . If a developer adds autonomous code-execution or a new multimodal dataset that crosses the line, the certificate pauses until a fresh audit clears it—much like the FDA’s 510(k)/PMA split for medical devices.

• MRO-charter clock. An MRO’s own designation lasts three years and can be ripped up sooner if independence erodes, its methods become obsolete, or a certified model causes major harm . Oversight of the overseers updates at least as fast as the tech.

3 | Are one-state guardrails enough?

SB 813 already covers any AI deployed in California, so most national providers will seek certification. Still, a genuine safety net needs more than one state’s knots. Enter reciprocity.

Expanding the vision: a practical path to interstate AI reciprocity

While SB 813 gives California a robust foundation, legislators in Wyoming (and elsewhere) asked how to spread the benefit without fifty separate audits. The answer I proposed is an interstate reciprocity layer. It is not yet in SB 813; rather, it is a natural extension that lets developers certify once and be recognised in many jurisdictions, while each state keeps the power to yank recognition the minute another state’s protections slip.

4.1 A simple legislative starting-point

To switch reciprocity on, California (or any pioneering state) could add a single sentence to its safe-harbor section. Something like:

“A certificate issued under a substantially equivalent multistakeholder regulatory framework of another state shall confer the same rebuttable presumption, unless the Attorney General determines that framework no longer affords equivalent protections.”

That one clause empowers the AG to recognise outside frameworks and keep a live list of reciprocal states.

4.2 What “substantially equivalent” could mean

The phrase must have teeth. An outside framework would need to meet, at minimum, these pillars:

1. Comprehensive risk scope —covers CBRN, malign persuasion, autonomy, exfiltration.

2. Guaranteed independence —board composition and funding caps that block capture.

3. Transparency & accountability —public annual reports and decade-long record retention.

4. Robust enforcement —real-time power to revoke certificates when models drift.

5. Continuing governmental oversight —periodic review of each MRO by its home-state AG (or equivalent).

6. Collaborative data-sharing —MOUs so AG offices trade incident reports, best-practice memos and evolving threat intel in near-real time.

4.3 Making reciprocity work: procedural mechanics

• Public registry & dynamic review. California’s AG would publish the recognised-states list; every listing sunsets (say) in three years, forcing re-inspection so standards evolve with the science.

• Agile de-recognition. If State X’s MRO weakens or certifies a reckless model, California can strike that state overnight—integrity preserved, no legislative lag.

• Interstate compact option. For deeper ties, two or more states could enshrine reciprocity in a compact, driver-licence-style. The Uniform Law Commission could draft model language so Wyoming and New Jersey start from the same page.

4.4 Why stake-holders win

• Developers: one dossier, many states—lower friction, stronger incentive to certify.

• States: pooled expertise and shared intel, yet full power to slam the door if another jurisdiction backslides.

• Public: consistent guardrails and quicker access to vetted AI.

• Nation: a bottom-up baseline forms while Congress deliberates—innovation and safety advance together.

4.5 Guardrails & challenges

Reciprocity must never spark a race to the bottom. That is why listings sunset and why de-recognition is swift. And remember: the safe-harbor is narrow and rebuttable—it shields developers only on personal-injury and property-damage claims, not consumer-protection, privacy, or civil-rights suits . Participation is voluntary; immunity is limited.

Additional clarifications

• Transparency. MRO plans are filed with the AG; future regulations should publish them (redacting trade secrets) to build public trust.

• Built-in safeguards. Whistle-blower protections (§ 8898.2(a)(7)), mandatory incident reports (§ 8898.2(a)(3)) and auditing of post-deployment practices (§ 8898.2(a)(1)) are core plan elements .

Closing – laboratories at work

I’ve spent my career in state-powered innovation: drafting the Uniform Electronic Transactions Act, co-ordinating early digital-signature standards, steering multi-state mega-procurementsthat pooled demand for better pricing, building open-source repositories shared across agencies, and countless other projects where states proved nimbler and bolder than Washington. More recently we’ve seen states pioneer everything from digital identity and electronic notarization to friction-less sales-tax collection. SB 813 stands firmly in that tradition—nimble, incentive-driven, and ready for replication.

Could your state benefit from a “certify once, recognised many” approach? I’m eager to refine these ideas with lawmakers, technologists and advocates. Drop me a comment at Civics.Com/contact and let’s keep building trustworthy AI, the federalist way.

I wrote this post for the Consumer Reports Innovation Lab, where it was published on October 18, 2024 

The marketplace for AI-powered personal agents is rapidly evolving. Companies like Amazon and Salesforce are already offering services that help consumers navigate online shopping, manage subscriptions, and automate routine tasks. These developments signal a shift in how we interact with digital services and make purchasing decisions.

Consumer Reports is exploring the potential for developing pro-consumer AI agents that prioritize user interests above all else. This approach comes with unique legal and design challenges that set it apart from purely commercial offerings.

The idea of using such agents has the potential to fundamentally reshape how consumers use their data, navigate complex services, and make decisions. If developed thoughtfully, these agents could safeguard privacy and act as trusted intermediaries. While there are many interesting questions of law and practice to enable and safeguard this, thankfully, existing legal frameworks have already begun to anticipate and support such innovations.

In this post, I’ll examine three key areas:

1. The existing legal framework that supports the use of AI agents for transactions;

2. Design paths for creating truly user-centric AI agents; and

3. The potential impact of these agents on consumer empowerment in the digital marketplace.

By understanding these foundations, we can work towards AI agents that genuinely serve consumers’ best interests.

The Forgotten Framework: UETA and the Rise of LLM Agents

For decades, the law has envisioned a world where electronic agents can represent us and act on our behalf. In 1999, the Uniform Electronic Transactions Act (UETA) laid out a framework for e-commerce. UETA was created to address the legal uncertainties surrounding electronic transactions and to provide a consistent framework across states.

This law is the very reason we can confidently use electronic signatures and contracts in our daily digital interactions. It is a cornerstone of the information age, providing legal certainty for online commerce and other electronic transactions. More to the point, UETA provides explicit provisions for electronic agents to conduct transactions autonomously.

This uniform law has been adopted across the United States, statutorily enacted in 52 states and territories, and truly is the law of the land. This legal foundation provides clear definitions and applicable rules for concepts like electronic signatures, automated transactions, and attribution for the acts of autonomous agents. Fast forward to today, and this vision can finally come to life impactfully by supporting the use of new software services, including advanced AI assistants and LLM-based agentic software applications, including for individuals.

This existing legal foundation provides clear definitions and rules for key concepts:

• Electronic Agent: “A computer program or an electronic or other automated means used independently to initiate an action or respond to electronic records or performances in whole or in part, without review or action by an individual.” This definition perfectly describes the capabilities of LLM-powered AI agents.

• Automated Transaction: “A transaction conducted or performed, in whole or in part, by electronic means or electronic records, in which the acts or records of one or both parties are not reviewed by an individual in the ordinary course in forming a contract, performing under an existing contract, or fulfilling an obligation required by the transaction.” This clarifies the legal validity of agent-led transactions.

• Attribution: UETA also establishes how to determine on whose behalf an electronic agent is operating, ensuring accountability. Essentially, under UETA, an electronic record or signature is attributable to a person if it was the act of that person, which can be shown in any manner, including the efficacy of any security procedures applied.

LLM agents may make unexpected errors when conducting automated transactions, which is a significant concern. UETA establishes a framework for error prevention and correction, particularly emphasizing agreed-upon security procedures. For instance, a consumer and an online retailer could establish a spending limit for the consumer’s AI agent. If the agent attempts to exceed this limit, the security procedure would trigger an alert, preventing the error.

Importantly, if a merchant fails to implement an agreed-upon security procedure and an error occurs, UETA provides the consumer with the right to reverse the transaction. Conversely, if the retailer fails to implement an agreed-upon security procedure, such as verifying the purchase amount with the consumer before finalizing the transaction, and the agent makes a purchase beyond the agreed-upon limit, UETA could provide the consumer with legal grounds to reverse the transaction and recoup the excess funds.

With the emergence of AI agents, we now have the technology capable of meaningfully fulfilling UETA’s vision. These agents can communicate in natural language, negotiate, retrieve information, and even execute decisions—but critically, they can also be built to operate on behalf of the consumer, avoiding conflicting interests. Rather than invent new legal frameworks, we can leverage and extend existing ones like UETA to achieve predictable legal outcomes and accelerate the responsible development of personal AI.

For example, imagine a personal AI agent negotiating a better price for a subscription service on your behalf. Under UETA, this automated transaction would be legally binding, just as if you had negotiated it yourself.

Beyond price negotiation, such agents could automatically handle your insurance claims, gather quotes for home repairs, or even help you manage your investments according to your risk tolerance. Imagine receiving proactive alerts from your AI agent about better deals on services you frequently use or having it automatically adjust your utility plans based on your actual consumption patterns to save you money. These examples illustrate the potential of personal AI agents to simplify our lives and give us more control over our interactions with complex systems.

Such capabilities make LLM agents uniquely suited to leverage the legal framework established by UETA and extend it to new domains of personal empowerment.

Being Loyal: Building Agents That Work for You

While UETA provides the legal foundation for AI agents, the next step is to ensure these agents can operate securely, reliably, and in alignment with the user’s interests. The most compelling use case for personal AI agents is their ability to advocate on behalf of consumers without bias or conflicting interests. Unlike AI systems embedded within purely profit-seeking enterprises, or to advance a commercial objective or to fulfill a narrow “customer service” framework, personal AI agents could be entrusted with a “duty of loyalty” that binds the service provider to operate the agent in the best interests of the user. These agents could manage tasks like travel bookings or e-commerce purchases with the same trustworthiness as a high-end fiduciary representative, advocating only your interests.

Robust encryption, privacy standards, and transparent data stewardship practices could bolster this trustworthiness. These types of measures begin with terms of service and governance-based assurance, and that are also be encoded into the design of the system. To achieve this level of trust, AI agents must also implement clear attribution mechanisms. This means that any action the agent takes can be reliably traced back to the user, establishing accountability and legal responsibility.

Looking ahead, it’s crucial to consider how AI agents will interact not just with traditional online systems, but where AI agents negotiate and transact with each other on our behalf. This could lead to a more efficient and potentially fairer marketplace. For example, your personal AI agent could automatically negotiate the best price for a product or service by interacting with the AI agents of multiple vendors, comparing offers, and securing the most favorable terms, all while adhering to your pre-defined preferences and limits. Furthermore, exploring concepts like delegation of authority in multi-agent systems can pave the way for even more powerful consumer empowerment tools. While LLM agents can interact with natural language and web-based systems surprisingly well, eventual high-velocity agent-to-agent transactions would require the development of common protocols and standards for inter-agent communication and negotiation. However such standards are not needed to use LLM agents with existing online services and platforms. UETA envisions and supports transactions with one electronic agent, two electronic agents, or large numbers of electronic agents. There is room to grow under the existing law.

With this robust legal foundation in place, the next challenge is to design AI agents that not only comply with these laws but also operate effectively on behalf of consumers.

Design Paths for Consumer AI Agents

Designing consumer AI agents requires a thoughtful approach to balancing security, user experience, and legal or regulatory considerations. Consider the following three potential models, each with its own advantages and challenges:

• Full Authentication Model whereby the agent uses the same authentication and authorization credentials as its user;

• Intermediary Model, whereby the agent is operated by another party who uses the agent to act on the user’s behalf; and the

• Decentralized Identity Model, whereby the agent leverages decentralized identifiers and verifiable credentials to interact with third parties, giving users direct control over their digital identity.

Each model presents unique trade-offs and aligns differently with user trust, system complexity, and risk frameworks. Let’s examine each of these design paths in more detail, considering their strengths, weaknesses, and potential applications in the context of consumer AI agents.

Full Authentication

The Full Authentication path positions the AI agent as a direct extension of the user. By acting with the user’s authorization and utilizing their credentials and permissions, the agent can access online platforms, add items to shopping carts, compare prices, and even complete purchases autonomously according to pre-defined rules. The main strength of this approach is its simplicity. It uses current technology to enable the AI agent to perform various tasks without requiring companies to develop new infrastructure or protocols. This seamless interaction is achieved through existing standards, making it easy to deploy and integrate.

To effectively execute this, the agent would need capabilities to interact using the same interfaces that would be made available to an authenticated user.

However, this approach also carries significant security risks, as it requires granting the agent extensive access to user accounts and credentials. There are also substantial risks in terms of data stewardship and liability (and it’s for this reason regulators have discouraged the use of screen scraping and are encouraging the development of more secure interfaces for third-parties to authenticate on a users’ behalf).

For instance, if the agent misuses the data or performs unintended actions, it may be unclear who should be held responsible—the user, the agent provider, or the third party with whom the agent transacted. Moreover, compliance with data protection and privacy regulations like GDPR or CCPA is more challenging to implement because the agent’s full access could potentially implicate user data rights. This ambiguity can hinder adoption, as users and companies may be reluctant to grant the required level of permissions.

Intermediary Path

In contrast, the Intermediary path positions the AI agent as part of a distinct entity that acts as a negotiator or advocate on behalf of the user. Instead of using the user’s credentials, the agent operates under its own identity and permissions, creating a clear separation between the user and the agent service provider. Here, the agent is provided to the user by another party, such as a consumer group or other service provider, and is designed to operate on behalf of the user with third parties like online vendors.

In this setup, the agent operates under a set of rules that define its role, allowing it to handle transactions, share specific data points, and communicate the user’s preferences in a controlled manner. This granular control may empower users with greater agency over their data and privacy. To enable this, the Intermediary path would require new protocols or handshake mechanisms to establish the agent’s legitimacy and scope of authority with third parties like online merchants and other organizations the user seeks to transact with.

To function effectively as an intermediary, the agent could leverage existing standards like OAuth 2 and OpenID Connect, but in a different way. In effect, the intermediary acts as an authorized application of the consumer, with permissions explicitly granted by the user to take specific actions on their behalf. This means the intermediary holds tokens or authorizations that permit it to execute tasks as the user’s representative without the agent ever directly holding the user’s core credentials. This model maintains a clear distinction, allowing the intermediary to act independently while still adhering to permissions that have been transparently defined and authorized by the user.

This approach offers a range of advantages, primarily stemming from the clear delineation of roles and responsibilities, which helps simplify accountability and can foster greater trust. In a technical legal sense, the party providing the agent service would be the legal “agent” of the user in this case, greatly clarifying the roles and relationships with the user (who would be the “principal,” legally) and third parties with whom transactions are conducted. In simpler terms, this means the organization providing the AI agent service could be legally responsible for the agent’s actions, because that organization is the legal agent and may owe the user a duty of care to act reasonably and competently.

This separation clarifies liability and simplifies compliance with data protection and other laws. Furthermore, the agent can engage in advanced activities such as dynamic pricing negotiations or crafting customized agreements with service providers, offering enhanced value to the user. However, a significant challenge in adopting the Intermediary path lies in the need for standardization. Creating the necessary infrastructure and achieving industry-wide consensus on configuring existing protocols in new ways (eg, to support an authorized agent role with standards like OAuth 2 and OpenID Connect) and filling the remaining gap with new protocols involves substantial coordination and time, making this a more complex and long-term solution.

Decentralized Identity: A Glimpse into the Future?

Looking further ahead, decentralized identity systems offer an intriguing possibility. Decentralized identity approaches enable users to control and selectively share their data with service providers and other third parties through verifiable credentials, theoretically eliminating the need for centralized authentication. This approach aligns well with the goals of personal AI agents, empowering users with granular control and principal authority over their digital identities and interactions. While still in its early stages, decentralized identity technology holds some potential for shaping the future of consumer AI agents. However, the novel technologies and consequent switching costs for all the parties involved—especially online merchants and other organizations the consumer wishes to interact with—would be considerable. Therefore, while promising, this remains a more speculative and longer-term potential path that calls for continued innovation and collaboration.

Ultimately, these three design paths offer varying levels and pathways of control, security, and functionality. The decision on which path to adopt will depend heavily on the use case, user expectations, and industry acceptance. While the Full Authentication path is practical for quick adoption and basic tasks, the Intermediary path offers a higher level of security and compliance at the cost of complexity, while Decentralized Identity remains, for the moment, even more complex and speculative. Continued research and development are crucial to address the inherent challenges of each path and unlock the full potential of consumer AI agents.

The Future of Personal AI Agents: Reimagining Consumer Empowerment

The implications of LLM agents for consumer empowerment are profound. If built with the right legal and technical safeguards, they could shift the balance of power, allowing individuals to navigate complex systems—whether financial, commercial, legal, or social—with an AI working solely in their interests. These agents could help consumers make informed choices, protect their privacy, and advocate for their needs in previously impossible ways.

The existing legal framework, starting with UETA, provides a solid foundation on which to build. By leveraging this legal basis and focusing on designing AI agents that align with consumer interests, we can create technologies that empower consumers, giving them tools to engage in the digital world with confidence and consumer-directed autonomy. Understanding this legal foundation allows us to explore how AI agents can be designed to prioritize the consumer’s interests.

Personal AI agents represent a significant shift in how consumers can interact with digital services. By leveraging existing legal frameworks like UETA and focusing on consumer-centric design, we can create AI systems that truly empower individuals. As we move forward, collaboration among technologists, legal experts, policymakers, and consumer advocates will be crucial to ensure these agents are developed securely, reliably, and responsibly. The potential for personal AI agents to level the playing field for consumers in the digital landscape is immense, making this an exciting and important area for continued innovation and development.

In the next blog post, I will delve deeper into how fiduciary duties, especially the duty of loyalty, could serve as a powerful model for AI agents acting and transacting in the interest of consumers, distinct from the interests of merchants and other counterparties to transactions.

It's not just about boosting productivity, though it certainly does that. For me, NotebookLM unlocks new levels of creativity and insight that were simply impossible before. As someone who constantly grapples with massive amounts of complex information—legal documents, research papers, data sets—I'm always searching for tools that can help me synthesize, analyze, and ultimately understand that information on a deeper level. NotebookLM is a game-changer for serious thinkers and doers. It's like having a super-powered research assistant working alongside you, helping you to dig through data, analyze arguments, and ultimately, think faster and better. Here’s a complete (as of the date of this post) collection of NotebookLM documentation you can scan to get a quick look at what it does and how to use it.

But what truly blew my mind is NotebookLM's AI-generated podcast feature. Initially, I dismissed it as a cool party trick, but after experiencing the quality firsthand, I can confidently say it's astounding. The two-host audio conversations are not just "good for AI," they're genuinely good – surpassing the vast majority of human-produced podcasts. They've completely transcended the uncanny valley – that eerie feeling you get when encountering AI that is almost human but not quite, leaving you with a sense of unease – delivering a listening experience that's both engaging and enjoyable. Most importantly, the underlying intelligence does a great job of surfacing and synthesizing the important points, perpectives, and even questions posed by the source materials you feed it. So the podcast ends up being astonishingly on-point.

This is particularly remarkable because the AI doesn't just mimic human speech, it goes through a process of drafting, revising, and refining its content, just like a human writer. It even throws in those little pauses and "ums" that make a conversation sound natural. The result is a clear, concise audio summary that feels like you're listening to a conversation between two colleagues who have a deep understanding of the topic at hand.

The applications for this technology are endless. Imagine students getting custom audio explainers tailored to their learning styles, professionals getting up to speed on a new topic during their commute, or even families having deeper, more meaningful conversations guided by evidence and diverse viewpoints. This is the kind of future that NotebookLM is making possible.

I've been experimenting with the podcast feature in some creative ways, by adding custom instructions to steer the content and make specific points. It's incredible to see how responsive the AI is to these prompts. It's like having a personalized audio production team at your fingertips. I just made a podcast about NotebookLM (embedded at the top of this post) as an example.

It's not about replacing human potential, it's about amplifying it. As I often say to the professionals I train, "If you're not using AI to enhance your productivity and creativity, you're falling behind." NotebookLM is a must-have tool for anyone who wants to stay ahead of the curve.

My own presentation begins at 1:08:41 but I commend the entire hearing panel for your review and consideration.

Panelists:

• Dazza Greenwood, https://www.linkedin.com/in/dazzagreenwood/

• Alex Reibman, https://www.linkedin.com/in/alex-reibman-67951589/

• Campbell Hutcheson, https://www.linkedin.com/in/campbell-hutcheson-80409a83/

• Anh Mac, https://www.linkedin.com/in/anh-mac/

• Nam Nguyen, https://www.linkedin.com/in/hoangnamm21/

Co-Chairs:

• Chris Rothfuss, Senate Co-Chair, https://en.wikipedia.org/wiki/Chris_Rothfuss

• Cyrus Western, House Co-Chair, https://en.wikipedia.org/wiki/Cyrus_Western

Up until now, building these agentic systems has been a painstaking, manual process. Researchers and engineers have had to meticulously hand-craft each component, experiment with different combinations, and rigorously configure for specific tasks. It’s a time-consuming bottleneck in the development of truly powerful LLM-based agents.

But what if we could automate this design process? What if we could let AI design the AI? That’s the audacious goal of a new research area called Automated Design of Agentic Systems (ADAS). 

How ADAS Works: AI Coding AI

The key insight behind ADAS is to use code as the design language for agentic systems. This leverages a few powerful ideas:

1. Turing Completeness: Programming languages are “Turing Complete,” meaning they can theoretically represent any computational process – including the intricate designs of agentic systems.

2. LLM Coding Proficiency: Modern LLMs are becoming increasingly adept at writing and understanding code, making them ideal candidates for automating agent design.

Imagine a “meta agent” – an automated LLM-based process specifically designed to identify and create new agents. It iteratively creates agents in code, tests them on specific tasks, learns from the results, and stores successful designs in an “archive” for future inspiration. This process, called Meta Agent Search, mimics the way human researchers iterate and build upon previous discoveries.  Check out their GitHub repo and see how it works for yourself.

The Surprising Results: Learned Agents Outshine Hand-Designed Ones

The early results of ADAS are remarkable. In experiments across various domains, including logic puzzles, reading comprehension, math, and even multi-task problem solving, learned agents consistently outperform state-of-the-art hand-designed agents. 

Even more surprisingly, these learned agents show a remarkable ability to generalize. One striking example is how an agent initially designed for solving complex math problems was able to transfer to reading comprehension tasks, maintaining competitive performance. This cross-domain generalization highlights the robustness of the agent designs uncovered by ADAS. An agent designed to solve math problems can be transferred to reading comprehension tasks and still achieve competitive performance. This suggests that ADAS is uncovering fundamental design patterns that transcend individual domains.

Implications and The Future

The research into ADAS is just beginning, but it holds the promise of turbo-charging how we create and deploy LLM-based agents. It’s a powerful example of how AI can not only solve problems but also design the solutions to those problems – a glimpse into a future where AI systems become increasingly self-sufficient and capable of shaping their own evolution.

A Shift in Perspective

The ABA’s new guidance marks an important shift in how the legal profession views generative AI. While not explicitly mandating its use, the opinion certainly suggests that understanding and potentially utilizing generative AI tools is becoming increasingly important for competent legal practice. This perspective aligns with the evolving nature of legal technology competence, drawing parallels to how use of email, computerized legal research, and eDiscovery have become standard skills in the lawyer’s arsenal of tool use.

Recognizing the Benefits

Formal Opinion 512 acknowledges the potential of generative AI to enhance both the efficiency and quality of legal services. By highlighting these benefits, the ABA is effectively encouraging lawyers to explore and consider how these tools might improve their practice and better serve their clients. This recognition is a clear indication that the legal profession is moving towards embracing innovative technologies rather than viewing them primarily with skepticism.

Balancing Innovation and Ethics

While the opinion is forward-thinking in its approach to generative AI, it appropriately emphasizes the importance of responsible use. The guidance carefully outlines how existing ethical rules apply to this new technology, ensuring that the core values of the legal profession are maintained even as new tools are adopted. This balanced approach demonstrates the ABA’s commitment to fostering innovation while upholding the highest standards of professional conduct.

Ethical Considerations

The ABA Formal Opinion 512 outlines several crucial ethical considerations for lawyers using generative artificial intelligence (GAI) tools in legal practice. These considerations include maintaining competence, ensuring confidentiality, proper communication with clients, and upholding supervisory responsibilities. Below are the key points and recommendations for alignment with the opinion:

Competence:

• Lawyers must have a reasonable understanding of the capabilities and limitations of the GAI tools they use. This includes understanding the potential for inaccurate outputs, such as hallucinations or biased content, due to the underlying data or algorithms. Lawyers must independently verify and review the accuracy of GAI outputs and should not rely solely on these tools without applying their professional judgment. Continuous learning and staying updated with advancements in GAI technology are necessary to maintain competence.

Confidentiality:

• Protecting client information is paramount when using GAI tools. Lawyers must evaluate the risks of unauthorized disclosure or access, particularly when using self-learning GAI tools. These tools can potentially expose client information in unintended ways, necessitating informed consent before inputting sensitive data. The opinion emphasizes that informed consent must be specific and clear, detailing the risks and benefits of using such tools. General boilerplate provisions in engagement letters are insufficient for this purpose.

Communication:

• Lawyers are required to inform clients about the use of GAI tools when it impacts the representation. This includes situations where the use of GAI affects fees, decision-making processes, or significantly influences case outcomes. Disclosure is also necessary if clients inquire about the use of these tools. Lawyers must provide adequate explanations to enable clients to make informed decisions, adhering to Model Rule 1.4.

Supervisory Responsibilities:

• Supervisory lawyers must implement policies and training programs to ensure the ethical use of GAI tools within their firms. This includes overseeing both lawyers and non-lawyers to ensure compliance with professional standards. Training should cover the ethical and practical aspects of using GAI tools, including data security, privacy, and the limitations of these technologies. Supervisors must ensure that any use of GAI tools by non-lawyers aligns with ethical guidelines and does not compromise client confidentiality or the quality of legal services.

Fees:

• Lawyers must charge reasonable fees for the use of GAI tools, clearly communicating the basis for these charges to clients. They cannot bill clients for time spent learning to use GAI tools unless specifically agreed upon. If a GAI tool is used to expedite tasks, the fees must reflect the actual time spent and the efficiency gained. Disbursements related to GAI tools must be reasonable and transparently communicated, avoiding any additional profit beyond the actual cost incurred.

These ethical considerations underscore the importance of responsible and transparent use of GAI tools in legal practice. The ABA’s guidance helps ensure that the adoption of these technologies enhances legal services while maintaining the profession’s highest ethical standards.

It is especially encouraging to see the explicit recognition of the need for continuous vigilance given the dynamic evolution of this technology. The opinion holds that lawyers must stay updated with technological advancements and ethical standards to provide competent legal services and, critically, that further guidance is anticipated as GAI tools and their applications evolve.

Building on State-Level and MIT Initiatives

It’s noteworthy that the ABA’s guidance specifically cites the exemplary work done by state bar associations that have released rules and ethics opinion on the topic. This acknowledgment reflects a growing consensus across the legal community about the importance of addressing generative AI in legal practice. Moreover, it’s encouraging to see that ideas and approaches originating from initiatives like the MIT Task Force on Responsible Use of Generative AI for Law are now being more fully integrated into mainstream legal thinking.

The ABA’s Formal Opinion 512 represents a significant milestone in the legal profession’s journey towards embracing generative AI. By providing clear, thoughtful guidance on how to apply existing ethical rules to this new technology, the ABA is not standing in the way for lawyers to responsibly harness the power of AI to enhance their practice and better serve their clients. As the legal landscape continues to evolve, this opinion will undoubtedly serve as a crucial reference point for lawyers navigating the exciting intersection of law and artificial intelligence.

The testimony can be found here, and is embedded below.

The full hearing (which covered several interesting topics) can be found here.

A "Landscaping" Conundrum

The case itself involved an insurance dispute hinging on whether installing an in-ground trampoline qualified as "landscaping." Sounds simple, right? But as Judge Kevin Newsom of the Eleventh Circuit Court of Appeals discovered, pinpointing the "ordinary meaning" of even seemingly straightforward terms can be trickier than it seems.

Judge Newsom, a self-described "plain-language guy," dutifully consulted his dictionaries. Yet, the definitions he found felt incomplete, failing to fully capture the essence of how the word "landscaping" is used in everyday life. He even examined photos of the trampoline installation, his intuition telling him it didn't quite fit the bill. But he struggled to articulate *why*. As he noted, “Nothing in them really struck me as particularly 'landscaping'-y.”

From Dictionaries to Doubt...and an AI Assist

Enter ChatGPT. Out of frustration, Judge Newsom decided—almost as a joke—to ask the AI chatbot for its take on the meaning of "landscaping." To his surprise, ChatGPT delivered a reasoned definition that resonated with his own developing understanding of the term. "Perhaps in a fit of frustration, and most definitely on what can only be described as a lark, I said to one of my clerks, ‘I wonder what ChatGPT thinks about all this.’”

Intrigued, Judge Newsom ventured further. What did ChatGPT think about the trampoline? Could *that* be considered "landscaping"? The AI's answer—a confident "yes," backed by logical explanations—pushed Judge Newsom to consider the potential of this technology in a whole new light. ChatGPT responded: "Yes, installing an in-ground trampoline can be considered a part of landscaping... It’s a deliberate change to the outdoor environment, often aimed at enhancing the overall landscape and usability of the area."

The Promise (and Perils) of AI-Powered Interpretation

Judge Newsom suggests that the appeal lies in AI's ability to tap into something fundamental about legal interpretation: the importance of understanding how words are *actually* used by ordinary people. Large language models (LLMs), trained on vast datasets of online text, offer a potential window into this everyday usage. As he puts it, “The ordinary-meaning rule... has always emphasized 'common language', 'common speech', and 'common parlance'—in short, as I’ve explained it elsewhere, 'how people talk.'”

Moreover, unlike static dictionary definitions, AI can analyze language in context, recognizing nuances and shades of meaning that traditional methods might miss. This "contextual intelligence" is key to unlocking the intended meaning of legal texts. Judge Newsom explains, "The combination of the massive datasets used for training and this cutting-edge ‘mathematization’ of language enables LLMs to absorb and assess the use of terminology in context.”

However, Judge Newsom doesn't shy away from the potential pitfalls. He acknowledges the risk of AI "hallucinations"—generating inaccurate or misleading information—underscoring the need for careful scrutiny of their outputs. “First, the elephant in the room: What about LLMs’ now-infamous 'hallucinations'?" He also raises concerns about potential biases within training data, emphasizing the importance of transparency and inclusivity: “The absence of offline usages from the training pool—and in particular, the implications for underrepresented populations—strikes me as a sufficiently serious concern.”

Kudos to Judge Newsom

Judge Newsom deserves kudos for his courage and foresight. Including the prompts and outputs from ChatGPT as a published appendix in his opinion is not only transparent but also very useful. Moreover, his willingness to share this view, which challenges conventional wisdom, is commendable and reflects an openness to innovation that is crucial for the legal field. As Judge Newsom rightly notes, the experiment with ChatGPT "no longer strikes me as ridiculous" and indeed "might have something useful to say about the common, everyday meaning of the words and phrases used in legal texts."

Proceed with Caution...and Curiosity

So, can ChatGPT replace judges and lawyers? Not quite. But Judge Newsom's experiment offers a compelling glimpse into a future where AI could play a valuable role in legal interpretation. As he aptly puts it, LLMs should be viewed as one tool among many, offering additional insights that can help us better understand the law and ensure its fair and just application. “It seems to me scarcely debatable that the LLMs’ training data are at the very least relevant to the ordinary-meaning analysis.”

The conversation is just beginning, and Judge Newsom is among the first to highlight the true potential of AI in the legal field. While many in legal circles have yet to fully grasp how well this technology fits various legal tasks and processes, Judge Newsom's concurring opinion is just the beginning. This technology is unusually well suited for a wide range of legal interpretation, analysis, and other key functions. As AI continues to develop, the intersection of artificial intelligence and the law promises to be a transformative space to watch.

I’d been digging deep into GPT-3 throughout the previous year, working with Megan Ma and others to clarify how well or poorly that earlier version of the technology could understand and correctly apply the fiduciary duty of loyalty and other related legal frameworks and rules. I was impressed by what GPT-3 could do, and well aware of its limitations as well. But when I got my hands on GPT-3.5, which is the model that powered ChatGPT, I was - and continue to be - astonished at its human-like natural language capabilities. Within a couple of weeks, I determined the ChatGPT model was performing well on the same fiduciary duties tests and evaluations that had stymied GPT-3 and was able to go well beyond anything I had previously devised to test it. I had to invent new tests just to begin to map the contours of the new capabilities. Here is a snapshot of some of those early tests, from Dec 17th of 2023: https://www.civics.com/pub/chatgpt-session-2022-12-17/ Later this year, when GPT-4 was finally released to the public, it blew away even those boundaries.

I’d first seen GPT-4 a few months before ChatGPT was released, when Pablo Arredondo showed me a private demo of his pre-release version of CoCounsel. So I was aware there were even more powerful models in the pipeline, but being aware and being able to have open, easy, web-accessible access are two very different things. When OpenAI, to their credit, made GPT-4 available to everyone through their premium access and the API, it was a revelation for those of us who apply this technology to legal use cases and more broadly to solve business challenges or realize creative new ideas in industry and the professions.

Meanwhile, I hasten to add there are still serious limits and flaws with this technology. It is prone to so-called hallucination and providing factually incorrect information, for example, and there are myriad conundrums about the role of intellectual property and personal or confidential data, to name a few issues. I have found that anchoring the technology to authoritative data, such as by adding that information to the prompt (e.g., via the context window) or through processes like Retrieval Augmented Generation, the hallucinations go down and the factual grounding goes way up, as well as the reasoning-like process.

In light of these shortcomings, and to begin providing guidance on the responsible use of GenAI for law, I started a Task Force at MIT on the Responsible Use of Generative AI for Law and Legal Processes, which convened a group of super-stars who collaborated to develop some solid, if preliminary, principles and guidelines (published here, toward the bottom of the page: law.MIT.edu/ai). I’m delighted to say that I also served as an advisory member of the California State bar association working group that started with the MIT draft and formulated it into a very solid (not perfect, but very good) set of more formal guidance for attorneys to observe when using this technology as part of their practice (available here: https://board.calbar.ca.gov/docs/agendaItem/Public/agendaitem1000031702.pdf).

The primary need among attorneys at this point in time is to learn about this technology and to acquire skills in knowing how, when, and for what to use the technology. Much of this can happen by simply using the technology with an eye toward experimentation and exploration. To that end, over the past year I’ve personally shared quite a few resources for lawyers, as well as other professionals (e.g., in tax, audit, consulting, etc.) on the emerging skill known as prompt engineering, both through open and free resources at law.MIT.edu/ai and more so through private consulting and workshops. Next year, I’m on track to release several more resources and to provide a few new services to make these skills ever more accessible. More on that in the weeks and months to come.

The past year has seen many people and groups raise a lot of fears and objections and outright resistance to this technology, for a range or reasons and from a range of perspectives and priorities. I think this is natural and I know it is to be expected, and yet, in my view, the temptation - especially among lawyers and institutions of the law - to react with prohibitions and overly restrictive regulations is a mistake. Appropriate regulation and policy should balance the enormous utility of this technology against the largely speculative risks, and in general, the main thrust of investment and policy at this point should be toward the beneficial adoption, adaptation, and leveraging of this very useful new technology and the capabilities it affords. Nonetheless, there are demonstrable limits and flaws with the technology and there remains a distance to go before these and other issues are fully addressed.

In the arena of law, in 2024 we’ll see more helpful guidance along the lines of the pioneering guidelines by the California Bar late this year to help lawyers use and integrate GenAI into their practices in a responsible and ethical manner.

I expect we will also see more focus on the second-order implications of this technology in broader institutional contexts, such as updated training of law students and new hire lawyers, better judicial processes (and perhaps more importantly, non-judicial alternative dispute resolution systems) to make this technology available in effective ways for unrepresented and under-represented litigants and criminal defendants, and some potential reforms in the rules of evidence to better deal with the coming wave of deep fakes, among many other ripples in law and legal processes.

More broadly, I foresee a new area of off-line, on-premises, and even on-device LLMs taking hold this year. For example, this week many tech enthusiasts are talking about llamafile (https://hacks.mozilla.org/2023/11/introducing-llamafile/), a groundbreaking multi-gigabyte file that revolutionizes personal computing by bundling both the model weights and the necessary code to run Large Language Models like ChatGPT on your own device. That’s right - you can now easily run a functional LLM on your desktop and even on your laptop! This innovation marks a significant leap in making advanced AI more accessible, bridging the gap between professional AI applications and everyday tech enthusiasts.

A key implication of this on-device approach to running GenAI is that users can now integrate their own confidential, proprietary, and otherwise sensitive data in a completely air-gapped system. I’ve already been exploring the usefulness and security benefits of this approach with enterprise clients of my consulting company, CIVICS.com, but for this blog post, I want to connect this capability to something even more important, namely, YOU! What I mean is that individuals will soon have the tools needed to easily run our own powerful GenAI systems and we’ll be able to connect the rich treasure trove of our personal data to anchor the technology to our contexts, our knowledge, our relationships, and our unique goals and priorities. My esteemed friend Doc Searls has recently begun speculating about the advent of so-called “Personal AI” (e.g., https://projectvrm.org/2023/11/11/individual-empowerment-and-agency-on-a-scale-weve-never-seen-before/) and I predict this will be among the true killer apps (or more accurately, sets of connected apps) to drive adoption and beneficial use of GenAI in the coming year and beyond.

I also see the emergence of automated or quasi-autonomous personal agents and the increasing integration of Generative AI with a wide set of existing widely used apps and platforms not only as major trends in their own right, but also as capabilities that will be super-charged by on-device models with private access to personal and sensitive data.

In my law.MIT.edu capacity, I’ll be working with our team to kick-start 2024 with some major GenAI initiatives, including the annual MIT IAP Computational Law Workshop happening this January with a remarkable set of speakers, topics, and learning activities, and an associated GenAI Online Legal Hackathon. We’ll be announcing those shortly, and to get on the list you can use our pre-registration form here: https://forms.gle/92WwhEWwpGdLyfE5A The MIT Computational Law Report is also about to announce a special collection on GenAI for Law, which will be featuring written works as well as open source applications and Jupyter or Colab Notebooks representing a range of legal use cases that can be achieved with GenAI. And that’s just January!

In my private capacity, through the CIVICS.com consultancy, I’ll be leaning into projects with companies, law firms, and some open source initiatives, who are making innovative use of GenAI both to make their current work faster, less expensive, and better as well as for creating totally new types of products, services, and even novel lines of business. I have also totally revised my standard lunch-talk and private workshop offerings to provide more accessible and flexible opportunities for companies and legal teams to bring me in for learning sessions or to focus on emerging projects. If you’d like to set up a consultation, a talk, or workshop in 2024, reach out through CIVICS.com here: https://www.civics.com/contact

In my public capacity, I plan to keep contributing to standards efforts and professional association efforts, such as through my membership on the ABA Task Force on AI and contributing to open and free community building and skill sharing efforts, such as through meetups, hackathons, and my favorite group Legal Hackers. In a couple of weeks, for example, anybody who wants to meet up with a group of like-minded creative types to share generative AI prompts, ideas and solutions is invited to join the Bay Area Legal Hackers Happy Hour in Oakland. You can learn more and register for this event here.

Looking ahead, I predict two overarching GenAI trends in 2024, first a lot of catch-up by companies, teams, and individuals who are currently aware of this new technology but only have a superficial understanding of it and few skills in using it. We are still at the early part of the adoption curve, and that is normal. We will climb that curve in the coming year. Second, I foresee a number of major changes and breakthroughs in the technology itself, both in the form of integrations of the technology in current common products as well as totally new capabilities and deployment models. I mentioned on-device and secure personal data design patterns as one example of a new deployment model, and there are many others in the pipeline and some that have not even been conceived of yet.

In the face of all this emerging change, the task today is to get educated. Now.

It was truly an honor to share the stage with co-panelists Zachary St. Martin, Head of US Commercial & IP Legal, TikTok & ByteDance, and Michele C. Lee, Deputy General Counsel, Pinterest under the skillful facilitation of moderator Ian Ballon, Co-Chair, Global Intellectual Property & Technology Practice Group, Greenberg Traurig LLP, Moderator.

We discussed how the legal system will come to grips with the massive power of AI, its ability to absorb the global aggregation of visual, written as well as musical data and reproduce, imitate, portray, reinterpret, or in one way or another, manifest that data into some other state of intellectual property.  And we talked about the future!

I learned a lot about the details of how copyright law is implicated in some of the major recent cases involving generative AI, and I shared a few things about the technology itself that may be relevant to the legal analysis and how the law approaches generative AI.

Here’s a direct link to the video of this session: https://www.digitalhollywood.com/ai-summit---session-ten

These regulations will be administered by the Cyberspace Administration of China (CAC), the nation's principal internet regulator. However, the new rules exempt generative AI research and technologies specifically developed for use abroad.

These regulations also mandate developers to register their algorithms with the Chinese government. Furthermore, if their services are regarded as having a "social mobilization ability" that could potentially sway public opinion, they must undergo a "security assessment". This seems like an extension of existing Chinese efforts to control online discourse.

The new law emphasizes the requirement to "adhere to core socialist values". It details various prohibited uses of generative AI, ranging from those aimed at citizen protection—such as a ban on promoting terrorism and spreading obscene pornography—to those designed to consolidate government authority over this emerging technology. For instance, it prohibits the use of generative AI to subvert state power, damage the country's image, and undermine national unity.

The new rules were formulated by the CAC but were also approved by seven other agencies including the Ministry of Education, the Ministry of Public Security, and the State General Administration of Radio and Television. These developments suggest that China's regulation of generative AI will continue to evolve as the technology itself advances.

Here is an initial English translation (via Claude 2) of the new rules:

Interim Measures for the Administration of Generative AI Services

The Office of the Central Cyberspace Affairs Commission and the National Internet Information Office have approved the issuance of Order No. 15 “Interim Measures for the Administration of Generative AI Services” at the 12th office meeting on May 23, 2023. It has also been agreed upon by the National Development and Reform Commission, Ministry of Education, Ministry of Science and Technology, Ministry of Industry and Information Technology, Ministry of Public Security, and National Radio and Television Administration. The measures are hereby promulgated and will come into force on August 15, 2023.

Director of the Office of the Central Cyberspace Affairs Commission, Zhuang Rongwen

Director of the National Development and Reform Commission, Zheng Jiele

Minister of Education, Huai Jinpeng

Minister of Science and Technology, Wang Zhigang

Minister of Industry and Information Technology, Xin Zhonglong

Minister of Public Security, Wang Xiaohong

Director of the National Radio and Television Administration, Cao Shuhui

July 10, 2023

Chapter I General Provisions

Article 1 These measures are formulated in accordance with the Cybersecurity Law of the People's Republic of China, Data Security Law of the People's Republic of China, Personal Information Protection Law of the People's Republic of China, Law of the People's Republic of China on Progress of Science and Technology and other laws and regulations, in order to promote the healthy development and standardized application of generative AI, safeguard national security and public interests, and protect the lawful rights and interests of citizens, legal persons and other organizations.

Article 2 These measures shall apply to services that provide generated text, images, audio, videos and other content to the public in China using generative AI technologies (hereinafter referred to as generative AI services). Where the state has separate provisions on the use of generative AI services for activities such as news publishing, film and television production, and literary creation, those provisions shall prevail.

Research, development and application of generative AI technologies by industry organizations, enterprises, educational and scientific research institutions, public cultural institutions and relevant professional institutions that do not provide generative AI services to the domestic public shall not be subject to the provisions of these measures.

Article 3 The state adheres to the principle of balancing development and security, and combining innovation promotion and rule of law. Effective measures shall be taken to encourage innovative development of generative AI, generate positive and uplifting high-quality content, explore optimized application scenarios, and build an application ecosystem.

Article 4 Providing and using generative AI services shall abide by laws, regulations, respect social ethics and morals, and comply with the following provisions:

(1) Uphold socialist core values, and shall not generate content that incites subversion of state power, overthrow of the socialist system, endangers national security and interests, damages the national image, incites division of the country and undermines national unity and social stability, advocates terrorism and extremism, advocates ethnic hatred and discrimination, violence, obscenity and pornography, as well as false and harmful information prohibited by laws and regulations.

(2) Take effective measures in algorithm design, training data selection, model generation and optimization, service provision and other processes to prevent discrimination based on ethnicity, beliefs, nationality, region, gender, age, occupation, health status, etc.

(3) Respect intellectual property rights and business ethics, keep trade secrets, and shall not use algorithm, data, platform and other advantages to engage in monopolistic behavior and unfair competition.

(4) Respect the lawful rights and interests of others, shall not endanger the physical and mental health of others, and shall not infringe upon others' rights of portrait, reputation, honor, privacy, and personal information.

(5) Based on the characteristics of the service types, take effective measures to improve the transparency of generative AI services and the accuracy and reliability of generated content.

Chapter II Technological Development and Governance

Article 5 Encourage innovative applications of generative AI technologies in various industries and fields to generate positive, healthy and uplifting high-quality content, explore optimized application scenarios, and build application ecosystems. Support industry organizations, enterprises, educational and scientific research institutions, public cultural institutions and relevant professional institutions in carrying out collaboration in generative AI technology innovation, data resource construction, transformation and application, risk prevention and other aspects.

Article 6 Encourage independent innovation in core technologies such as generative AI algorithms, frameworks, chips and supporting software platforms, carry out fair and mutually beneficial international exchanges and cooperation, and participate in the formulation of relevant international rules for generative AI. Promote the construction of public training data resource platforms and generative AI infrastructure. Promote collaborative sharing of computing power resources and improve the efficiency of computing power resource utilization. Promote classified and orderly opening up of public data to expand high-quality public training data resources. Encourage the use of secure and trustworthy chips, software, tools, computing power and data resources.

Article 7 Generative AI service providers (hereinafter referred to as providers) shall carry out preprocessing, optimized training and other training data processing activities in accordance with the law, and comply with the following provisions:

(1) Use data and basic models from lawful sources;

(2) Where intellectual property rights are involved, do not infringe upon intellectual property rights lawfully enjoyed by others;

(3) Where personal information is involved, obtain individual consent or comply with other circumstances stipulated by laws and regulations;

(4) Take effective measures to improve the quality of training data and enhance the authenticity, accuracy, objectivity and diversity of training data;

(5) Other relevant provisions of laws and regulations such as the Cybersecurity Law of the People's Republic of China and the Data Security Law of the People's Republic of China, and relevant regulatory requirements of competent authorities.

Article 8 For data annotation conducted during the research and development of generative AI technologies, providers shall formulate clear, specific and operational annotation rules in line with the requirements of these measures; carry out annotation quality assessment and sample verification of annotation accuracy; provide necessary training for annotators to enhance awareness of law-abiding, and supervise and guide annotators to carry out annotation work in a standardized manner.

Chapter III Service Regulations

Article 9 Providers shall perform the responsibilities of network information content producers in accordance with the law and fulfill network information security obligations. Where personal information is involved, they shall perform the responsibilities of personal information processors and fulfill personal information protection obligations in accordance with the law.

Providers shall sign service agreements with registered users of their generative AI services (hereinafter referred to as users) to clarify the rights and obligations of both parties.

Article 10 Providers shall specify and disclose the applicable population, occasions, purposes and other aspects of their services, guide users to have a scientific and rational understanding of generative AI technologies and use them in accordance with the law, and take effective measures to prevent excessive reliance or addiction of minor users to generative AI services.

Article 11 Providers shall perform protection obligations for users' input information and usage records in accordance with the law. They shall not collect unnecessary personal information, illegally retain input information and usage records that can identify users, or illegally provide users' input information and usage records to others. Providers shall handle users' requests for access, copy, correction, supplementation and deletion of their personal information in a timely manner according to law.

Article 12 Providers shall identify generated image and video content in accordance with the Management Regulations on Deep Synthesis of Internet Information Services.

Article 13 Providers shall provide safe, stable and sustained services during service provision to ensure normal use by users.

Article 14 If providers discover illegal content, they shall take disposal measures such as stopping generation, stopping transmission, elimination in a timely manner, take rectification measures such as model optimization training, and report to relevant competent authorities. If providers discover users engaging in illegal activities using generative AI services, they shall take disposal measures such as warning, restricting functions, suspending or terminating services to users in accordance with the law and agreements, keep relevant records, and report to relevant competent authorities.

Article 15 Providers shall establish and improve complaint and reporting mechanisms, set up convenient complaint and reporting channels, publicize processing procedures and feedback time limits, and promptly accept, handle public complaints and reports and provide feedback on processing results.

Chapter IV Supervision and Inspection and Legal Liabilities

Article 16 Authorities such as cyberspace affairs, development and reform, education, science and technology, industry and information technology, public security, radio and television administration, press and publication shall strengthen the administration of generative AI services according to their respective responsibilities. Relevant competent authorities shall improve scientific supervision methods suited to the characteristics of generative AI technologies and their application services in relevant industries and fields, and formulate corresponding classified and hierarchical supervision rules or guidelines.

Article 17 Providers of generative AI services with public opinion attributes or social mobilization capabilities shall carry out safety assessments and complete the filing and change, cancellation filing procedures for algorithm recommendation management in accordance with relevant national regulations.

Article 18 Users who discover generative AI services that do not comply with laws, regulations and the provisions of these measures shall have the right to complain and report to relevant competent authorities.

Article 19 Relevant competent authorities shall carry out supervision and inspection of generative AI services according to their duties. Providers shall cooperate in accordance with the law, explain the sources, scale, types, annotation rules, algorithm mechanisms, etc. of the training data as required, and provide necessary technical support, data and other assistance.

Relevant agencies and personnel participating in security assessments and supervision and inspection of generative AI services shall keep confidential the state secrets, commercial secrets, personal privacy and personal information that they learn of when performing their duties according to law, and shall not disclose or illegally provide them to others.

Article 20 For generative AI services originating from outside the territory of the People’s Republic of China that fail to comply with the laws, administrative regulations and these Measures, the national cyberspace affairs authority shall notify relevant agencies to take technological measures and other necessary measures for disposal.

Article 21 Providers that violate the provisions of these Measures shall be punished by relevant competent departments in accordance with the Cybersecurity Law of the People’s Republic of China, Data Security Law of the People’s Republic of China, Personal Information Protection Law of the People's Republic of China, Law of the People's Republic of China on Progress of Science and Technology and other laws and administrative regulations. For violations without stipulations in laws and regulations, relevant competent departments shall issue warnings, make public criticisms, order corrections within a time limit according to their duties. Those refusing to correct or with serious circumstances shall be ordered to suspend relevant services. Those constituting violations of public security administration shall be punished for public security administration according to law. Those constituting crimes shall be investigated for criminal responsibility according to law.

Chapter V Supplementary Provisions

Article 22 The definitions of the following terms in these Measures are:

(1) Generative AI technology refers to models and related technologies with capabilities of generating content such as text, images, audio and video.

(2) Generative AI service providers refer to organizations and individuals that provide generative AI services using generative AI technologies (including providing generative AI services by offering programmable interfaces).

(3) Generative AI service users refer to organizations and individuals that generate content using generative AI services.

Article 23 Where laws and administrative regulations require administrative licensing for the provision of generative AI services, providers shall obtain the license in accordance with the law. Foreign investment in generative AI services shall comply with relevant laws and administrative regulations on foreign investment.

Article 24 These Measures shall come into force as of August 15, 2023.

To illustrate this approach, I’ll provide examples of using success criteria to evaluate and improve outputs generated by an AI in response to initial prompts. I’ll discuss several tools and methodologies for setting goals and measuring outcomes, and I’ll show how AI itself can aid the prompt engineering process.

The possibilities with large language models are vast, but we need to give them direction. The best way I know what direction to go is with reference to my destination. You could say that a destination is like a goal you can measurably arrive at. By focusing on success criteria for achieving a goal, we can compose prompts that steer AI toward valuable, purposeful responses. My hope is that this “Success is All You Need” approach inspires you to define the goals you have in the back of your mind when you're using an LLM, to imagine the desired end results, and to craft prompts to get you there.

Naturally, there are lots of reasons for using LLMs, and some of them don't require or even benefit from starting with a goal like this. LLMs are great for open brainstorming, for example. But sometimes, we do have things we want to, or need to, accomplish. There are a lot of ways LLMs can help us achieve those outcomes. Let's jump right in, and share what we learn along the way.

The capabilities of modern LLMs like GPT-4, Claude+, and Bard to create new value are enormous, but we are at the very beginning of understanding how to access them. To get better outputs, a lot of progress can be made by improving the inputs, and you do that by composing more effective prompts. To get the best results for what you hope to achieve, the real magic of better prompting starts with a certain kind of thinking about what it is you actually want and how you would objectively know when you get it.

As prompt engineering and related work on improving the use of LLMs evolve, I continually come back to the need to measure success. There are many, many different aspects of LLMs that would benefit from the right success measures, but today I want to narrowly focus on some ways to apply success metrics to the end-user art and skill of prompting the rights things in the right way. The kind of success metric I want to talk about is the objective assessment we use to evaluate whether a business hypothesis has been proven.

First, a little background on this relatively unknown thing called a "business hypothesis". I've been on teams using various ways to establish success metrics for technology projects since the late 1990's, but it wasn't until the 2010's that I encountered the idea of a business hypothesis that can be used as a way of doing business. It was The Lean Startup book that made it clear to me how aspirational business goals can be framed in a way that is testable, measurable, and iteratively achievable.

The essential insight is to be able to articulate "we will know we have succeeded when" a given objective criterion is met. This depends on a clear problem statement or goal definition, and this can't be applied without objective measurement, but the main idea is to be able to somehow pin down the quintessential marker of when that goal has been attained. For me, the fundamental value of this approach is not simply in measuring relevant things, checking criteria boxes, and reporting successfully performing activities. Rather, it's about verifiably achieving the intended outcomes.

When I realized this type of objective experimentation could be applied outside of the scientific method, it changed the way I approached many challenges and problems in business and has even extended to certain legal and policy initiatives.

Problems, Goals, and Measuring What Matters

Before thinking about success, of course, the very first step is defining a problem to be solved or a goal to be achieved. This is, in fact, the initializing point. And I have noticed a recent uptick in posts about how "problem formulation" is key to effective prompts (h/t to Liz Chase and Jon Neiditz for talking about this trend). I could not agree more. And yet, problem definition is just the starting line. The finish line requires connecting the dots between 1) the problem and 2) how to objectively know when the solution occurs.

Moreover, when it comes to dreaming up and building new things, I think we can frequently get better results by thinking in terms of positive future goals to be achieved and not necessarily past problems to be struggled with. Some say, what you resist persists. In any event, whether the desired result is a solution to a problem, or achievement of a fresh new goal, be sure to have clear, objective criteria for establishing when you are actually done.

There are a LOT of ways people and organizations have approached measuring aspects of success beyond the scientific method, per se. Some of them include:

• Key Performance Indicators (KPIs): These are quantifiable measurements that reflect the critical success factors of an organization. They are a way of measuring performance against strategic and operational goals. KPIs help organizations understand if they're heading in the right direction—and if not, where they need to divert their attention to improve. It's common for KPIs to be tied to specific strategic objectives, such as increasing customer satisfaction or improving operational efficiency.

• Objectives and Key Results (OKRs): This is a goal-setting framework that helps organizations set ambitious goals with measurable results. OKRs are typically implemented on a quarterly basis and consist of an Objective, which defines a goal to be achieved, and up to 5 Key Results, which are specific measures used to track the achievement of that goal. The aim of OKRs is to ensure that everyone in the organization is moving in the same direction with transparency and clarity on expected outcomes. They are meant to set strategy and goals over a specified amount of time for an organization, teams and individuals.

• Balanced Scorecard (BSC): The Balanced Scorecard is a performance metric used in strategic management to identify, improve, and control a business's various functions and resulting outcomes. It includes financial measures that tell the results of actions already taken and complements those with measures on lead indicators: customer satisfaction, internal processes, and learning and growth.

• SMART Goals: SMART stands for Specific, Measurable, Achievable, Relevant, and Time-bound. This system is used for setting objectives in project management, employee performance management, and personal development.

• Management by Objectives (MBO): This is a management model aimed at improving the performance of an organization by clearly defining objectives that are agreed to by both management and employees.

• Key Result Area (KRA): KRAs refer to the general areas of outcomes or outputs for which a department's or individual's role is responsible. A typical role targets three to five KRAs.

• Critical Success Factors (CSFs): These are elements that are vital for a strategy to be successful. CSFs provide the key goals or objectives for your business and keep teams focused on the same crucial areas of business.

• Performance Prism: This is a performance measurement framework that focuses on stakeholders. It examines the relationship between the organization's stakeholders' wants and needs, strategies, processes, capabilities, and stakeholder contributions.

• Hoshin Kanri: Also known as Policy Deployment, it is a strategic planning method used to ensure that the strategic goals of a company drive progress and action at every level within the company.

I'm directly familiar with OKRs and KPIs, based on having been on various teams for which these were required aspects of the work. I'm somewhat familiar with some others, but tbh, I just asked an LLM to help me identify good examples (a link to that ChatGPT session is included below). And yes, I've at least Googled enough to confirm they all exist!. I think each of these has some useful information and perspective on ways of measuring success that can be applied to prompt composition and evaluation of LLM outputs. The formulation "I'll know this has succeeded when XYZ" remains my anchor for the purposes of this post. Now, let’s look at some examples!

Examples of Goals, Success Criteria, and Measuring Options

Example 1 - DMV Innovation Strategy

Prompt 1: Generate three different plans a big four consultant can deliver for an innovation strategy to a state department of motor vehicles. A successful plan meets each of the following three success criteria: 1) significantly and measurably improves service delivery for individual citizen license renewals and businesses such as taxi, trucking, and delivery fleets; 2) leverages COTs technology and widely used open source; and 3) will eventually reduce ongoing costs for the department of motor vehicles.

Prompt 2: Rate each plan against each of the three success criteria. Display in a single markdown table with rows for each plan and columns for each success criteria. Rate from 1 (lowest) to 5 (highest). Include a row below each plan to describe the rationale for each rating. In a row below the rationale, describe how success would be measured.

Output for DMV Prompt 1

Plan 1: Digital Transformation and Automation

1. Implement a robust online portal that allows individual citizens and businesses to renew licenses, update information, pay fees, and schedule tests. This portal could leverage a combination of COTS (Commercial Off-The-Shelf) products and open-source technologies like Java or Python for backend development, React or Angular for frontend, and PostgreSQL or MongoDB for database management.

2. Invest in AI-powered chatbots to handle queries, guide users through processes, and resolve issues in real-time, thereby reducing the load on DMV personnel. This could involve the use of open-source machine learning libraries such as TensorFlow or PyTorch.

3. Introduce a queue management system to minimize waiting time and improve efficiency in physical offices. This can be achieved by integrating COTS products with existing systems.

4. Optimize internal operations by automating administrative tasks, such as record keeping, compliance monitoring, and reporting, using Robotic Process Automation (RPA) tools.

Plan 2: Data-Driven Innovation

1. Develop a unified data platform, integrating data from various sources into a single, easy-to-navigate system. Utilize open-source data management systems like Hadoop or Apache Cassandra, and COTS business intelligence tools for data visualization and reporting.

2. Use predictive analytics and machine learning to anticipate service demand, optimize resources, and proactively address potential issues. This can improve customer service by ensuring adequate staffing during peak periods and timely maintenance of systems.

3. Implement advanced analytics to understand trends in customer behavior, preferences, and dissatisfaction areas. Use this insight to refine the DMV's services continuously.

4. Employ IoT technology in monitoring and maintaining fleet vehicles for businesses. This would involve real-time data collection from vehicles to identify issues before they become major problems, thus reducing downtime and repair costs.

Plan 3: Customer-Centric Innovation

1. Establish a mobile application, using open-source mobile development frameworks like React Native or Flutter, to provide on-the-go access to DMV services. The application could include features like real-time tracking of application status, digital licenses, reminders for renewals, and online payment options.

2. Utilize COTS Customer Relationship Management (CRM) systems to improve customer service by providing personalized assistance, tracking customer interactions, and improving communication.

3. Introduce self-service kiosks at DMV offices, leveraging COTS technologies, to automate simple transactions like renewals, payment of fines, and information updates.

4. Initiate a feedback system to continuously gather input from customers, identify pain points, and drive continuous improvement. Employ open-source tools for survey generation and data collection, like LimeSurvey or SurveyJS, and data analysis tools like Python's Pandas or R.

All three plans would drive efficiency, improve customer service, and in the long run, reduce operational costs by lowering the requirement of manual work and reducing system downtime. Furthermore, they would make services more accessible to citizens and businesses, hence improving overall satisfaction.

Output for DMV Prompt 2

There are many ways to work with “success criteria” prompting technique. One application is to use it as part of a cycle of reflection, identifying the most fit aspects of multiple options to synthesize an even better output. For example, based on the above plans, one could prompt something like, “Based on this analysis, please devise a single plan that takes the best elements of each of the 3 proposed plans to create a plan that rates overall higher for each criteria. Again, display this in a single markdown table with the plan on a row and each success criteria in columns, and please include a row below the plan to describe how this new plan would perform better than any of the previous plans and a row below that to describe how success would be measured.” But let’s look at another example of this tactic, that shows how to apply it to a situation that has even clearer objective success criteria.

Example 2 - Mobile App

Prompt 1: Generate three different plans a big four consultant can deliver for a mobile app developer to use A/B testing to improve ROI on their least profitable simcity style game with a free tier, a monthly subscription tier, and in-game purchase options. A successful plan meets each of the following three success criteria: 1) it defines a clear and discrete potential improvement for user action; 2) if the improved use action was deployed to the entire customer base costs would reduce or revenue would increase; and 3) the user action would not decrease user satisfaction with the app.

Prompt 2: Rate each plan against each of the three success criteria. Display in a single markdown table with rows for each plan and columns for each success criteria. Rate from 1 (lowest) to 5 (highest). Include a row below each plan to describe the rationale for each rating. In a row below the rationale, describe how success would be measured.

Output for Mobile App Prompt 1

Plan 1: Improving User Engagement through Tutorial Enhancement

Objective: Enhance the introductory tutorial to improve user understanding and engagement, leading to increased conversions to the subscription tier and in-game purchases.

1. A/B Test Hypothesis: Users who understand the game better will engage more and be more likely to make in-game purchases or subscribe. By testing an enhanced tutorial against the existing one, we can evaluate if increased understanding drives revenue.

2. A/B Test Implementation: Split the new user base into two groups. Group A receives the existing tutorial, while Group B receives the enhanced tutorial. Collect data on subsequent user engagement and conversion rates.

3. Expected Outcome: If the enhanced tutorial is effective, Group B should show higher engagement and a higher conversion rate to the subscription tier and in-game purchases.

4. ROI Improvement: Increased conversions would result in higher revenue without necessitating an increase in costs.

5. User Satisfaction: An effective, engaging tutorial should improve the user experience and increase satisfaction, not decrease it.

Plan 2: Balancing In-Game Rewards

Objective: Adjust the balance of in-game rewards to encourage more frequent usage and incentivize in-game purchases.

1. A/B Test Hypothesis: By adjusting the in-game rewards, users may be more motivated to engage more frequently with the game and consider in-game purchases to advance quicker.

2. A/B Test Implementation: Create two versions of the game: one with the current reward structure (Group A) and one with the revised reward structure (Group B). Measure user engagement, frequency of logins, and rate of in-game purchases over a defined period.

3. Expected Outcome: If the hypothesis is correct, Group B should show increased engagement, more frequent usage, and a higher rate of in-game purchases.

4. ROI Improvement: If more users are incentivized to make in-game purchases, revenue should increase without significant additional costs.

5. User Satisfaction: If executed well, adjusting in-game rewards should increase enjoyment and engagement without negatively impacting user satisfaction.

Plan 3: Subscription Tier Benefit Optimization

Objective: Enhance the benefits offered in the monthly subscription tier to increase conversions from the free tier.

1. A/B Test Hypothesis: By offering additional benefits in the subscription tier, users may perceive more value and be more likely to convert from the free tier.

2. A/B Test Implementation: Maintain the current subscription tier benefits for Group A, while Group B receives the enhanced subscription tier. Measure the conversion rate from the free tier to the subscription tier over a set period.

3. Expected Outcome: If the hypothesis holds true, Group B should show a higher conversion rate to the subscription tier.

4. ROI Improvement: An increase in subscription conversions leads directly to increased monthly revenue with no additional costs.

5. User Satisfaction: Enhanced benefits should increase perceived value and satisfaction for users without detracting from the experience of those on the free tier.

Output for Mobile App Prompt 2

It’s useful to see how the same prompts can yield very different results, sometimes generating even more good ideas. Here’s an example of the outputs generated the second time I ran these prompts.

Output for Mobile App Prompt 1 - Second Run

Plan 1: Optimizing the Free to Subscription Tier Transition

Objective: To enhance the conversion rate of users from the free tier to the subscription tier.

1. Proposed A/B Test: Compare two different methods of introducing the subscription tier to free users. Group A will be presented with an introduction to subscription benefits immediately after certain achievements in the game (e.g., upon reaching a new level), while Group B will receive a time-based prompt (e.g., after two weeks of play). The call to action will be prominent in both versions.

2. Potential Improvement: By optimizing the timing and context of introducing the subscription, there can be an increase in the conversion rate from free to paid users.

3. Expected ROI Impact: Increasing the conversion rate will raise subscription revenues without incurring any substantial costs.

4. User Satisfaction: By carefully crafting the presentation and timing of these prompts, it ensures that users are not overwhelmed by constant upsell efforts, thereby preserving user satisfaction.

Plan 2: Enhancing In-Game Purchase Decisions

Objective: To increase the frequency and volume of in-game purchases.

1. Proposed A/B Test: Experiment with two different ways of presenting in-game purchase options. Group A sees the current format while Group B is introduced to a more gamified system where exclusive rewards or discounts are unlocked upon reaching certain milestones in the game.

2. Potential Improvement: The new presentation method may incentivize users to make more in-game purchases in pursuit of exclusive rewards, improving the revenue generated from this source.

3. Expected ROI Impact: By encouraging more in-game purchases, the game's revenue increases. The development costs are low as the new gamified system can be a modification of the existing one.

4. User Satisfaction: If the gamified system for in-game purchases is designed carefully, it can increase the thrill of the game, further enhancing user satisfaction.

Plan 3: Reducing Server Load through Optimized Synchronization

Objective: To reduce server load and hence operational costs without sacrificing the user experience.

1. Proposed A/B Test: Implement two different methods of data synchronization between the user's device and the game servers. Group A will continue with real-time sync, while Group B will use an intelligent sync system that primarily syncs data when players make significant progress or when they're not actively playing.

2. Potential Improvement: An intelligent sync system may reduce server load and associated costs significantly without affecting gameplay experience.

3. Expected ROI Impact: Reduced server load means lower operational costs, thereby improving the game's ROI.

4. User Satisfaction: As long as the intelligent sync system is designed well, players shouldn't notice a difference in their gameplay experience, keeping user satisfaction levels high.

Output for Mobile App Prompt 2 - Second Run

Notice how there is some overlap and some totally new ideas when the same prompts were run a second time? This is, of course, one of the biggest sources of value available with generative AI. In nearly no time and close to no cost, a very large number of ideas can be generated with the right prompts, and techniques like reflection against explicit and specific well-suited success criteria can quickly float the fittest options to the top for consideration and decision by the operator.

While I’m exploring options and idea generation (like the examples above) a lot lately, the “Success is All You Need” technique can also be used in other ways, such as by periodically critiquing and improving several versions of a single draft or other work. Likewise, I’ve started to experiment with using this technique as part of semi-automated workflows, gathering actual data at key junctures from external processes to evaluate against success criteria. There are, doubtless, many other ways to apply this approach as well. Read to the bottom, and you’ll see an opportunity to try some of your own ideas and share then back!

Closing Thoughts and Hopes for the Future

You may be thinking something like, "I have no idea how actually to do any of this". I find myself thinking that all the time. As it turns out, LLMs are also quite good at generating prompts when given even very general information about what is desired. One great way to do this is to prompt something along these lines simply: "Please help me compose a clear and effective prompt or set of prompts to [fill in your goal here]. But you can go much further by specifying you want to help define the problem or goal, then help identify the success criteria. If you are not getting good results from these approaches, another effective prompt is to ask something like "What other information would be needed in order to compose a prompt that" achieves the goal?

In preparation for publishing this post, I also consulted ChatGPT (using the GPT-4 model) to get some quick feedback on whether I was missing anything, especially in the examples above. The chat session is here, and it includes some good tips on ways to be even more specific, prioritizing success criteria, and surfacing risks and challenges, among other things. When it comes to using LLMs better, LLMs are your friend!

I hope this post sparks something in you. I hope you suddenly remember, or spontaneously realize, a problem you really want to solve or (perhaps even better) and shiny new goal you want to achieve and that this approach to using LLMs is useful for you moving forward from that thought to an actionable set of steps to measurably create the better future you imagine.

If you try the "Success is All You Need" prompt technique or some variation of it, I invite and encourage you to share your prompts and the outputs with me. I'd like to keep updating this post and possibly make more posts on various aspects of this, and I'd love to include some of your examples, along with your questions, issues, and ideas about this approach to prompt composition. You can share that through this link (which includes a simple, standard Creative Commons license, so I can post some of the incoming examples in future posts and generally share the knowledge on this topic, with attribution to you of course, if you wish). Here's the form: https://forms.gle/c8j3G9uZwRqxEGXg7

Find the original source video, hosted at Vimeo and courtesy of Journal Technologies, below:

Join the conversation with Olga Mack and me on June 14th, 2023, from 12:00-1:00 pm PST via Zoom to discover the power of generative AI for legal processes. These disruptive technologies revolutionize the way you work. Save the date, and stay ahead of the curve. There will be a Q&A following the presentation. Registration is required. Click here to sign up for the event. This webinar will be recorded. MCLE credit ($36) is available to California attorneys. I’d like to thank the Daily Journal for hosting this event!

Below is a cached video of the session.

During the workshop, participants engaged in an informative session dedicated to explaining LLMs and their relevance to lawyers. The discussion shed light on how LLMs can empower legal professionals by revolutionizing legal research, document drafting, and overall efficiency.

Additionally, the workshop featured a showcase of relevant MIT research projects, demonstrating cutting-edge advancements in natural language understanding and machine learning that have the potential to shape the future of legal services. One research project demo in particular showcased how private and confidential information can remain secure while still unleashing the powerful benefits of this technology.

Another key topic discussed was legal prompt engineering, which provided insights into effectively designing prompts to maximize LLM performance and obtain contextually appropriate legal responses. I led this session, which expanded upon the previous public sessions published at law.MIT.edu with more examples. I also previewed a new approach called "Success Is All You Need" which significantly improves performance as measured by specific goals. I'll post about that in the coming days here on DazzaGreenwood.com.

Notably, we also discussed a collaborative effort by a law.MIT.edu convened Task Force to develop principles and guidelines for the responsible use of Generative AI in the legal profession. This ongoing work aims to ensure that LLMs are implemented in a manner that upholds professional standards and the core values of the legal field. I was joined by Task Force members Olga Mack and Aileen Schultz. You can read more about the workshop in this post by Olga.

I want to thank fellow workshop organizers Sandy Pentland, Thomas Hardjono, and, most significantly, Robert Mahari (who did the lion’s share of the planning and execution).

Embedded below is the program for this workshop.

Here is a high level summary of where things stand, as of today: A recent lawsuit filed by a man named Roberto Mata against the airline Avianca has ignited a profound discussion about the implications of AI-generated content and its use in legal proceedings. The current matter unfolded when Steven A. Schwartz, the lawyer representing Mata, relied on a generative AI model - ChatGPT - to prepare his court filing, citing numerous judicial decisions supposedly relevant to the case. However, it was discovered that these cases, along with the quotes and citations used, were entirely fabricated by ChatGPT. This resulted in a situation where a lawyer submitted a document full of invented cases and quotations to the court. According to the New York Times, Judge Castel (who presides over this case) said in an order that:

he had been presented with “an unprecedented circumstance,” a legal submission replete with “bogus judicial decisions, with bogus quotes and bogus internal citations.” He ordered a hearing for June 8 to discuss potential sanctions.

As I remarked on Twitter, my own immediate reaction when I first became aware of this situation echoed a common caution I express about using generative AI for legal work - “It can generate fabricated results, so it's imperative to critically evaluate and verify the information before using it.”

But the more I considered the facts as they are currently known, the more facts I wanted to know. Specifically, I’d love to see the entire prompt chain leading up to, including, and after the results that were filed with the court. It’s entirely possible the lesson here may not be simply that an unwitting lawyers mistakenly relied upon information that any ordinary person would have taken as fact.

In a blog post by Simon Wilison published earlier today on this matter, he notes:

I’ve been trying to come up with prompts to expand that false “Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019)” case into a full description, similar to the one in the screenshots in this document.

Even with ChatGPT 3.5 it’s surprisingly difficult to get it to do this without it throwing out obvious warnings.

I’m trying this today, May 27th. The research in question took place prior to March 1st. In the absence of detailed release notes, it’s hard to determine how ChatGPT might have behaved three months ago when faced with similar prompts.

So there’s another version of this story where that first set of citations was an innocent mistake, but the submission of those full documents (the set of screenshots from ChatGPT that were exposed purely through the presence of the OpenAI down arrow) was a deliberate attempt to cover for that mistake.

Wilison's insights highlight one plausible scenario. Still, it underscores the fact that there are numerous potential fact patterns that may yet come to light. We may be only scratching the surface of the story at this point, and as further details unfold, more wrinkles and complexities may emerge as well.

Ideally, the proceedings to follow will result in a full and public airing of the entire sequences of prompts leading to the results submitted to the court. Once the full circumstances come to light, it wouldn't be surprising if the key lessons are more about the frailties and foibles of certain individuals and only remotely about the competent and responsible use of generative AI in legal contexts, which was my initial reaction. Or perhaps there are other factors in play. Intriguingly, the facts may take us places we don’t expect. Let’s take care to have the lessons flow from the facts.

My slides are embedded below. I may upload a video as well.