Generating Test Datasets with Effect

07 Sep, 2026

You have written a prompt. It is, by your own estimation, a very good prompt. You tested it once, it produced a lovely answer, and you are now considering shipping it to production.

Please sit down. We need to talk.

Building reliable applications on Claude rests on two disciplines, and you have so far practiced only one of them: Prompt engineering.

Prompt engineering is your toolkit. It's the process of writing better prompts. Multishot prompting, structuring with XML tags, and a host of other best practices that keep evolving everyday. All techniques that help Claude understand what you're asking for and how you want it to respond.

Prompt evaluation, on the other hand, asks a different question. Instead of "how do I write this prompt?", it asks "does this prompt, in fact, work?". The answer is in the medium of automated testing. You test outputs against expected answers (or at least the characteristics of an expected answer). Instead of eyeballing the result againt a handful or examples, we're going to build a rubric or evaluation pipeline that scores your output. Then we iterate on the prompt against our "objective" scores. It's a lot more work. Costs more time and money, and it's not perfect, but it's offers vastly more confidence than a vibecheck.

What does an evaluation pipeline actually look like?

A typical workflow follows five steps:

Step 1 — draft a prompt. We begin by writing an initial prompt we wish to improve. This basic prompt is our baseline. For an example, lets write a prompt that answers questiosn for people: "Please answer the following question..."

Step 2 — create an eval dataset. An evaluation dataset contains sample inputs representing the kinds of questions our prompt will face in production. Ideally across a broad range of scenarios.

Say the dataset contains three questions: "What's 2+2?", "How do I make oatmeal?", and "How far away is the Moon?" A modest panel of arithmetic, porridge, and astronomy. In real-world evaluations you might have hundreds, or thousands of records.

Step 3 — feed through Claude. We take each question, merge it with the prompt template to form a complete prompt, and send each one off. Claude might respond "2 + 2 = 4", offer cooking instructions for the porridge, and report the distance to the Moon for the third. Three questions in, three answers out.

But are they good answers?

Step 4 — feed through a grader. The grader examines both the original question and Claude's answer, and pronounces judgment in the form of an objective score. Typically pass/fail or a scale from 1 to 10.

The grader assigns the math question a 10. Perfect. The Moon question earns a 9. Very good. The oatmeal question limps in at 4.

Average the scores and you have your objective measurement: (10 + 4 + 9) ÷ 3 = 7.66. Now our prompt is no longer "pretty good" It is a 7.66.

Step 5 — change the prompt and repeat. Armed with a baseline, we modify the prompt. Perhaps we add an instruction to answer with more detail and then run the grader. The average climbs to 8.7. The additional instruction helped. The point is that we know this because a number went up, not because it felt nicer.

And that's the whole game. Objective measurements, comparing scores, systematic iteration.

If you have been paying attention then you are probably asking yourself how the grader knows what a 4 is. And shouldn't we also have graders to grade the graders? And then who grades the grader-graders? Yes, you're right the whole thing is a mess. However, instead of thinking about this as a sham, it's more accurate to think about it in terms of margin of error. Vibechecking has a massive margin of error, and systematic testing reduces the margin of error drastically. You cannot elimitate error, but you can minimise it. Grader-graders would certainly reduce the margin of error even further. Our goal at this stage, is less about double-decimal precision and more about practical confidence in our prompts working.

The Prompt We Shall Measure

Lets write a prompt that helps people write AWS-specific code. Three kinds of output: Python code, JSON configuration files, and regular expressions.

A user describes a task; we return clean output in one of those three formats. No explanation. No header. No footer. Just the artifact.

Here is our starting prompt, version 1:

const prompt = `
Please provide a solution to the following task:
${task}
`;

It asks for a solution. That is the full extent of its ambition, and it will be measured accordingly.

What Do We Measure It Against?

A prompt evaluation needs inputs. We take a prompt, feed it an input, run the combination through Claude, and analyze what comes back — and we repeat that for every input we have.

So the first thing we need is not a grader, nor a scoring rubric. It is a pile of inputs. An evaluation dataset: an array of objects, each with a task property describing what we want Claude to accomplish.

In Effect we do not describe that shape in a comment and hope. We describe it in a schema:

const TestCase = Schema.Struct({ task: Schema.String });

One field, one string.

The Trouble with Hand-Written Inputs

Hand-writing a dataset is honest work, and for three items it is even pleasant. But an evaluation that consists of the three tasks you personally thought of will test only the three tasks you personally thought of. The inputs are drawn from the same imagination as the prompt, and they share the same blind spots.

We want variety. You want tasks you would not have written. And you want more than three, eventually, without spending all afternoon inventing AWS chores. Let have Claude generate the dataset for us. There is no reason to summon the full might of Claude. A model like Haiku is the right tool, it's faster and cheaper.

const HaikuLayer = AnthropicLanguageModel.model("claude-haiku-4-5", {
  max_tokens: 1000,
});

Now the prompt that describes the dataset we want:

const generationPrompt = `
Generate an evaluation dataset for a prompt evaluation. The dataset will be used to evaluate prompts
that generate Python, JSON, or Regex specifically for AWS-related tasks. Generate an array of JSON objects,
each representing task that requires Python, JSON, or a Regex to complete.

Example output:
\`\`\`json
[
    {
        "task": "Description of task",
    },
    ...additional
]
\`\`\`

* Focus on tasks that can be solved by writing a single Python function, a single JSON object, or a regular expression.
* Focus on tasks that do not require writing much code

Please generate 3 objects.
`;

We want three things. Tasks that are AWS-related. Tasks solvable by a single function, a single JSON object, or a single regex. And tasks that do not require writing much code. We are testing the prompt, not drafting an infrastructure migration.

Getting JSON We Can Actually Use

Ask Claude for JSON and it will give you JSON, but saran wrapped in a markdown fence and decorated with a cheerful sentence about what it just made.

Something like this:

```json
[
  { "task": "Write a Python function that..." }
]
```

I've generated three AWS-related tasks covering Python, JSON, and regex!

Feed that to a JSON parser and you get a SyntaxError.

Prefilling and Stop Sequences

The prompt option accepts more than a string. It accepts an array of messages where the last message can be role: "assistant":

const response = yield* LanguageModel.generateText({
  prompt: [
    { role: "user", content: generationPrompt },
    { role: "assistant", content: "```json" },
  ],
});

This is assistant message prefilling. Claude receives the conversation and believes it has already begun answering, and that its answer so far consists of an opened code block. All it can do is continue.

Eventually Claude finishes the array, closes the fence with ```, and launches into commentary anyway.

So we add a stop sequence: a string that, the instant it appears in the output, ends generation. It is provider configuration, so we set it the Anthropic way, on this one call:

const response = yield* LanguageModel.generateText({
  prompt: [
    { role: "user", content: generationPrompt },
    { role: "assistant", content: "```json" },
  ],
}).pipe(
  AnthropicLanguageModel.withConfigOverride({ stop_sequences: ["```"] }),
);

Claude reaches for the closing ```, and the gate slams shut.

Decode, Don't Parse

We now have a string containing an array.

Consider what that actually buys us. JSON.parse returns any. Give the surrounding function a return type of Array<{ task: string }> and TypeScript will nod politely and believe you, because you have not proved anything, you have merely asserted it.

That modest schema from earlier helps here:

const Dataset = Schema.parseJson(Schema.Array(TestCase), { space: 2 });

Schema.parseJson wraps a schema in the JSON string boundary. Decoding runs JSON.parse and then validates the result against Schema.Array(TestCase). A response of the wrong shape fails here, loudly.

Decoding is an Effect, so we yield* it like anything else:

return yield* Schema.decodeUnknown(Dataset)(response.text);

Note what the type checker now knows. This expression has type Effect<readonly { task: string }[], ParseError, never>. The success channel earned its shape by validation, and the failure channel carries a typed ParseError, which is a thing you can catch, log, or retry against.

Wrapped up as a reusable function:

const generateDataset = Effect.fn("generateDataset")(function* () {
  const response = yield* LanguageModel.generateText({
    prompt: [
      { role: "user", content: generationPrompt },
      { role: "assistant", content: "```json" },
    ],
  }).pipe(
    AnthropicLanguageModel.withConfigOverride({ stop_sequences: ["```"] }),
  );

  return yield* Schema.decodeUnknown(Dataset)(response.text);
});

Running It

const dataset = yield* generateDataset();

This returns three different test cases, one from each of our target outputs. A run of mine produced:

[
  {
    task: "Write a Python function that extracts the AWS account ID from an ARN (Amazon Resource Name) string. The function should take an ARN like 'arn:aws:s3:::my-bucket' or 'arn:aws:iam::123456789012:role/MyRole' and return the account ID if present, or None if not applicable."
  },
  {
    task: "Create a JSON object that represents an AWS IAM policy allowing a principal to perform s3:GetObject and s3:PutObject actions on a specific S3 bucket named 'my-data-bucket'. Include appropriate resource ARNs and effect statements."
  },
  {
    task: "Write a regular expression pattern that matches valid AWS S3 bucket names. The pattern should enforce AWS naming rules: 3-63 characters, lowercase letters, numbers, hyphens, must start and end with a letter or number, and cannot contain consecutive hyphens."
  }
]

A Python function, a JSON configuration, a regular expression. All AWS-flavored, small enough to solve in a single artifact.

Saving the Dataset

A dataset that lives only in a running process is a dataset you will regenerate every time you evaluate — at the cost of a fresh API call, and with different tasks each time, which rather defeats the purpose of comparing two prompt versions against the same inputs.

Let's write it to a file. Touching the filesystem is an effect like any other, and Effect asks us to say so by taking the service from context:

const fs = yield* FileSystem.FileSystem;
const path = yield* Path.Path;
const file = path.join(import.meta.dirname, "dataset.json");

Two services, and later a single layer supplies both.

Now, the encoding. You may reach for JSON.stringify here, and it would work. But look again at the schema we built:

const Dataset = Schema.parseJson(Schema.Array(TestCase), { space: 2 });

A schema is a two-way street. Schema.decodeUnknown runs it forwards, string to array. Schema.encode runs it backwards, array to string, and the { space: 2 } option we passed configures the JSON.stringify on that return trip:

const json = yield* Schema.encode(Dataset)(dataset);
yield* fs.writeFileString(file, json);

The same basic mechanism works in both directions. The shape that came in is the shape that goes out, pretty-printed at two spaces, and there is no second place for the two to disagree.

writeFileString returns Effect<void, PlatformError>, a full disk or a read-only directory arrives as a typed failure alongside the ParseError, rather than as a surprise.

The Layers

Three layers, provided at the edge:

const HaikuLayer = AnthropicLanguageModel.model("claude-haiku-4-5", {
  max_tokens: 1000,
});

const AnthropicLayer = AnthropicClient.layerConfig({
  apiKey: Config.redacted("ANTHROPIC_API_KEY"),
}).pipe(Layer.provide(FetchHttpClient.layer));

await program.pipe(
  Effect.provide(HaikuLayer),
  Effect.provide(AnthropicLayer),
  Effect.provide(NodeContext.layer),
  Effect.runPromise,
);

HaikuLayer names the model. AnthropicLayer reads the API key as a Config.redacted, so the key does not wander into a log line. And NodeContext.layer supplies both FileSystem and Path in one go, the Node implementations of the two services our program asked for.

A More Effect Way

The @effect/ai package has LanguageModel.generateObject that does a lot of this work for us.

const DatasetObject = Schema.Struct({ tasks: Schema.Array(TestCase) });

const structured = yield* LanguageModel.generateObject({
  prompt: generationPrompt,
  schema: DatasetObject,
  objectName: "dataset",
});

structured.value.tasks; // decoded, validated, and typed

No prefill, no stop sequence, no separate decode step. The model is steered toward the schema's structure at the request level, and structured.value arrives already decoded.

Observe the one wrinkle. generateObject constrains its schema's encoded type to Record<string, unknown>, an object, not an array. Our dataset is an array, so it must wear a struct to be admitted: Schema.Struct({ tasks: ... }), and we reach through .tasks on the way out.

Was the hand-rolled version wasted effort, then?

Well, first, generateObject speaks only JSON-shaped objects. The moment you want a Python snippet, a bulleted list, or CSV out of a model, you are back to prefill and stop sequences, and now you own that trick outright.

Second, you now know what lives underneath an abstraction like generateObject: steering on one end, cutting on the other, decoding in between. When it misbehaves someday, you will not be staring at a black box.

Now we have a prompt to evaluate. A schema that decodes what arrives and encodes what departs. A dataset.json full of AWS tasks across Python, JSON, and regex. Tasks we did not have to invent ourselves, ready to be loaded by the evaluation that grades them.

And that's what we will cover next.

Repo

Code exercises set up here