Josh Pitzalis

Temperature with Effect

Suppose we're building a brainstorming machine. We ask Claude for a one-sentence movie idea, and we get: "A time-travelling archaeologist who must prevent ancient artefacts from being stolen."

Not bad. Let us brainstorm harder. We run it again, and get the time-travelling archaeologist. Again. The same archaeologist, the same artefacts, run after run.

To answer what's going on, we must first learn how Claude chooses its words at all.

How Claude Generates Text

When we send Claude a prompt like "What do you think?" and three things happen.

  1. First, tokenisation: the input is broken into smaller chunks.
  2. Second, prediction: Claude calculates a probability for every token that might come next.
  3. Third, sampling: it picks one.

Claude might assign the next word "about" a 30% probability, "would" 20%, "of" 10%, and so on down a long tail of candidates. It selects one token, appends it, and repeats the entire process. Predict, sample, append, until a full response exists.

That sampling step is where the dial sits.

What Temperature Does

Temperature is a decimal between 0 and 1 that reshapes those selection probabilities before sampling. Think of it as the Variety dial on Claude's responses.

At low temperatures, near 0, Claude becomes very deterministic. It almost always picks the highest-probability token. At temperature 0.0, "about" gets 100% of the probability. Every fork in the road, same turn.

At high temperatures, near 1, the probability spreads more evenly across the candidates. Lower-ranked tokens get a real chance of being chosen, and the output becomes more varied and creative.

We were brainstorming at temperature 0.0, so every run had the same maximum-probability path straight to the archaeologist.

Choosing the Right Temperature

Low temperatures, 0.0 to 0.3, suit tasks where you want the same answer twice: factual responses, coding assistance, data extraction, content moderation.

Medium temperatures, 0.4 to 0.7, suit most general work: summarisation, educational content, problem-solving, and creative writing that must stay within constraints.

High temperatures, 0.8 to 1.0, are for when variety is the point: brainstorming, creative writing, marketing content, joke generation.

The Pitching Machine, in Effect

Our brainstorming machine is one small function.

import { LanguageModel } from "@effect/ai";
import { Effect } from "effect";

const pitchMovie = Effect.fn("pitchMovie")(function* () {
  const response = yield* LanguageModel.generateText({
    prompt: "Generate a one-sentence movie idea.",
  });
  return response.text;
});

Notice that pitchMovie has no client, no API key, no model name, and no temperature. LanguageModel.generateText summons the model from the surrounding context, and the type checker records that our effect requires a LanguageModel it hasn't been given yet.

Setting the Dial in the Layer

We settle that requirement with layers, and the model layer's config object is where temperature lives. It accepts the same parameters as the Claude API: max_tokens, temperature, and friends.

import { AnthropicClient, AnthropicLanguageModel } from "@effect/ai-anthropic";
import { FetchHttpClient } from "@effect/platform";
import { Config, Layer } from "effect";

const SonnetLayer = AnthropicLanguageModel.model("claude-sonnet-5", {
  max_tokens: 1000,
  temperature: 1.0,
});

const AnthropicLayer = AnthropicClient.layerConfig({
  apiKey: Config.redacted("ANTHROPIC_API_KEY"),
}).pipe(Layer.provide(FetchHttpClient.layer));

This sets the dial for the entire program. Every request made anywhere under this layer runs at temperature 1.0.

Which raises a question. Our program wants to pitch movies at 0.0 and 1.0β€”in the same run. Do we need to build two model layers and shuffle our program between them?

Overriding the Dial Per Call

This is what withConfigOverride is for.

const program = Effect.gen(function* () {
  // Low temperature β€” more predictable
  const idea = yield* pitchMovie().pipe(
    AnthropicLanguageModel.withConfigOverride({ temperature: 0.0 }),
  );

  // High temperature β€” more creative
  const creative = yield* pitchMovie().pipe(
    AnthropicLanguageModel.withConfigOverride({ temperature: 1.0 }),
  );
});

withConfigOverride takes an effect and a config object, and merges that config over the layer's settings for that effect alone. Requests inside the pipe run at the overridden temperature; requests outside it never learn the override existed.

We did not add a temperature parameter to pitchMovie, nor thread it through every function between our program and the API call. The function stays ignorant of temperature forever, and the dial is turned from outside.

Run it, and the extremes behave as advertised. At 0.0, the archaeologist returns, faithfully, run after run. At 1.0, we get new themes, new characters, new plots each time.

Temperature does not guarantee different outputs

Temperature only changes the probability of getting a different output. Even at high temperatures, Claude may occasionally produce similar responses. The dial only loads the dice.

So the game here is matching the dial to the task. Consistent, factual answers want low temperature. Creative brainstorming wants it high. For everything in between, the middle of the dial serves well.