Evolutionary algorithms

Why you need this

GEPA's name starts with "Genetic": it is an evolutionary algorithm whose candidates are prompts. To follow the GEPA pages you need the parts every evolutionary algorithm shares: a population, a way to pick parents, a way to make children, and a rule for who survives. This page builds them on the smallest possible example, a string of eight bits, so every step fits in a table.

The idea

An evolutionary algorithm is a search that keeps many candidates at once instead of one, and improves them in rounds called generations.

Term	Meaning
Genome	One candidate, written as data. Here, a string of 8 bits.
Population	The set of candidates alive in the current generation.
Fitness	The objective: a number saying how good a genome is. Higher is better.
Selection	The rule for choosing parents. Fitter genomes should be chosen more often, but not always.
Mutation	A small random change to one genome, such as flipping a bit.
Crossover	Building a child from two parents, taking some parts from each. GEPA calls its version merge.
Elitism	Copying the best genome into the next generation unchanged, so the best score never goes down.

Two common selection rules, with the four fitness values 3, 1, 4, 6:

• Proportional selection: pick each genome with probability equal to its fitness divided by the total. The total is 14, so the chances are $3/14 \approx 0.214$, $1/14 \approx 0.071$, $4/14 \approx 0.286$, and $6/14 \approx 0.429$.
• Tournament selection (size 2): draw two genomes at random (the same one can be drawn twice) and keep the fitter. The best genome wins unless neither draw is it, so it is picked with probability $1 - (3/4)^2 = 7/16 \approx 0.44$. The worst wins only if both draws are it: $(1/4)^2 = 1/16$.

Both favor fitter genomes while still letting weaker ones reproduce sometimes. That matters because of diversity: if every member of the population becomes a copy of the same genome, crossover has nothing new to combine and the search can only wait for a lucky mutation. Losing diversity is the most common way an evolutionary search stalls.

Worked example

The target is the bit string 10110110. Fitness is the number of positions where a genome matches the target, from 0 to 8. Each generation:

1. Keep the best genome (elitism).
2. Make three children. For each: pick two parents by tournament of 2, cut both at a random point from 1 to 7, join the mother's bits before the cut to the father's bits after it, then flip each bit with chance $1/8$.
3. The next population is the kept best plus the three children.

This Python program runs three generations. It uses mulberry32, a small seeded random generator, so it prints the same thing every time. It ran on Python 3.14 (3.12 and newer behave the same); a JavaScript version with the same generator printed an identical table.

def mulberry32(seed):
    """Small seeded random generator; returns numbers in [0, 1)."""
    state = seed & 0xFFFFFFFF
    def rand():
        nonlocal state
        state = (state + 0x6D2B79F5) & 0xFFFFFFFF
        t = state
        t = ((t ^ (t >> 15)) * (t | 1)) & 0xFFFFFFFF
        t = ((t + (((t ^ (t >> 7)) * (t | 61)) & 0xFFFFFFFF)) & 0xFFFFFFFF) ^ t
        return ((t ^ (t >> 14)) & 0xFFFFFFFF) / 4294967296
    return rand

TARGET = "10110110"
rand = mulberry32(2)

def fitness(g):
    return sum(a == b for a, b in zip(g, TARGET))

def pick(pop):
    """Tournament of two: draw two members at random, keep the fitter."""
    a = pop[int(rand() * len(pop))]
    b = pop[int(rand() * len(pop))]
    return a if fitness(a) >= fitness(b) else b

def child(pop):
    mom, dad = pick(pop), pick(pop)
    cut = 1 + int(rand() * 7)                 # crossover point 1..7
    bits = list(mom[:cut] + dad[cut:])
    for i in range(8):                        # mutation: flip each bit with chance 1/8
        if rand() < 1 / 8:
            bits[i] = "1" if bits[i] == "0" else "0"
    return mom, dad, cut, "".join(bits)

pop = ["".join("1" if rand() < 0.5 else "0" for _ in range(8)) for _ in range(4)]
for gen in range(3):
    print(f"gen {gen}: " + "  ".join(f"{g}({fitness(g)})" for g in pop))
    best = max(pop, key=fitness)
    kids = [child(pop) for _ in range(3)]
    for mom, dad, cut, kid in kids:
        print(f"    {mom} x {dad} cut {cut} -> {kid}({fitness(kid)})")
    pop = [best] + [k[3] for k in kids]       # elitism: the best survives unchanged
print("gen 3: " + "  ".join(f"{g}({fitness(g)})" for g in pop))

Run it with python evolve.py. It printed (fitness in parentheses):

gen 0: 01100011(3)  01011001(1)  01010111(4)  10100111(6)
    01010111 x 01100011 cut 7 -> 00000111(4)
    10100111 x 10100111 cut 4 -> 11100110(6)
    10100111 x 01010111 cut 7 -> 10110111(7)
gen 1: 10100111(6)  00000111(4)  11100110(6)  10110111(7)
    10110111 x 10100111 cut 5 -> 10110110(8)
    10110111 x 10110111 cut 3 -> 10110101(6)
    10100111 x 10110111 cut 3 -> 10010110(7)
gen 2: 10110111(7)  10110110(8)  10110101(6)  10010110(7)
    10010110 x 10110111 cut 4 -> 10011111(5)
    10110110 x 10110101 cut 3 -> 10110101(6)
    10110111 x 10110111 cut 3 -> 10110111(7)
gen 3: 10110110(8)  10011111(5)  10110101(6)  10110111(7)

Reading the table step by step:

Generation	Best fitness	What happened
0	6	Four random strings. 10100111 is closest to the target.
0 to 1	7	The third child: crossover at cut 7 gives 1010011 + 1 = 10100111, then one mutation flips bit 4 (counting from 1) from 0 to 1, giving 10110111, one bit from the target.
1 to 2	8	The first child: 10110 from one parent plus 111 from the other gives 10110111; a mutation flips the last bit, giving 10110110, the target.
2 to 3	8	Elitism keeps the perfect string even though none of the new children matches it.

Notice the second child of generation 0: both tournaments picked the same parent, 10100111, so crossover changed nothing, and two mutations moved it sideways to another 6. Self-crossover is what low diversity looks like. Run the same program with seed 10 instead of 2 and, by generation 4, all four members are 10111110 (fitness 7): the population has collapsed to one genome, and only mutation can still move it.

In an optimization pipeline

GEPA keeps the same skeleton and replaces the parts that do not make sense for text.

Bit-string part	GEPA's version
Genome	A candidate: the instruction text of every predictor in the program.
Fitness	Per-example scores on a validation set, not a single number.
Selection	Sample a parent from the candidates that are best on at least one validation example, weighted by how many they win (see Pareto fronts).
Mutation	A strong language model reads the parent's instruction, a few inputs and outputs, and the metric's written feedback, then writes a new instruction.
Crossover	Merge: for a program with several predictors, take each predictor's instruction from whichever of two related candidates improved it.
Survival	A child joins the pool only if it beats its parent on the same small batch of examples.

The big change is mutation. Flipping a random bit is blind: most flips make things worse, and progress comes from trying many. A language model that reads "Expected 3, got 3.33: round down to whole boxes" makes a directed change aimed at the failure it just saw. That is why GEPA can improve a prompt in far fewer evaluations than a blind search would need, and why the feedback text your metric writes matters so much. GEPA: reflective prompt evolution walks through one full iteration.

Common mistakes

• No elitism. Symptom: the best fitness goes up and then back down between generations.
• Selection too greedy. Symptom: within a few generations every member is a copy of one genome, and progress stops.
• Mutation too strong. Symptom: children are essentially random and fitness never climbs; with prompts, the rewrite discards rules that were working.
• Mutation too weak. Symptom: children are near copies of their parents and the search crawls.
• Fitness that can be gamed. Symptom: the population converges on something that scores well and is useless, because evolution optimizes exactly what you measure.

Cost

Each generation evaluates every new child once. With population size $P$ (children per generation) and $G$ generations, the algorithm makes about $P \times G$ fitness evaluations, plus the first population. Memory is $P$ genomes. For the bit string, an evaluation is a comparison of 8 bits, so the cost is nothing. For prompts, a fitness evaluation is a run of the program on a set of examples, and a mutation is a call to a strong model, so both $P$ and $G$ are kept small and the algorithm is designed to evaluate a child on a few examples before spending a full evaluation on it.

Going further

• Pareto fronts: how GEPA chooses parents when fitness is a list of per-example scores.
• Genetic programming, where the genome is a program tree instead of a bit string.
• The island model, which runs several populations apart and occasionally swaps members to keep diversity.
• Novelty search, which rewards being different instead of being fit.