Pareto fronts

Why you need this

An optimizer that keeps only the single best candidate throws away everything else it learned. GEPA instead keeps every candidate that is best at something, even if its average is mediocre, and picks parents from that set. The idea behind this is the Pareto front. This page defines it, shows the version GEPA uses (one objective per validation example), and works through a small table by hand.

The idea

Start with more than one objective. Suppose each candidate program has an accuracy (higher is better) and a number of model calls per answer (lower is better). No single number ranks them, but some comparisons are still clear.

Candidate $a$ dominates candidate $b$ when $a$ is at least as good as $b$ on every objective and strictly better on at least one. A candidate that no other candidate dominates is Pareto optimal. The set of all Pareto-optimal candidates is the Pareto set, and their scores, drawn as points, form the Pareto front.

Program	Accuracy	Calls per answer	Dominated by
P1	90%	4	nobody
P2	85%	1	nobody
P3	80%	2	P2 (more accurate and cheaper)
P4	70%	1	P2 (more accurate, same cost)

The Pareto set is {P1, P2}. Choosing between them is a real trade (5 points of accuracy for three extra calls), but P3 and P4 are never the right choice.

Per-instance Pareto. GEPA uses the same definition with a twist: every validation example is its own objective. Write $s_i(p)$ for the score of program $p$ on example $i$, a number in $[0, 1]$. Then $a$ dominates $b$ when $s_i(a) \ge s_i(b)$ for every example $i$ and $s_i(a) > s_i(b)$ for at least one. With $n$ examples there are $n$ objectives. A candidate stays in the Pareto set if, roughly, no other candidate does at least as well on every single example.

Why bother? Ranking by the average $\bar{s}(p) = \frac{1}{n}\sum_i s_i(p)$ keeps one winner. But a candidate with a low average may be the only one that solved a hard example, because its instruction contains a rule the others lack. Throwing it away throws away that rule. Keeping the Pareto set keeps every candidate that has a lesson worth combining.

Worked example

Five candidates, A to E, scored on four validation examples, e1 to e4. Scores are 0 (wrong), 0.5 (partly right), or 1 (right).

	e1	e2	e3	e4	average
A	1	1	0	0	0.5
B	0	0	1	0.5	0.375
C	1	0.5	0	0	0.375
D	0.5	1	1	0	0.625
E	0	0	0.5	1	0.375

Step 1: the best score on each example. e1: 1, by A and C. e2: 1, by A and D. e3: 1, by B and D. e4: 1, by E alone.

Step 2: check domination pair by pair. C against A: on e1 both have 1, on e2 A has 1 and C has 0.5, on e3 and e4 both have 0. A is never worse and is better on e2, so A dominates C. Every other pair has a split: for example, B beats E on e3 (1 against 0.5) but E beats B on e4 (1 against 0.5), so neither dominates. D beats A on e3, A beats D on e1.

Step 3: the Pareto set is everyone not dominated: {A, B, D, E}.

Step 4: what the best average misses. D has the best average, 0.625. If you kept only D, you would lose the only candidate that gets e4 right (E) and the only one besides C that gets e1 fully right (A).

This Python program checks the table. Both programs on this page ran on Python 3.14; 3.12 and newer behave the same.

scores = {
    "A": [1.0, 1.0, 0.0, 0.0],
    "B": [0.0, 0.0, 1.0, 0.5],
    "C": [1.0, 0.5, 0.0, 0.0],
    "D": [0.5, 1.0, 1.0, 0.0],
    "E": [0.0, 0.0, 0.5, 1.0],
}
names = list(scores)
n = 4

def dominates(p, q):
    """p dominates q: at least as good on every example and strictly better on one."""
    sp, sq = scores[p], scores[q]
    return all(a >= b for a, b in zip(sp, sq)) and any(a > b for a, b in zip(sp, sq))

for q in names:
    by = [p for p in names if p != q and dominates(p, q)]
    print(q, "average", sum(scores[q]) / n, "dominated by", by or "nobody")

pareto = [q for q in names if not any(dominates(p, q) for p in names if p != q)]
print("Pareto set:", pareto)

for i in range(n):
    best = max(scores[p][i] for p in names)
    winners = [p for p in names if scores[p][i] == best]
    print(f"e{i + 1}: best {best} by {winners}")

wins = {p: sum(1 for i in range(n) if scores[p][i] == max(scores[q][i] for q in names)) for p in pareto}
print("examples won (ties count) by Pareto members:", wins)

Run it with python pareto.py. It printed:

A average 0.5 dominated by nobody
B average 0.375 dominated by nobody
C average 0.375 dominated by ['A']
D average 0.625 dominated by nobody
E average 0.375 dominated by nobody
Pareto set: ['A', 'B', 'D', 'E']
e1: best 1.0 by ['A', 'C']
e2: best 1.0 by ['A', 'D']
e3: best 1.0 by ['B', 'D']
e4: best 1.0 by ['E']
examples won (ties count) by Pareto members: {'A': 2, 'B': 1, 'D': 2, 'E': 1}

How GEPA picks a parent from this. GEPA samples a parent with probability proportional to how many examples it is best on, so a candidate that wins more examples gets more children. Its implementation adds one pruning step first, which matters when there are ties. In the installed gepa 0.1.4 source, it takes each example's set of best candidates, then walks through candidates from lowest average to highest and drops any candidate for which every example it is best on also has another surviving best candidate. Here B goes first: its only win, e3, is shared with D. C goes too: its only win, e1, is shared with A. What remains is A (best on e1 and e2), D (e2 and e3), and E (e4), with weights 2, 2, 1.

import random
from collections import Counter

from gepa.gepa_utils import remove_dominated_programs, select_program_candidate_from_pareto_front

# Candidates A..E are program indices 0..4; averages from pareto.py.
avg = [0.5, 0.375, 0.375, 0.625, 0.375]
front = {0: {0, 2}, 1: {0, 3}, 2: {1, 3}, 3: {4}}  # example -> candidates tied for best

pruned = remove_dominated_programs(front, scores=avg)
print("after GEPA pruning:", {f"e{k + 1}": sorted("ABCDE"[p] for p in v) for k, v in pruned.items()})

rng = random.Random(0)
draws = Counter("ABCDE"[select_program_candidate_from_pareto_front(front, avg, rng)] for _ in range(1000))
print("1000 parent draws:", dict(sorted(draws.items())))

Run it with python gepa_prune.py in an environment with DSPy 3.4 installed (which installs gepa). It printed:

after GEPA pruning: {'e1': ['A'], 'e2': ['A', 'D'], 'e3': ['D'], 'e4': ['E']}
1000 parent draws: {'A': 411, 'D': 410, 'E': 179}

The expected shares are $2/5 = 40\%$, $2/5 = 40\%$, and $1/5 = 20\%$, and 1000 draws land close to them. E, with an average of only 0.375, still becomes a parent about one time in five, because it holds the only solution to e4.

The demo below uses the textbook definition from Step 3 (best on at least one example and not dominated) on a larger table: six candidates by eight examples. Look for candidates marked * whose average is far below the one marked "top", then click them to see which examples only they win. "Sample parent" draws a parent weighted by examples won.

In an optimization pipeline

In stage 3, GEPA scores every accepted candidate on the whole valset and records the per-example scores. Before each iteration it rebuilds the per-example bests, samples a parent as above, and tries to improve it. A candidate stops being chosen as a parent once, on every example where it was best, another candidate scores higher or ties it with a better average; it stays in the pool, it just has no more children. The valset therefore plays a second role beyond reporting: its examples are the objectives. A valset that leaves out a kind of input gives GEPA no reason to keep the candidate that handles it.

Common mistakes

• Keeping only the best average. Symptom: the optimizer improves for a few iterations, then keeps proposing small variations of one instruction and stalls.
• Calling a candidate dominated because its average is lower. Domination needs "at least as good on every example". Symptom: candidates that solved unique examples disappear.
• A valset that is too small. With four examples there are only four ways to be best, so the set holds few candidates. Symptom: little variety among parents.
• Reporting a Pareto member as the final program. The Pareto set is for choosing parents. The program you ship is chosen by average on the valset and confirmed on the testset.

Cost

With $C$ candidates and $n$ validation examples, finding each example's best costs $C \times n$ comparisons, and checking every pair for domination costs about $C^2 \times n$. For the sizes GEPA deals with (tens of candidates, tens to hundreds of examples) this is instant next to a single model call. The real cost is upstream: every per-example score in the table is a rollout, so the table itself costs $C \times n$ program runs.

Going further

• GEPA: reflective prompt evolution, where this selection rule feeds the main loop.
• Multi-objective evolutionary algorithms such as NSGA-II, which sort a whole population into successive fronts.
• Quality-diversity search (MAP-Elites), which keeps the best candidate for each kind of behavior rather than each example.