Anti-aliasing in pixel art

Why you need this

A clean pixel line still has steps. At 1x on a small sprite those steps are the style, but on a longer curve, a large sprite, or a shallow diagonal they read as a saw edge, and the player sees a jagged sword rather than a straight one. Anti-aliasing softens the steps by hand, one pixel at a time, using colors the palette already has. It is a stage 2 technique, "draw the still", applied after the line itself is clean, and it has one rule that an engine programmer has to know as well as the artist: the softened edge must never face a transparent background.

The idea

Aliasing is the stair-step appearance of a diagonal or curved edge drawn on a grid. Each pixel is fully one color or fully another, so an edge that should cross a pixel halfway instead jumps a whole pixel at once. A 2:1 line is a sequence of these jumps; a circle is a sequence of them with changing run lengths (see Pixel art fundamentals).

Anti-aliasing (AA) is placing a pixel of an intermediate color where the edge would cross a pixel partway, so that the pixel is "half" one color and "half" the other. In photographs and 3D rendering this is done automatically by averaging. In pixel art it is done by hand, and the intermediate color comes from the ramp: the step between the line's color and the color next to it (see Palettes and ramps). No new colors are invented.

The placement rule: at each stair step, put one AA pixel at the corner where the step turns, so that the run above and the run below appear to blend into each other. One pixel per step. If a step is long, the AA pixel goes at the end of the run, not spread along it.

Worked example

. is a solid background color, 1 is the line color, a is the ramp step between them.

A 2:1 diagonal, before and after. Each run gets an AA pixel at its outer end, which is where the next run begins one row down:

1 1 . . . . . .      1 1 a . . . . .
. . 1 1 . . . .      . a 1 1 a . . .
. . . . 1 1 . .      . . . a 1 1 a .
. . . . . . 1 1      . . . . . a 1 1

Left: four hard steps. Right: at 1x the line reads as a single straight stroke slightly thicker than one pixel; the eye stops seeing the corners.

A 9-pixel circle, filled, with an outline. 1 is the outline, 2 is the fill, and a is the ramp step between outline and fill. The AA pixels sit on the inside of the outline, at the inner corner of each step:

. . . 1 1 1 . . .      . . . 1 1 1 . . .
. . 1 2 2 2 1 . .      . . 1 a 2 a 1 . .
. 1 2 2 2 2 2 1 .      . 1 a 2 2 2 a 1 .
1 2 2 2 2 2 2 2 1      1 a 2 2 2 2 2 a 1
1 2 2 2 2 2 2 2 1      1 2 2 2 2 2 2 2 1
1 2 2 2 2 2 2 2 1      1 a 2 2 2 2 2 a 1
. 1 2 2 2 2 2 1 .      . 1 a 2 2 2 a 1 .
. . 1 2 2 2 1 . .      . . 1 a 2 a 1 . .
. . . 1 1 1 . . .      . . . 1 1 1 . . .

Left: the outline's eight steps show as eight corners. Right: each corner has an a on the fill side, and the outline looks rounder. Nothing changed on the outside of the outline; the silhouette is pixel for pixel the same, so the sprite is exactly as readable against any background as before.

Inside, never outside

The circle above puts every AA pixel inside the shape, against the fill. That is not a stylistic choice. A sprite's outer edge borders whatever the engine draws it over, which the artist does not know: grass one moment, a dark cave wall the next. An AA pixel is a blend of the outline and the background it was drawn against. Blend the outline toward the grass color and the sprite will show a fringe of pale pixels when it stands in the cave. This fringe is the halo:

. . a 1 1 1 a . .      d d a 1 1 1 a d d
. a 1 2 2 2 1 a .      d a 1 2 2 2 1 a d
a 1 2 2 2 2 2 1 a      a 1 2 2 2 2 2 1 a

Left: the sprite with outer AA, a blended toward the light background it was drawn on. Right: the same pixels drawn over a dark background d. The a pixels are lighter than both the outline and the ground, so they stand out as a glowing rim. The fix is to have none: any sprite that will be drawn over an unknown background keeps a hard outer edge and does all its AA inside the outline. Backgrounds and tiles that always sit against the same neighbor can AA their outer edges, because for them the neighbor is known.

Between two regions inside a sprite

AA also softens the boundary between two materials or two tones inside the sprite. The intermediate color should already be in the palette: a red cloak r meeting a blue tunic b gets a purple p at the step corners, and the purple is the same one the palette already uses as a shared shadow.

r r r r . . . .      r r r r . . . .
r r r r r b b b      r r r r p b b b
r r r b b b b b      r r r p b b b b
r b b b b b b b      r p b b b b b b

r cloak, b tunic, p the palette's purple, . outside the region shown. Left: the cloak's hem steps down in three jumps. Right: one p at each step. If no palette color sits between the two regions, leave the edge hard rather than adding a color for one seam.

How much

One AA pixel per step is almost always right. Two in a row, or a run of them along a shallow line, turns the edge into a blurred band; the line looks out of focus rather than smooth. On a 1:1 diagonal there is no room for AA at all, because every pixel is already a step, and adding intermediates just thickens the line. On a 3:1 or shallower line, one AA pixel at each run end is enough; the middle of a long run stays hard.

When to skip it

Very small sprites: at 8 or 12 pixels an AA pixel is a large fraction of the sprite and reads as a smudge. Hard graphic styles: a game whose look is two-tone or flat-shaded gains nothing from softened edges and loses the crispness that defines it. Animated edges: an AA pixel that is in the right place on one frame is in the wrong place on the next, so a fast-moving limb often does better hard-edged than inconsistently softened.

Automatic AA from tools

Every paint program offers anti-aliased brushes and smooth scaling, and every one of them is wrong for pixel art. Automatic AA blends by averaging, so it produces new colors that are in no ramp: a 16-color sprite drawn with a smooth brush has 200 colors after one stroke. It blurs everything equally, including edges that should stay hard and the 1:1 diagonals that cannot take AA. It puts AA on the outer edge, against transparency, so it creates the halo on every sprite. Turn off brush smoothing, draw with a 1-pixel hard brush, and scale only by integers with nearest-neighbor filtering (see Sprite sheets and export). If a purchased pack arrives with smoothed edges, Adapting purchased packs covers reducing it back to its palette.

In a game's art pipeline

AA is the last step of stage 2, applied after the silhouette, flat colors, shading, and outline decisions are settled, because every one of those changes moves the edges. It uses the ramps from stage 1 and adds nothing to them. It matters most on the large forms that Shading and lighting and Character design for sprites produce, and on text and frames in UI and text in pixel art, where a hard-edged font at 3x can look coarse and a lightly anti-aliased one reads. Stage 5 has to preserve it: any non-integer scale or bilinear filter in Sprite sheets and export re-blurs every edge, hand-placed AA included, and the halo problem above is exactly what an engine's texture filtering does to a sprite's edge if the filter samples into transparent pixels.

Common mistakes

• AA on the outer edge of a sprite. Symptom: a pale or dark halo around the character whenever it stands on a background different from the editor's.
• Smearing. Two or three AA pixels along each step. Symptom: the edge looks blurry and slightly thicker; the sprite looks out of focus at 1x.
• Inventing a color for one edge. Symptom: the palette grows by one for every seam, and a palette swap leaves a wrong-colored line where the invented color did not get remapped.
• AA on a 1:1 diagonal. Symptom: the line doubles in thickness and looks soft instead of smoother.
• AA on an 8-pixel sprite. Symptom: the sprite's outline looks smudged and the AA pixels read as dirt.
• Smoothing brush left on. Symptom: a color count in the hundreds and every edge, including the ones meant to be hard, is soft.

Cost

AA roughly doubles the pixel-placing time on curves and shallow diagonals, because every step now gets a second decision: where the intermediate goes and which ramp color it is. On straight verticals and horizontals it costs nothing, since there are no steps. It adds 1 to 2 colors per ramp if the ramp did not already have a step between its outline and fill, which at 4 ramps is 4 to 8 palette entries, so it is usually planned for in stage 1 rather than added later. File size is unaffected. It starts to hurt in animation: an 8-frame walk cycle for a 32-pixel character with anti-aliased edges is 8 sets of edge pixels to keep consistent, so most games anti-alias the hero and leave the enemies hard-edged.

Going further

• Dithering, the other technique that places intermediate colors, over areas rather than edges.
• UI and text in pixel art, for anti-aliasing letterforms at small sizes.
• Sprite sheets and export, for the engine settings that keep hand-placed AA from being re-blurred.
• Shading and lighting, for the outline choices that decide where AA can go.