Segment: cutting the picture into named parts
✅ verified (lines as walls 🧪) · needs ComfyUI · step 1 of both routes · code: appendix B · the parts-table fields: appendix G
What this step is for
A flat illustration is one block of pixels, but what an artist sees is hair, a face, a dress, shoes. To turn it into a layered painting, the first thing to do is give the pixels to parts: which part each pixel belongs to, and what that part is called. Like a jigsaw, you separate, turn over and recognise the pieces before doing anything else; every later step – deciding the lines, completing the shapes, laying the flats, painting – assumes this division. Cut wrong here and everything after it is wrong: lines land on the wrong layer, flat colours are sampled from the wrong place, and completion invents one part onto another.
The difficulty is that a part’s boundary is not in the colour. One head of hair graduating from blonde to pink differs in colour by more than hair does from skin; two pieces of skin (a neck and an arm) share a colour and are two different parts. So this step cannot classify by colour: it needs boundaries, words, and a little human input.
After this you will be able to
- say what each of subject, region, sub-part, class and group means and which sits above which.
- write the parts table for a new picture: how to fill in the region numbers and how to pick the words.
- judge from the review image whether the segmentation is right, and know which field to change when it is not.
Concepts
Five tiers
What segmentation produces has tiers. The subject is the pixels where there is something (as opposed to background). The subject is cut into regions, connected blocks enclosed by boundaries, which have numbers but no names. A person merges regions into sub-parts and names them: back hair, fringe, face, neck, torso, arms, skirt, hem. Sub-parts are merged by “words a model can recognise” into classes (hair, face, arms and neck, dress), which is what the segmentation model votes with. Sub-parts are merged by “the layers an artist wants” into groups, and one group is one layer in the finished painting.
The sub-part is segmentation’s unit; classes are for the vote and groups are for the layers; a sub-part belongs to exactly one group.
The boundary barrier
The wall that encloses a region is the barrier. In this handbook the barrier is a drop in luminance: a pixel much darker than the brightest pixel near it is on a wall. An illustration is drawn that way in the first place – lines as walls, the interiors bucket-filled – and even in coloured lineart the line is darker than what is either side of it. A gradient is not a wall: the colour changes slowly, so no pixel nearby is much brighter.
Left: gradient hair, a flat dress, one dark line between them. Middle: the barrier, which is only the dark line and the outline. Right: the regions the barrier encloses, with the gradient not cut apart.
Lines as walls
A drop in luminance cannot see a very pale line. Once there is lineart (ch. 2), the arbitrated lines can be drawn as two-pixel walls and added to the barrier, so the regions split along them. A lock of hair lying across a neck is one region with the neck; make the pale line between them a wall and it becomes two.
Two parts of near-identical colour separated by one pale line: the barrier alone makes them one region, and with the line as a wall they become two.
The hand-written table and the majority vote
Once there are regions, who names them? Two sources. The hand-written table: a person reads the region-number map and writes each number under a sub-part’s name; the one-shot route uses only this. The majority vote: the segmentation model produces a mask per word (“hair”, “face”, “dress”…), and each region goes to whichever word’s mask covers most of it. The hand-written table is accurate but cannot separate two things inside one region; the vote can separate them, but is helpless about anything the words do not catch.
How much of each region a word’s mask covers decides whether the region goes to it; more than half takes the region, and below eighty per cent it is split per pixel.
Growing from a seed
The vote’s unit is a region. When hair wins a region that actually contains a small piece of neck, and the word covers only part of that neck, giving the whole region to hair is wrong. Growing from a seed: if that small piece is clearly a different colour from the winner, take it as a seed and grow outward by colour, stopping where the colour stops matching or where it gets too far from the seed.
Sub-pixel edges
Segmentation gives every pixel a hard label, so the edge is a staircase of pixel cells. But the source’s edges are antialiased: a boundary pixel’s colour is a blend of the colours either side, and the blend ratio tells you where the edge passes through that pixel. Solve for that position and the edge is accurate to a fraction of a pixel, which is what the vector shapes downstream are built from.
Left: an antialiased edge. Middle: hard labels, where the edge is a staircase. Right: the coverage alpha it solves for and the alpha = 0.5 isoline, accurate to a fraction of a pixel.
How it works
The subject. A pixel \(p\) belongs to the subject if and only if its opacity is high enough:
\[ S = \{\,p : \alpha(p) \ge \alpha_{\min}\,\} \]
\(\alpha_{\min}\) defaults to 200 (out of 0-255). Why opacity rather than luminance: many illustrations are PNGs with unpremultiplied alpha, where the transparent area still carries RGB, so judging by luminance takes that colour for subject. The other way round, a source’s alpha is often full of holes – a gap a few pixels wide between two legs has single-digit opacity while its RGB is skin. Treating such a pixel as background gives double lines, the black ground showing through, and completion seeping inward; so a hole whose RGB resembles its neighbours (a difference under 30) is put back into the subject.
The boundary barrier. Write \(L(p)\) for the luminance and \(N_w(p)\) for the window of side \(w\) centred on \(p\):
\[ B(p) = \Big[\ \max_{q \in N_w(p)} L(q) - L(p) > \tau\ \Big] \]
A pixel with \(B(p)=1\) is on a wall. The regions are the connected components of \(S \wedge \neg B\); a fragment of fewer than \(n_{\min}\) pixels is not a region and, along with the pixels on the walls, joins its nearest region by distance transform. Why “local maximum minus self” rather than a gradient: a dark line is bright on both sides, so the pixels on the line have a large drop, both sides become wall, the line is enclosed in the middle, and no region leaks across it; whereas a smooth gradient’s drop across the window is only its slope times half the window, far below \(\tau\), and is not a wall. The defaults are \(w = 7\), \(\tau = 25\) and \(n_{\min} = 300\). How to set \(\tau\): too small and the regions shatter into a thousand pieces, too large and the walls are overrun and the biggest region swallows half the picture; look at the number of regions and the area of the largest, and take something in between.
Lines as walls. Rasterize the lines that survived ch. 2’s arbitration into a 2 px set \(R\), and the barrier becomes \(B^{\prime} = B \vee R\). The new regions are finer than the old ones and inherit their labels: each new region takes the mode of the old labels it covers. The small piece that was cut off (the neck) still inherits hair, and correcting that is left to the next step’s vote – voting on its own, the word now reaches it.
The majority vote. For a region \(r\) and a class \(c\), the fraction of it the class’s mask \(M_c\) covers is
\[ v(r, c) = \frac{|r \cap M_c|}{|r|} \]
Classes are ordered by priority, small and specific first, and the first class with \(v \ge \theta\) wins, with \(\theta = 0.5\). A region whose winner has \(v < s\) is not given away whole but split per pixel: what is inside the mask goes to the winner and the rest keeps its old label, with \(s = 0.8\). A word whose mask covers more than 40% of the subject has degenerated into “the whole person” and is discarded. A sub-part the hand-written table marks as not voting keeps its whole region.
Growing from a seed. Where the winner is class \(A\) and the pixels of class \(B\)’s mask enclose a patch \(K\) inside that region with \(|K| \ge n_{\text{seed}}\) and \(\|\mu_K - \mu_A\| > d_{\text{sep}}\) (\(\mu\) being the median colour), that patch is a seed. A pixel \(p\) in the region goes to \(B\) if and only if
\[ \begin{gathered} \|c(p) - \mu_K\| < \tau_c, \\ \|c(p) - \mu_K\| < \|c(p) - \mu_A\|, \\ d(p, K) \le D, \end{gathered} \]
and is connected to the seed. The three conditions are: its colour resembles the seed, it resembles the seed more than the winner, and it is not far from the seed. The defaults are \(n_{\text{seed}} = 300\), \(d_{\text{sep}} = 30\), \(\tau_c = 35\) and \(D = 15\). \(d_{\text{sep}}\) stops a mask that has spilled onto a same-coloured neighbour (the dress’s mask spilling onto skin encloses a patch the same colour as the winner, which is not evidence); \(D\) stops the colour from growing along to another same-coloured part dozens of pixels away.
Sub-pixel edges. A boundary pixel’s colour is a blend of the two local colours: \(c = \alpha\, c_A + (1 - \alpha)\, c_B\), where \(\alpha\) is how much of that pixel belongs to \(A\). Projecting it onto the line between the two colours solves for
\[ \alpha = \frac{\langle c - c_B,\ c_A - c_B \rangle}{\|c_A - c_B\|^2}, \]
clamped to \([0, 1]\). The \(\alpha = 0.5\) isoline is the edge, positioned to a fraction of a pixel. The isoline is traced into a polyline through a signed distance field and then simplified with RDP (tolerance \(\varepsilon = 0.3\) px). Each sub-part’s ring is expanded outward by 0.6 px: neighbouring parts have to overlap a little, or the black ground shows through at the seam (ch. 5 covers seams).
The method
- Find the subject. Take the subject by opacity and fill the alpha holes back in. The one-shot route uses the subject with its holes left alone, so that it stays line for line identical to the earliest version.
- Cut the regions. The drop in luminance is the barrier, which encloses the regions, and fragments join their nearest region. Once there is lineart, the lines can be used as walls to cut again.
- Name the regions. The first run stops after writing the region-number map; a person reads it and writes the hand-written table: which numbers make up which sub-part, ordered bottom to top. Then run it again.
- Let the model correct it. One mask per word, words merged into classes, a vote per region, and where the winner is not convincing enough the region is split per pixel.
- What the vote cannot fix, grow from a seed by colour.
- Each sub-part’s edge. Solve the coverage, take the isoline, simplify it, expand it. What comes out is one ring per sub-part plus a review image.
Doing it
mkdir -p work/pic/phase1-layers/step01-segment && cp dtp/templates/segment/main.py work/pic/phase1-layers/step01-segment/
uv run python work/pic/phase1-layers/step01-segment/main.py # the first run stops after writing regions.png
uv run python work/pic/phase1-layers/step01-segment/main.py # fill in ASSIGN in parts.py and run again: 40 seconds plus 2 seconds per word
The second pass with lines as walls is another step: copy the template again, and at the top point its two upstream constants at the lineart step, whose lines become the walls, and at the first segmentation, whose table it inherits.
Look at the three-column review image: the hand-written table on the left, the state after the vote in the middle, and the pixels that changed on the right; and look at at least four 1:1 close-ups – the face, the neck, a hand and a foot. Three things to judge by:
- every piece has the right name, with no whole piece in the wrong place.
- the edges follow the antialiasing, biting into no neighbour and leaving no gap around anything.
- every difference between the middle and left columns is a correction rather than damage (you can say why each change in the right column happened).
The verdict goes at the end of this step’s README.
Common problems
| symptom | cause | what to change |
|---|---|---|
| a whole piece has the wrong name and no word catches it | in the hand-written table it shares a region with its neighbour, and the vote works region by region | once there is lineart, cut again with the lines as walls; or find a more specific word so that growing from a seed has evidence to work with |
| a mask spills onto a same-coloured neighbour | the dress’s mask covers skin | the seed’s colour-difference threshold stops it; check that word’s mask area |
| a seed grows too far, onto another part | the colours are close and the pixels are connected | reduce the growth distance |
| double lines between parts, or the black ground showing through | holes in the source’s alpha were taken for background | do not loosen the subject threshold; fill the holes back in by RGB instead |
| a word’s mask is the whole person | the word is too broad (skin, upper body) | discard it and use a specific word (red bow rather than ribbon) |
| the model’s mask is a size larger than the part | the segmentation model’s edges are not accurate | use it for position and not for shape (the eye’s shape comes from colour in ch. 6) |
| a part has more pixels after the lines become walls | that area was always its own; before, it was broken up and misassigned | look at the close-ups, not the totals |
Knobs
| Knobs | default | when to change it |
|---|---|---|
alpha_min | 200 | almost never; loosening it to 80 pulls semi-transparent edges into the subject |
split_below | 0.8 | lower it when two things each take half a region and the winner takes the whole of it |
grow_tol grow_dist | 35 / 15 | raise the first when a seed will not grow, lower the second when it grows too far |
| the lineart step whose lines become walls | none | set it to redo the cut once there is lineart |
| the segment step whose table is inherited | none | set it alongside the walls, so the hand-written table need not be filled in again |
The constant names are at the top of this stage’s main.py and are listed in appendix A.
Summary
- Segmentation goes by boundaries, not colour: a drop in luminance is a wall, a gradient is not, and lineart can be a wall.
- The machine cuts the regions and a person names them; the model’s words correct that division and never invent a part.
- Two things inside one region cannot be separated by the vote; that takes lines as walls or growing from a seed.
- Edges have to be solved to a fraction of a pixel, because every vector shape downstream rests on them.
- Look at the review image and at the four close-ups; going by counts alone gets it backwards.
Further reading
- Code: appendix B, under
segment– the barrier and the regions, the hand-written table, the majority vote and growing from a seed, lines as walls, and the sub-pixel rings. - Worked example: ch. 19, “Segment” (the number of regions, how much each sub-part changed, and two close-ups of the lines as walls).
- References: Segment Anything (SAM), https://segment-anything.com/; Grounding DINO (finding boxes by word), https://github.com/IDEA-Research/GroundingDINO; Ramer-Douglas-Peucker polyline simplification, https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm; isoline tracing (marching squares), https://en.wikipedia.org/wiki/Marching_squares.