Skip to content
← Blog

A 2-px grid and coloured edges: post-processing for StyleFlow

The second post on the Style Transfer project. The previous one explained the StyleFlow architecture. This one is about what happens when you run it on real photos.

Two problems you see immediately

The raw network output has two obvious issues:

  1. A vertical stripe pattern about 2 pixels wide across the whole image, with tiny colour fringes. Up close it looks like mesh fabric.
  2. Strange colour bands along the left and right edges.

The first question was whether I had ported the code wrong or whether it was the model itself. I ran it exactly like the original app: a square 512×512 image, the same weights. The result had the same grid and the same bands. So they belong to the architecture and the checkpoint, and retraining the same architecture would not remove them.

Causes

The edge bands. The sub-network in every coupling ends in a ZeroConv2d, which pads its input with the constant 1 before convolving:

out = nn.functional.pad(input, [1, 1, 1, 1], value=1)

Pixels near the edge are partly computed from fake data. Over 30 flow steps that error builds up into a colour band.

The 2-px grid. The 2×2 squeeze turns each 2×2 patch into channels, and every transform in the flow acts on those channels. When the reverse pass spreads them back into pixels, the four pixels of a patch come out systematically different, repeating evenly over the image.

Fixing the edges: reflect padding, then crop

Before the network, the image is padded by 32 pixels on each side by mirroring itself. The network sees realistic content at the edge, the "fake data" moves out into the padding, and the padding is cropped off afterwards.

Two small details:

  • The padding must be divisible by 4, because the network squeezes twice, and smaller than the image side, because reflection cannot be longer than what it mirrors.
  • 32 came from trying: 16 pixels still left bands, and 128 looked no better than 32 while costing more time.

Fixing the grid: a guided filter steered by photo and result

A plain blur removes the grid, but also the real edges of objects and the brush strokes. I use a guided filter (He et al., 2013): in each small window, the output is a linear function of a guide image. Where the guide is flat, the output is smoothed and the grid disappears. Where the guide has an edge, the linear model keeps it.

qi=ak Ii+bk,ak=covk(I,p)vark(I)+εq_i = a_k \, I_i + b_k, \qquad a_k = \frac{\mathrm{cov}_k(I, p)}{\mathrm{var}_k(I) + \varepsilon}

The hard part is choosing the guide:

GuideResult
the result alonethe grid is strong enough to count as "edges", so part of it survives
the photo aloneno grid, but the brush strokes added by the style get ironed flat
half photo + half pre-filtered resultkeeps the photo's real edges, keeps the brush strokes, and the photo half has no grid

I use the third, filtering in two passes with radius 2 and eps = 4e-3:

def clean_grid(stylised, photo, radius=2, eps=4e-3):
    pre = guided_filter(stylised, stylised, radius, eps)
    return guided_filter(stylised, 0.5 * photo + 0.5 * pre, radius, eps)

Bigger and sharper: putting back the photo's fine detail

CPU time grows with image area, so the network only runs with a 512-pixel long side. Upscaling a 512 result straight to 1024 turns fur, leaves and hair into blocks.

The refine step relies on one observation: the high-resolution photo already contains the fine detail the network never saw. The method:

  1. Upscale the result to the target size with bicubic interpolation.
  2. Take the luminance of the high-resolution photo and subtract a Gaussian-blurred copy of itself. What is left is the band of detail a 512 image cannot hold. The blur scales with the upscale factor.
  3. Add that detail to the result, multiplied by 0.8.

Only luminance is added, not colour, so the colours stay the painting's while edges and textures stay as sharp as the photo. A factor of 0 falls back to a plain bicubic upscale. The step takes under a second.

The style-strength slider

The site has a strength slider from 0.3 to 1. The obvious approach is to blend the pixels of the photo and the result, but that gives a double exposure.

Instead I blend the style codes. The photo has its own 1920-d code, computed by the same VGG-19. The code fed to the network is:

c=s⋅cstyle+(1−s)⋅cphotoc = s \cdot c_{style} + (1 - s) \cdot c_{photo}

The network still renders one coherent image, just with statistics somewhere between the photo and the painting.

The full order

upload ─▶ strip EXIF, apply rotation ─▶ fit long side to 1024 (the "hi" image) hi ─▶ fit long side to 512, both sides divisible by 4 (the "lo" image) lo ─▶ reflect-pad 32 ─▶ StyleFlow ─▶ crop padding ─▶ clean_grid ─▶ refine with hi ─▶ PNG

None of this needs training. It is all pure tensor functions in postprocess.py, unit-tested, and each step can be switched off for comparison.