Style Transfer
Image style transfer with StyleFlow: an invertible Glow network keeps the content intact, and the style is injected through VGG-19 statistics on the reverse pass. One forward pass, running on CPU.
The problem
Produce a new image that keeps the layout and content of a photo but carries the brushwork, colours and texture of a painting. The classic approach (Gatys) optimises every pixel over hundreds of iterations per image. This project is feed-forward: train once, then each image takes a single pass.
How it works
The model is StyleFlow, a Glow-style normalizing flow. The network is fully invertible, so the content is preserved instead of being redrawn as in encoder–decoder models.
- Encode the content. The photo goes forward through Glow (2 blocks, each a 2×2 squeeze followed by 15 flow steps: ActNorm → invertible 1×1 convolution in LU form → additive coupling) into a latent
z. Every step inverts exactly, sozholds all of the image. - Get the style code. A frozen normalised VGG-19 reads the painting. Per-channel mean and std at
relu1_1…relu4_1are concatenated into a 1920-d vector. - Decode with style. Glow runs in reverse from
z. At the head of each block, SAN reads the current features together with the style code to predict target mean/std, and AdaIN_SET swaps the features' statistics for them.
content ─▶ Glow.forward ─▶ z style ─▶ VGG-19 ─▶ cat(mean, std) = 1920-d code z, code ─▶ Glow.reverse (SAN + AdaIN_SET per block) ─▶ stylized
class StyleFlow(nn.Module):
def forward(self, content_images, domain_class):
z_c = self.glow(content_images, forward=True) # encode content
return self.glow(z_c, forward=False, style=domain_class) # decode with styleTraining
Glow with n_flow = 15, n_block = 2, trained with Adam (lr = 1e-4) on three losses:
- Content: MSE between VGG features (
relu4_1) of the result and of the photo. - Style: matching mean/std of VGG features over
relu1_1…relu4_1, weighted bykeep_ratioto favour the strongest style channels. - Smoothness: Total Variation against grainy noise.
Inference uses the checkpoint saved at step 187,500 (187500.ckpt).
Post-processing
The raw output has a 2-px stripe grid (from the 2×2 squeeze) and colour bands at the edges (from ZeroConv2d padding with the constant 1). The original app with the same weights shows the same artefacts, so they belong to the architecture, not to the port. The fixes:
- reflect padding of 32 px before the network, cropped afterwards: no more edge bands;
- a guided filter steered by half photo, half result: the grid disappears while real edges and brush strokes stay;
- refine: the network runs at 512 px on the long side, and the 1024-px output gets the photo's own fine luminance detail back (luminance only, the colours stay the painting's).
The strength slider blends the painting's style code with the photo's own code rather than blending pixels, so the result stays a single coherent rendering.
Deployment
The first version was a standalone Streamlit app. The model is now the app/style_transfer/ module of this site's FastAPI backend, with a Next.js front end:
- class and attribute names ported verbatim so the checkpoint loads with
strict=True; - images keep their aspect ratio instead of being squashed to 512×512 as in the original;
- CPU-only torch, loaded lazily on the first request, with the style code of each preset cached;
- one image at a time in a threadpool, rate-limited to 12 requests per 10 minutes per IP;
- uploads are actually decoded to check the format and stripped of EXIF/GPS; results are deleted after 24 hours;
- weights (~275 MB) are kept out of git, fetched with
python -m app.cli style fetchand checked by sha256.
On CPU, one image takes about 22 seconds when run directly on a machine, and longer inside Docker.
Related posts (Vietnamese)
Architecture
content image → Glow forward (2 blocks × 15 flows, invertible) → latent z
style image → VGG-19 → mean/std at relu1_1…relu4_1 → 1920-d style code
z + style code → Glow reverse (SAN + AdaIN_SET at the head of each block) → result
post-processing: reflect padding, guided filter against the grid, the photo's fine luminance detail