Skip to content
← Projects

Style Transfer

Image style transfer with StyleFlow: an invertible Glow network keeps the content intact, and the style is injected through VGG-19 statistics on the reverse pass. One forward pass, running on CPU.

The problem

Produce a new image that keeps the layout and content of a photo but carries the brushwork, colours and texture of a painting. The classic approach (Gatys) optimises every pixel over hundreds of iterations per image. This project is feed-forward: train once, then each image takes a single pass.

How it works

The model is StyleFlow, a Glow-style normalizing flow. The network is fully invertible, so the content is preserved instead of being redrawn as in encoder–decoder models.

  1. Encode the content. The photo goes forward through Glow (2 blocks, each a 2×2 squeeze followed by 15 flow steps: ActNorm → invertible 1×1 convolution in LU form → additive coupling) into a latent z. Every step inverts exactly, so z holds all of the image.
  2. Get the style code. A frozen normalised VGG-19 reads the painting. Per-channel mean and std at relu1_1…relu4_1 are concatenated into a 1920-d vector.
  3. Decode with style. Glow runs in reverse from z. At the head of each block, SAN reads the current features together with the style code to predict target mean/std, and AdaIN_SET swaps the features' statistics for them.

content ─▶ Glow.forward ─▶ z style ─▶ VGG-19 ─▶ cat(mean, std) = 1920-d code z, code ─▶ Glow.reverse (SAN + AdaIN_SET per block) ─▶ stylized

class StyleFlow(nn.Module):
    def forward(self, content_images, domain_class):
        z_c = self.glow(content_images, forward=True)                  # encode content
        return self.glow(z_c, forward=False, style=domain_class)      # decode with style

Training

Glow with n_flow = 15, n_block = 2, trained with Adam (lr = 1e-4) on three losses:

  • Content: MSE between VGG features (relu4_1) of the result and of the photo.
  • Style: matching mean/std of VGG features over relu1_1…relu4_1, weighted by keep_ratio to favour the strongest style channels.
  • Smoothness: Total Variation against grainy noise.

Inference uses the checkpoint saved at step 187,500 (187500.ckpt).

Post-processing

The raw output has a 2-px stripe grid (from the 2×2 squeeze) and colour bands at the edges (from ZeroConv2d padding with the constant 1). The original app with the same weights shows the same artefacts, so they belong to the architecture, not to the port. The fixes:

  • reflect padding of 32 px before the network, cropped afterwards: no more edge bands;
  • a guided filter steered by half photo, half result: the grid disappears while real edges and brush strokes stay;
  • refine: the network runs at 512 px on the long side, and the 1024-px output gets the photo's own fine luminance detail back (luminance only, the colours stay the painting's).

The strength slider blends the painting's style code with the photo's own code rather than blending pixels, so the result stays a single coherent rendering.

Deployment

The first version was a standalone Streamlit app. The model is now the app/style_transfer/ module of this site's FastAPI backend, with a Next.js front end:

  • class and attribute names ported verbatim so the checkpoint loads with strict=True;
  • images keep their aspect ratio instead of being squashed to 512×512 as in the original;
  • CPU-only torch, loaded lazily on the first request, with the style code of each preset cached;
  • one image at a time in a threadpool, rate-limited to 12 requests per 10 minutes per IP;
  • uploads are actually decoded to check the format and stripped of EXIF/GPS; results are deleted after 24 hours;
  • weights (~275 MB) are kept out of git, fetched with python -m app.cli style fetch and checked by sha256.

On CPU, one image takes about 22 seconds when run directly on a machine, and longer inside Docker.

Related posts (Vietnamese)

Architecture

content image → Glow forward (2 blocks × 15 flows, invertible) → latent z
style image → VGG-19 → mean/std at relu1_1…relu4_1 → 1920-d style code
z + style code → Glow reverse (SAN + AdaIN_SET at the head of each block) → result
post-processing: reflect padding, guided filter against the grid, the photo's fine luminance detail