StyleFlow: image style transfer with an invertible flow network
This is the first of three posts on the Style Transfer project. This one covers the network architecture. The next covers the image artefacts the architecture causes and how to fix them; the last covers putting the model on the web.
The problem
Given a content image (a photo) and a style image (a painting), produce a new image that keeps the photo's layout but carries the painting's colours, brushwork and texture.
Gatys' classic method optimises the pixels directly over hundreds of iterations per image. The results are beautiful, but every image is its own optimisation problem. Feed-forward networks such as AdaIN are much faster, needing a single pass, but they use an encoder–decoder: the image is compressed into features and then redrawn from scratch, so content detail is easily lost or tangled with the painting's strokes.
StyleFlow takes a different route: an invertible network. The image goes forward through the network into a latent code, then backward into an image. Because the network inverts exactly, with no intervention the output is identical to the input. Style is simply a deliberate change made during the backward pass.
Glow: invertible building blocks
The backbone is a Glow with 2 blocks of 15 flow steps each (n_block = 2, n_flow = 15).
Each block does two things:
- 2×2 squeeze: each 2×2 patch of pixels becomes 4 channels. A 3-channel RGB image becomes 12 channels at half resolution; the second block takes it to 48 channels at a quarter resolution. It is a rearrangement and loses nothing.
- 15 flow steps, each made of:
- ActNorm: per-channel learnt affine normalisation, inverted by dividing and subtracting.
- Invertible 1×1 convolution, stored as an LU decomposition, with the weight matrix
P · L · (U + diag(s)). That makes the inverse cheap and stable. - Coupling: split the channels in half, keep the first half, and use a sub-network computed from the first half to transform the second.
The checkpoint I use was trained with additive coupling, not affine:
The reverse is just adding back: . The sub-network can be arbitrarily complex and the transform still inverts exactly, because itself never has to be inverted.
class AffineCoupling(nn.Module):
def forward(self, input):
in_a, in_b = input.chunk(2, 1)
out_b = in_b - self.net(in_a) # additive: affine = False
return torch.cat([in_a, out_b], 1)
def reverse(self, output):
out_a, out_b = output.chunk(2, 1)
in_b = out_b + self.net(out_a)
return torch.cat([out_a, in_b], 1)The style code: 1920 numbers from VGG-19
Style is described by feature statistics, as in AdaIN. A normalised VGG-19 (frozen, never trained) reads the style image. At four layers, relu1_1, relu2_1, relu3_1, relu4_1, I take the per-channel mean and std and concatenate them:
That 1920-d vector is everything the network knows about the painting: dominant colours, contrast, texture at several scales. It carries no spatial information, so the painting's layout does not leak into the result.
Where the style goes in
The forward pass knows nothing about style: the photo just goes through both blocks into the latent z.
The reverse pass walks the blocks in the opposite order, and at the head of each block there are two extra layers:
- SAN (style-aware normalisation) takes two inputs: the current features (through a convolutional head down to a 768-d vector) and the 1920-d style code (through linear layers 1920 → 1024 → 512). The two branches are combined to predict a target mean/std for every channel of the block.
- AdaIN_SET normalises the features to zero mean and unit std, then scales and shifts them with the mean/std SAN just predicted.
Then the flow steps run in reverse as usual and the squeeze is undone.
content image ─▶ Block 1 ─▶ Block 2 ─▶ z style image ─▶ VGG-19 ─▶ mean/std at 4 layers ─▶ 1920-d code
z ─▶ [SAN + AdaIN_SET] ─▶ Block 2 reversed ─▶ [SAN + AdaIN_SET] ─▶ Block 1 reversed ─▶ result
Because SAN reads the current features and not just the style code, the injected statistics depend on the image being processed. The same painting is "applied" differently to a portrait and to a landscape.
All of inference fits in two lines, two passes through the same Glow:
class StyleFlow(nn.Module):
def forward(self, content_images, domain_class):
z_c = self.glow(content_images, forward=True)
return self.glow(z_c, forward=False, style=domain_class)Training
The model is trained with Adam (lr = 1e-4) on a three-part loss:
- Content: MSE between the VGG
relu4_1features of the result and of the photo. - Style: matching mean/std of VGG features over
relu1_1…relu4_1, weighted bykeep_ratioto favour the channels that carry the style most clearly. - Smoothness: Total Variation against grainy noise.
The checkpoint used on the site is the one saved at step 187,500 (187500.ckpt).
Trade-offs
- Content is preserved very well, since the forward pass loses nothing. In exchange, the network can only change feature statistics, so styles that need to change shapes (say, turning a face into cubist planes) are out of reach.
- One pass for every style: no training per painting. Adding a style just means computing another 1920-d code.
- Heavy: two blocks × 15 flows at full resolution, with no encoder–decoder bottleneck. On CPU that is the main cost; the third post is about living with it.
- The 2×2 squeeze leaves traces on the output. That is the subject of the next post.