Adjustable Real-time Style Transfer

1. Introduction

Style transfer: Synthesizing an image with content similar to a given image and style similar to another.

1.1 Motivation

There are two main problems in style transfer on the existing methods. (i) The first weak point is that they generate only one stylization for a given content/style pair. (ii) One other issue of them is their high sensitivity to the hyper-parameters.

1.2 Goal

To solve these problems, the authors provide a novel mechanism that allows adjustment of crucial hyper-parameters, after the training and in real-time, through a set of manually adjustable parameters.

2. Background

2.1 Style transfer using deep networks

Style transfer can be formulated as generating a stylized image $p$ whose content is similar to a given content image $c$ and its style is close to another given style image $s$.

$$p=\mathrm{\Psi }(c,s)$$

The similarity in style can be vaguely defined as sharing the same spatial statistics in low-level features of a network (use VGG-16 in this paper), while similarity in content is roughly having a close Eculidean distance in high-level features.

The main idea is that the features obtained by the network contain information about the content of the input image while the correlation between these features represents its style.

In order to increase the similarity between two images, minimize the following distances between their extracted features:

$$\begin{array}{l}{L}_c^l(p)=\Vert {\varphi }^l(p)-{\varphi }^l(s){\Vert }_2^2\cdots Eq.(1)\\ L_c^l(p)=\Vert G({\varphi }^l(p))-G({\varphi }^l(s)){\Vert }_F^2\cdots Eq.(2)\end{array}$$

where

• ${\varphi }^l(x)$: Activation of a pre-trained network at layer $l$.
  • $x$: Given the input image.
• $L_c^l(p),;L_c^l(p)$: Content and style loss at layer $l$ respectively.
• $G({\varphi }^l(s))$: Gram matrix associated with ${\varphi }^l(p)$.
  • Gram matrix $G_{ij}^l=\sum _k{\varphi }_{ik}^l{\varphi }_{jk}^l$: Variance of RGB between image textures.

The total loss is calculated as a weighted sum of losses a set of content layers $C$ and style layers $S$:

$$L_c(p)=\sum _{l\in C}{w}_c^l{L}_c^l(p),L_s(p)=\sum _{l\in S}{w}_s^l{L}_s^l(p)\cdots Eq.(3)$$

where $w_c^l,w_s^l$ are hyper-parameters to adjust the contribution of each layer to the loss. The problem is that these hyper-parameters have to be manually fine-tuned through try and error.
Finally, the objective of style transfer can be defined as:

$$mi{n}_p(L_c(p)+L_s(p))\cdots Eq.(4)$$

2.2 Real-time feed-forward style transfer

We can solve the objective in Eq. (4) using the iterative method but it can be very slow and has to be repeated for any given input.
A much faster method is to directly train a deep network $T$ which maps a given content image $c$ to a stylized image $p$. $T$ is a feed-forward CNN (parameterized by $\theta$) with residual connections between down-sampling and up-sampling layers and is trained on content images like as:

$$mi{n}_{\theta }(L_c(T(c))+L_s(T(c)))\cdots Eq.(5)$$

The second problem from this is that this generates only one stylization for a pair of style and content images.

3. Proposed Method

To address the two issues that are mentioned above, the authors condition the generated stylized image on additional input parameters where each parameter controls the share of the loss from a corresponding layer.

 

As for figural style, they enable the users to adjust $w_c^l,w_s^l$ without retraining the model by replacing them with input parameters and conditioning the generated style images on these parameters:

$$p=\mathrm{\Phi }(c,s,{\alpha }_c{\alpha }_s)$$

${\alpha }_c$ and ${\alpha }_s$ are vectors of parameters where each element corresponds to a different layer in content layers $C$ and style layers $S$ respectively. ${\alpha }_c^l$ and ${\alpha }_s^l$ replace the hyper-parameters $w_c^l$ and $w_s^l$:

$$L_c(p)=\sum _{l\in C}{\alpha }_c^l{L}_c^l(p),L_s(p)=\sum _{l\in S}{\alpha }_s^l{L}_s^l(p)\cdots Eq.(6)$$

To learn the effect of ${\alpha }_c$ and ${\alpha }_s$ on the objective, the authors use a technique called conditional instance normalization.
This method transforms the activations of a layer $x$ in the $T$ to a normalized activation $z$ which is conditioned on additional inputs ${\alpha }_{=}[{\alpha }_c,{\alpha }_s]$:

$$z={\gamma }_{\alpha }(\frac{x-\mu }{\sigma })+{\beta }_{\alpha }\cdots Eq.(7)$$

where $\mu$ and $\sigma$ are mean and standard deviation of activations at layer $x$ across spatial axes and ${\gamma }_{alpha}$ , ${\beta }_{\alpha }$ are the trained mean and standard deviation this transformation. These parameters can be approximated using a second neural network (in here, MLP) which will be trained end-to-end with $T$:

$${\gamma }_{\alpha },{\beta }_{\alpha }=\mathrm{\Lambda }({\alpha }_c,{\alpha }_s)\cdots Eq.(8)$$

Since $L^l$ can be different in scale, do normalize them using their exponential moving average as a normalizing factor, i.e. each $L^l$ will be normalized to:

$$L^l(p)=\frac{\sum _{\{}i\in C\cup S\}\overline{L^i}(p)}{\overline{L^l}(p)}\cdots Eq.(9)$$

where $\overline{L^l}(p)$ is the exponential moving average of $L^l(p)$.

4. Experiment setting

They trained $T$ and $\mathrm{\Lambda }$ jointly by sampling random values for $\alpha$ from $U(0,1)$. And they used ImageNet as content images while using paintings from Kaggle Painter by Numbers and textures from Descibable Texture Dataset as style images for training.
Similar to previous approaches, they used the last feature set of conv3 as content layer $C$ and used the last feature set of conv2, conv3, conv4 layers from VGG-19 network as style layers $S$. Since there is only one content layer, they fix ${\alpha }_c=1$.

5. Experiment

Reference

Babaeizadeh, Mohammad, and Golnaz Ghiasi. "Adjustable real-time style transfer." arXiv preprint arXiv:1811.08560 (2018).