Texture synthesis

The simplest way to generate a large image from a sample image is to tile it. This means multiple copies of the sample are simply copied and pasted side by side. The result is rarely satisfactory. Except in rare cases, there will be the seams in between the tiles and the image will be highly repetitive.

Stochastic texture synthesis methods produce an image by randomly choosing colour values for each pixel, only influenced by basic parameters like minimum brightness, average colour or maximum contrast. These algorithms perform well with stochastic textures only, otherwise they produce completely unsatisfactory results as they ignore any kind of structure within the sample image.

Algorithms of that family use a fixed procedure to create an output image, i. e. they are limited to a single kind of structured texture. Thus, these algorithms can both only be applied to structured textures and only to textures with a very similar structure. For example, a single purpose algorithm could produce high quality texture images of stonewalls; yet, it is very unlikely that the algorithm will produce any viable output if given a sample image that shows pebbles.

This method, proposed by the Microsoft group for internet graphics, is a refined version of tiling and performs the following three steps:

1. The output image is filled completely by tiling. The result is a repetitive image with visible seams.
2. Randomly selected parts of random size of the sample are copied and pasted randomly onto the output image. The result is a rather non-repetitive image with visible seams.
3. The output image is filtered to smooth edges.

The result is an acceptable texture image, which is not too repetitive and does not contain too many artifacts. Still, this method is unsatisfactory because the smoothing in step 3 makes the output image look blurred.

These methods, using Markov fields,[3] non-parametric sampling,[4] tree-structured vector quantization[5] and image analogies[6] are some of the simplest and most successful general texture synthesis algorithms. They typically synthesize a texture in scan-line order by finding and copying pixels with the most similar local neighborhood as the synthetic texture. These methods are very useful for image completion. They can be constrained, as in image analogies, to perform many interesting tasks. They are typically accelerated with some form of Approximate Nearest Neighbor method since the exhaustive search for the best pixel is somewhat slow. The synthesis can also be performed in multiresolution, such as through use of a noncausal nonparametric multiscale Markov random field.[7]

Image quilting

Patch-based texture synthesis creates a new texture by copying and stitching together textures at various offsets, similar to the use of the clone tool to manually synthesize a texture. Image quilting[8] and graphcut textures[9] are the best known patch-based texture synthesis algorithms. These algorithms tend to be more effective and faster than pixel-based texture synthesis methods.

More recently, deep learning methods were shown to be a powerful, fast and data-driven, parametric approach to texture synthesis. The work of Leon Gatys[10] is a milestone: he and his co-authors showed that filters from a discriminatively trained deep neural network can be used as effective parametric image descriptors, leading to a novel texture synthesis method.

Another recent development is the use of generative models for texture synthesis. The Spatial GAN[11] method showed for the first time the use of fully unsupervised GANs for texture synthesis. In a subsequent work,[12] the method was extended further—PSGAN can learn both periodic and non-periodic images in an unsupervised way from single images or large datasets of images. In addition, flexible sampling in the noise space allows to create novel textures of potentially infinite output size, and smoothly transition between them. This makes PSGAN unique with respect to the types of images a texture synthesis method can create.