Digital image processing

The basis for modern image sensors is metal-oxide-semiconductor (MOS) technology,[5] which originates from the invention of the MOSFET (MOS field-effect transistor) by Mohamed M. Atalla and Dawon Kahng at Bell Labs in 1959.[6] This led to the development of digital semiconductor image sensors, including the charge-coupled device (CCD) and later the CMOS sensor.[5]

The charge-coupled device was invented by Willard S. Boyle and George E. Smith at Bell Labs in 1969.[7] While researching MOS technology, they realized that an electric charge was the analogy of the magnetic bubble and that it could be stored on a tiny MOS capacitor. As it was fairly straightforward to fabricate a series of MOS capacitors in a row, they connected a suitable voltage to them so that the charge could be stepped along from one to the next.[5] The CCD is a semiconductor circuit that was later used in the first digital video cameras for television broadcasting.[8]

The NMOS active-pixel sensor (APS) was invented by Olympus in Japan during the mid-1980s. This was enabled by advances in MOS semiconductor device fabrication, with MOSFET scaling reaching smaller micron and then sub-micron levels.[9][10] The NMOS APS was fabricated by Tsutomu Nakamura's team at Olympus in 1985.[11] The CMOS active-pixel sensor (CMOS sensor) was later developed by Eric Fossum's team at the NASA Jet Propulsion Laboratory in 1993.[12] By 2007, sales of CMOS sensors had surpassed CCD sensors.[13]

An important development in digital image compression technology was the discrete cosine transform (DCT), a lossy compression technique first proposed by Nasir Ahmed in 1972.[14] DCT compression became the basis for JPEG, which was introduced by the Joint Photographic Experts Group in 1992.[15] JPEG compresses images down to much smaller file sizes, and has become the most widely used image file format on the Internet.[16] Its highly efficient DCT compression algorithm was largely responsible for the wide proliferation of digital images and digital photos,[17] with several billion JPEG images produced every day as of 2015.[18]

Electronic signal processing was revolutionized by the wide adoption of MOS technology in the 1970s.[19] MOS integrated circuit technology was the basis for the first single-chip microprocessors and microcontrollers in the early 1970s,[20] and then the first single-chip digital signal processor (DSP) chips in the late 1970s.[21][22] DSP chips have since been widely used in digital image processing.[21]

The discrete cosine transform (DCT) image compression algorithm has been widely implemented in DSP chips, with many companies developing DSP chips based on DCT technology. DCTs are widely used for encoding, decoding, video coding, audio coding, multiplexing, control signals, signaling, analog-to-digital conversion, formatting luminance and color differences, and color formats such as YUV444 and YUV411. DCTs are also used for encoding operations such as motion estimation, motion compensation, inter-frame prediction, quantization, perceptual weighting, entropy encoding, variable encoding, and motion vectors, and decoding operations such as the inverse operation between different color formats (YIQ, YUV and RGB) for display purposes. DCTs are also commonly used for high-definition television (HDTV) encoder/decoder chips.[23]

In 1972, the engineer from British company EMI Housfield invented the X-ray computed tomography device for head diagnosis, which is what is usually called CT (computer tomography). The CT nucleus method is based on the projection of the human head section and is processed by computer to reconstruct the cross-sectional image, which is called image reconstruction. In 1975, EMI successfully developed a CT device for the whole body, which obtained a clear tomographic image of various parts of the human body. In 1979, this diagnostic technique won the Nobel Prize.[4] Digital image processing technology for medical applications was inducted into the Space Foundation Space Technology Hall of Fame in 1994.[24]

Digital filters are used to blur and sharpen digital images. Filtering can be performed by:

• convolution with specifically designed kernels (filter array) in the spatial domain[25]
• masking specific frequency regions in the frequency (Fourier) domain

The following examples show both methods:[26]

Filter type	Kernel or mask	Example
Original Image	${\displaystyle {\begin{bmatrix}0&0&0\\0&1&0\\0&0&0\end{bmatrix}}}$
Spatial Lowpass	${\displaystyle {\frac {1}{9}}\times {\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}}}$
Spatial Highpass	${\displaystyle {\begin{bmatrix}0&-1&0\\-1&4&-1\\0&-1&0\end{bmatrix}}}$
Fourier Representation	Pseudo-code: image = checkerboard F = Fourier Transform of image Show Image: log(1+Absolute Value(F))
Fourier Lowpass
Fourier Highpass

Images are typically padded before being transformed to the Fourier space, the highpass filtered images below illustrate the consequences of different padding techniques:

Zero padded	Repeated edge padded

Notice that the highpass filter shows extra edges when zero padded compared to the repeated edge padding.

MATLAB example for spatial domain highpass filtering.

img=checkerboard(20);                           % generate checkerboard
% **************************  SPATIAL DOMAIN  ***************************
klaplace=[0 -1 0; -1 5 -1;  0 -1 0];             % Laplacian filter kernel
X=conv2(img,klaplace);                          % convolve test img with
                                                % 3x3 Laplacian kernel
figure()
imshow(X,[])                                    % show Laplacian filtered
title('Laplacian Edge Detection')

Affine transformations enable basic image transformations including scale, rotate, translate, mirror and shear as is shown in the following examples:[26]

Transformation Name	Affine Matrix	Example
Identity	${\displaystyle {\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}}}$
Reflection	${\displaystyle {\begin{bmatrix}-1&0&0\\0&1&0\\0&0&1\end{bmatrix}}}$
Scale	${\displaystyle {\begin{bmatrix}c_{x}=2&0&0\\0&c_{y}=1&0\\0&0&1\end{bmatrix}}}$
Rotate	${\displaystyle {\begin{bmatrix}\cos(\theta )&\sin(\theta )&0\\-\sin(\theta )&\cos(\theta )&0\\0&0&1\end{bmatrix}}}$	where θ = π/6 =30°
Shear	${\displaystyle {\begin{bmatrix}1&c_{x}=0.5&0\\c_{y}=0&1&0\\0&0&1\end{bmatrix}}}$

To apply the affine matrix to an image, the image is converted to matrix in which each entry corresponds to the pixel intensity at that location. Then each pixel's location can be represented as a vector indicating the coordinates of that pixel in the image, [x, y], where x and y are the row and column of a pixel in the image matrix. This allows the coordinate to be multiplied by an affine-transformation matrix, which gives the position that the pixel value will be copied to in the output image.

However, to allow transformations that require translation transformations, 3 dimensional homogeneous coordinates are needed. The third dimension is usually set to a non-zero constant, usually 1, so that the new coordinate is [x, y, 1]. This allows the coordinate vector to be multiplied by a 3 by 3 matrix, enabling translation shifts. So the third dimension, which is the constant 1, allows translation.

Because matrix multiplication is associative, multiple affine transformations can be combined into a single affine transformation by multiplying the matrix of each individual transformation in the order that the transformations are done. This results in a single matrix that, when applied to a point vector, gives the same result as all the individual transformations performed on the vector [x, y, 1] in sequence. Thus a sequence of affine transformation matrices can be reduced to a single affine transformation matrix.

For example, 2 dimensional coordinates only allow rotation about the origin (0, 0). But 3 dimensional homogeneous coordinates can be used to first translate any point to (0, 0), then perform the rotation, and lastly translate the origin (0, 0) back to the original point (the opposite of the first translation). These 3 affine transformations can be combined into a single matrix, thus allowing rotation around any point in the image.[27]

Mathematical morphology is suitable for denoising images. Structuring element are important in Mathematical morphology.

The following examples are about Structuring elements. The denoise function, image as I, and structuring element as B are shown as below and table.

e.g. ${\displaystyle (I')={\begin{bmatrix}45&50&65\\40&60&55\\25&15&5\end{bmatrix}}B={\begin{bmatrix}1&2&1\\2&1&1\\1&0&3\end{bmatrix}}}$

Define Dilation(I, B)(i,j) = ${\displaystyle max\{I(i+m,j+n)+B(m,n)\}}$. Let Dilation(I,B) = D(I,B)

D(I', B)(1,1) = ${\displaystyle max(45+1,50+2,65+1,40+2,60+1,55+1,25+1,15+0,5+3)=66}$

Define Erosion(I, B)(i,j) = ${\displaystyle min\{I(i+m,j+n)-B(m,n)\}}$. Let Erosion(I,B) = E(I,B)

E(I', B)(1,1) = ${\displaystyle min(45-1,50-2,65-1,40-2,60-1,55-1,25-1,15-0,5-3)=2}$

After dilation ${\displaystyle (I')={\begin{bmatrix}45&50&65\\40&66&55\\25&15&5\end{bmatrix}}}$ After erosion ${\displaystyle (I')={\begin{bmatrix}45&50&65\\40&2&55\\25&15&5\end{bmatrix}}}$

An opening method is just simply erosion first, and then dilation while the closing method is vice versa. In reality, the D(I,B) and E(I,B) can implemented by Convolution

Structuring element	Mask	Code	Example
Original Image	None	Use Matlab to read Original image original = imread('scene.jpg'); image = rgb2gray(original); [r, c, channel] = size(image); se = logical([1 1 1 ; 1 1 1 ; 1 1 1]); [p, q] = size(se); halfH = floor(p/2); halfW = floor(q/2); time = 3; % denoising 3 times with all method	Original lotus
Dilation	${\displaystyle {\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}}}$	Use Matlab to dilation imwrite(image, "scene_dil.jpg") extractmax = zeros(size(image), class(image)); for i = 1 : time dil_image = imread('scene_dil.jpg'); for col = (halfW + 1): (c - halfW) for row = (halfH + 1) : (r - halfH) dpointD = row - halfH; dpointU = row + halfH; dpointL = col - halfW; dpointR = col + halfW; dneighbor = dil_image(dpointD:dpointU, dpointL:dpointR); filter = dneighbor(se); extractmax(row, col) = max(filter); end end imwrite(extractmax, "scene_dil.jpg"); end	Denoising picture with dilation method
Erosion	${\displaystyle {\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}}}$	Use Matlab to erosion imwrite(image, 'scene_ero.jpg'); extractmin = zeros(size(image), class(image)); for i = 1: time ero_image = imread('scene_ero.jpg'); for col = (halfW + 1): (c - halfW) for row = (halfH +1): (r -halfH) pointDown = row-halfH; pointUp = row+halfH; pointLeft = col-halfW; pointRight = col+halfW; neighbor = ero_image(pointDown:pointUp,pointLeft:pointRight); filter = neighbor(se); extractmin(row, col) = min(filter); end end imwrite(extractmin, "scene_ero.jpg"); end
Opening	${\displaystyle {\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}}}$	Use Matlab to Opening imwrite(extractmin, "scene_opening.jpg") extractopen = zeros(size(image), class(image)); for i = 1 : time dil_image = imread('scene_opening.jpg'); for col = (halfW + 1): (c - halfW) for row = (halfH + 1) : (r - halfH) dpointD = row - halfH; dpointU = row + halfH; dpointL = col - halfW; dpointR = col + halfW; dneighbor = dil_image(dpointD:dpointU, dpointL:dpointR); filter = dneighbor(se); extractopen(row, col) = max(filter); end end imwrite(extractopen, "scene_opening.jpg"); end
Closing	${\displaystyle {\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}}}$	Use Matlab to Closing imwrite(extractmax, "scene_closing.jpg") extractclose = zeros(size(image), class(image)); for i = 1 : time ero_image = imread('scene_closing.jpg'); for col = (halfW + 1): (c - halfW) for row = (halfH + 1) : (r - halfH) dpointD = row - halfH; dpointU = row + halfH; dpointL = col - halfW; dpointR = col + halfW; dneighbor = ero_image(dpointD:dpointU, dpointL:dpointR); filter = dneighbor(se); extractclose(row, col) = min(filter); end end imwrite(extractclose, "scene_closing.jpg"); end	Denoising picture with closing method

In order to apply the denoising method to an image, the image is converted into grayscale. A mask with denoising method is logical matrix with ${\displaystyle [111;111;111]}$. The denoising methods start from the center of the picture with half of height, half of width, and end with the image boundary of row number, column number. Neighbor is a block in the original image with the boundary [the point below center: the point above, the point on left of center: the point on the right of center]. Convolution Neighbor and structuring element and then replace the center with a minimum of neighbor.

Take the Closing method for example.

Dilation first

1. Read the image and convert it into grayscale with Matlab.
   1. Get the size of an image. The return value row numbers and column numbers are the boundaries we are going to use later.
   2. structuring elements depend on your dilation or erosion function. The minimum of the neighbor of a pixel leads to an erosion method and the maximum of neighbor leads to a dilation method.
   3. Set the time for dilation, erosion, and closing.
2. Create a zero matrix of the same size as the original image.
3. Dilation first with structuring window.
   1. structuring window is 3*3 matrix and convolution
   2. For loop extract the minimum with window from row range [2 ~ image height - 1] with column range [2 ~ image width - 1]
4. Fill the minimum value to the zero matrix and save a new image
   1. For the boundary, it can still be improved. Since in the method, a boundary is ignored. Padding elements can be applied to deal with boundaries.

Then Erosion (Take the dilation image as input)

1. Create a zero matrix of the same size as the original image.
2. Erosion with structuring window.
   1. structuring window is 3*3 matrix and convolution
   2. For loop extract the maximum with window from row range [2 ~ image height - 1] with column range [2 ~ image width - 1]
3. Fill the maximum value to the zero matrix and save a new image
   1. For the boundary, it can still be improved. Since in the method, boundary is ignored. Padding elements can be applied to deal with boundaries.
4. Results are as above table shown

Digital cameras generally include specialized digital image processing hardware – either dedicated chips or added circuitry on other chips – to convert the raw data from their image sensor into a color-corrected image in a standard image file format. Additional post processing techniques increase edge sharpness or color saturation to create more naturally looking images.

Westworld (1973) was the first feature film to use the digital image processing to pixellate photography to simulate an android's point of view.[28] Image processing is also vastly used to produce the chroma key effect that replaces the background of actors with natural or artistic scenery.

Face detection process

Face detection can be implemented with Mathematical morphology, Discrete cosine transform which is usually called DCT, and horizontal Projection (mathematics).

General method with feature-based method

The feature-based method of face detection is using skin tone, edge detection, face shape, and feature of a face (like eyes, mouth, etc.) to achieve face detection. The skin tone, face shape, and all the unique elements that only the human face have can be described as features.

Process explanation

1. Given a batch of face images, first, extract the skin tone range by sampling face images. The skin tone range is just a skin filter.
   1. Structural similarity index measure (SSIM) can be applied to compare images in terms of extracting the skin tone.
   2. Normally, HSV or RGB color spaces are suitable for the skin filter. E.g. HSV mode, the skin tone range is [0,48,50] ~ [20,255,255]
2. After filtering images with skin tone, to get the face edge, morphology and DCT are used to remove noise and fill up missing skin areas.
   1. Opening method or closing method can be used to achieve filling up missing skin.
   2. DCT is to avoid the object with tone-like skin. Since human faces always have higher texture.
   3. Sobel operator or other operators can be applied to detect face edge.
3. To position human features like eyes, using the projection and find the peak of the histogram of projection help to get the detail feature like mouse, hair, and lip.
   1. Projection is just projecting the image to see the high frequency which is usually the feature position.

Image quality can be influenced by camera vibration, over-exposure, gray level distribution too centralized, and noise, etc. For example, noise problem can be solved by Smoothing method while gray level distribution problem can be improved by Histogram Equalization.

Smoothing method

In drawing, if there is some dissatisfied color, taking some color around dissatisfied color and averaging them. This is an easy way to think of Smoothing method.

Smoothing method can be implemented with mask and Convolution. Take the small image and mask for instance as below.

image is ${\displaystyle {\begin{bmatrix}2&5&6&5\\3&1&4&6\\1&28&30&2\\7&3&2&2\end{bmatrix}}}$

mask is ${\displaystyle {\begin{bmatrix}1/9&1/9&1/9\\1/9&1/9&1/9\\1/9&1/9&1/9\end{bmatrix}}}$

After Convolution and smoothing, image is ${\displaystyle {\begin{bmatrix}2&5&6&5\\3&9&10&6\\1&9&9&2\\7&3&2&2\end{bmatrix}}}$

Oberseving image[1, 1], image[1, 2], image[2, 1], and image[2, 2].

The original image pixel is 1, 4, 28, 30. After smoothing mask, the pixel becomes 9, 10, 9, 9 respectively.

new image[1, 1] = ${\displaystyle {\tfrac {1}{9}}}$ * (image[0,0]+image[0,1]+image[0,2]+image[1,0]+image[1,1]+image[1,2]+image[2,0]+image[2,1]+image[2,2])

new image[1, 1] = floor(${\displaystyle {\tfrac {1}{9}}}$ * (2+5+6+3+1+4+1+28+30)) = 9

new image[1, 2] = floor({${\displaystyle {\tfrac {1}{9}}}$ * (5+6+5+1+4+6+28+30+2)) = 10

new image[2, 1] = floor(${\displaystyle {\tfrac {1}{9}}}$ * (3+1+4+1+28+30+73+3+2)) = 9

new image[2, 2] = floor(${\displaystyle {\tfrac {1}{9}}}$ * (1+4+6+28+30+2+3+2+2)) = 9

Gray Level Histogram method

Generally, given a gray level histogram from an image as below. Changing the histogram to uniform distribution from an image is usually what we called Histogram equalization.

Figure 1

Figure 2

In discrete time, the area of gray level histogram is ${\displaystyle \sum _{i=0}^{k}H(p_{i})}$(see figure 1) while the area of uniform distribution is ${\displaystyle \sum _{i=0}^{k}G(q_{i})}$(see figure 2). It's clear that the area won't change, so ${\displaystyle \sum _{i=0}^{k}H(p_{i})=\sum _{i=0}^{k}G(q_{i})}$.

From the uniform distribution, the probability of ${\displaystyle q_{i}}$ is ${\displaystyle {\tfrac {N^{2}}{q_{k}-q_{0}}}}$ while the ${\displaystyle 0<i<k}$

In continuous time, the equation is ${\displaystyle \displaystyle \int _{q_{0}}^{q}{\tfrac {N^{2}}{q_{k}-q_{0}}}ds=\displaystyle \int _{p_{0}}^{p}H(s)ds}$.

Moreover, based on the definition of a function, the Gray level histogram method is like finding a function ${\displaystyle f}$ that satisfies f(p)=q.

Improvement method	Issue	Before improvement	Process	After improvement
Smoothing method	noise with Matlab, salt & pepper with 0.01 parameter is added to the original image in order to create a noisy image.		read image and convert image into grayscale convolution the graysale image with the mask ${\displaystyle {\begin{bmatrix}1/9&1/9&1/9\\1/9&1/9&1/9\\1/9&1/9&1/9\end{bmatrix}}}$ denoisy image will be the result of step 2.
Histogram Equalization	Gray level distribution too centralized		Refer to the Histogram equalization