Articulated body pose estimation

The basic idea of part based model can be attributed to the human skeleton. Any object having the property of articulation can be broken down into smaller parts wherein each part can take different orientations, resulting in different articulations of the same object. Different scales and orientations of the main object can be articulated to scales and orientations of the corresponding parts. To formulate the model so that it can be represented in mathematical terms, the parts are connected to each other using springs. As such, the model is also known as a spring model. The degree of closeness between each part is accounted for by the compression and expansion of the springs. There is geometric constraint on the orientation of springs. For example, limbs of legs cannot move 360 degrees. Hence parts cannot have that extreme orientation. This reduces the possible permutations.[6]

The spring model forms a graph G(V,E) where V (nodes) corresponds to the parts and E (edges) represents springs connecting two neighboring parts. Each location in the image can be reached by the ${\displaystyle x}$ and ${\displaystyle y}$ coordinates of the pixel location. Let ${\displaystyle \mathbf {p} _{i}(x,\,y)}$ be point at ${\displaystyle \mathbf {i} ^{th}}$ location. Then the cost associated in joining the spring between ${\displaystyle \mathbf {i} ^{th}}$ and the ${\displaystyle \mathbf {j} ^{th}}$ point can be given by ${\displaystyle S(\mathbf {p} _{i},\,\mathbf {p} _{j})=S(\mathbf {p} _{i}-\mathbf {p} _{j})}$. Hence the total cost associated in placing ${\displaystyle l}$ components at locations ${\displaystyle \mathbf {P} _{l}}$ is given by

${\displaystyle S(\mathbf {P} _{l})=\displaystyle \sum _{i=1}^{l}\;\displaystyle \sum _{j=1}^{i}\;\mathbf {s} _{ij}(\mathbf {p} _{i},\,\mathbf {p} _{j})}$

The above equation simply represents the spring model used to describe body pose. To estimate pose from images, cost or energy function must be minimized. This energy function consists of two terms. The first is related to how each component matches the image data and the second deals with how much the oriented (deformed) parts match, thus accounting for articulation along with object detection.[7]

The part models, also known as pictorial structures, are of one of the basic models on which other efficient models are built by slight modification. One such example is the flexible mixture model which reduces the database of hundreds or thousands of deformed parts by exploiting the notion of local rigidity.[8]

The kinematic skeleton is constructed by a tree-structured chain.[9] Each rigid body segment has its local coordinate system that can be transformed to the world coordinate system via a 4×4 transformation matrix ${\displaystyle T_{l}}$,

${\displaystyle T_{l}=T_{\operatorname {par} (l)}R_{l},}$

where ${\displaystyle R_{l}}$ denotes the local transformation from body segment ${\displaystyle S_{l}}$ to its parent ${\displaystyle \operatorname {par} (S_{l})}$. Each joint in the body has 3 degrees of freedom (DoF) rotation. Given a transformation matrix ${\displaystyle T_{l}}$ , the joint position at the T-pose can be transferred to its corresponding position in the world coordination. In many works, the 3D joint rotation is expressed as a normalized quaternion ${\displaystyle [x,y,z,w]}$ due to its continuity that can facilitate gradient-based optimization in the parameter estimation.

The first deep learning models that emerged focused on extracting the 2D positions of human joints in an image. Such models take in an image and pass it through a convolutional neural network to obtain a series of heatmaps (one for each joint) which take on high values where joints are detected.[10][11]

When there are multiple people per image, two main techniques have emerged for grouping joints within each person. In the first, "bottom-up" approach, the neural network is trained to also generate "part affinity fields" which indicate the location of limbs. Using these fields, joints can be grouped limb by limb by solving a series of assignment problems.[11] In the second, "top-down" approach, an additional network is used to first detect people in the image and then the pose estimation network is applied to each image.[12]

With the advent of multiple datasets with human pose annotated in multiple views,[13][14] models which detect 3D joint positions became more popular. These again fell into two categories In the first, a neural network is used to detect 2D joint positions from each view and these detections are then triangulated to obtain 3D joint positions.[15] The 2D network may be refined to produce better detections based on the 3D data.[16] Furthermore, such approaches often have filters in both 2D and 3D to refine the detected points.[17][18] In the second, a neural network is trained end-to-end to predict 3D joint positions directly from a set of images, without 2D joint position intermediate detections. Such approaches often project image features into a cube and then use a 3D convolutional neural network to predict a 3D heatmap for each joint.[19][16][20]

Concurrently with the work above, scientists have been working on estimating the full 3D shape of a human or animal from a set of images. Most of the work is based on estimating the appropriate pose of the skinned multi-person linear (SMPL) model[21] within an image. Variants of the SMPL model for other animals have also been developed.[22][23][24] Generally, some keypoints and a silhouette are detected for each animal within the image, and then the parameters 3D shape model are fit to match the position of keypoints and silhouette.

The above algorithms all rely on annotated images, which can be time-consuming to produce. To address this issue, computer vision researchers have developed new algorithms which can learn 3D keypoints given only annotated 2D images from a single view or identify keypoints given videos without any annotations.

Personal care robots may be deployed in future assisted living homes. For these robots, high-accuracy human detection and pose estimation is necessary to perform a variety of tasks, such as fall detection. Additionally, this application has a number of performance constraints.

Traditionally, character animation has been a manual process. However, poses can be synced directly to a real-life actor through specialized pose estimation systems. Older systems relied on markers or specialized suits. Recent advances in pose estimation and motion capture have enabled markerless applications, sometimes in real time.[25]

Car accidents account for about two percent of deaths globally each year. As such, an intelligent system tracking driver pose may be useful for emergency alerts . Along the same lines, pedestrian detection algorithms have been used successfully in autonomous cars, enabling the car to make smarter decisions.

Commercially, pose estimation has been used in the context of video games, popularized with the Microsoft Kinect sensor (a depth camera). These systems track the user to render their avatar in-game, in addition to performing tasks like gesture recognition to enable the user to interact with the game. As such, this application has a strict real-time requirement.[26]

Pose estimation has been used to detect postural issues such as scoliosis by analyzing abnormalities in a patient's posture,[27] physical therapy, and the study of the cognitive brain development of young children by monitoring motor functionality.[28]

Other applications include video surveillance, animal tracking and behavior understanding, sign language detection, advanced human–computer interaction, and markerless motion capturing.