LabelMe

Since the text labels for objects provided in LabelMe come from user input, there is a lot of variation in the labels used (as described above). Because of this, analysis of objects can be difficult. For example, a picture of a dog might be labeled as dog, canine, hound, pooch, or animal. Ideally, when using the data, the object class dog at the abstract level should incorporate all of these text labels.

WordNet is a database of words organized into a structural way. It allows assigning a word to a category, or in WordNet language: a sense. Sense assignment is not easy to do automatically. When the authors of LabelMe tried automatic sense assignment, they found that it was prone to a high rate of error, so instead they assigned words to senses manually. At first, this may seem like a daunting task since new labels are added to the LabelMe project continuously. To the right is a graph comparing the growth of polygons to the growth of words (descriptions). As you can see, the growth of words is small compared with the continuous growth of polygons, and therefore is easy enough to keep up to date manually by the LabelMe team.[3]

Once WordNet assignment is done, searches in the LabelMe database are much more effective. For example, a search for animal might bring up pictures of dogs, cats and snakes. However, since the assignment was done manually, a picture of a computer mouse labeled as mouse would not show up in a search for animals. Also, if objects are labeled with more complex terms like dog walking, WordNet still allows the search of dog to return these objects as results. WordNet makes the LabelMe database much more useful.

Having a large dataset of objects where overlap is allowed provides enough data to try and categorize objects as being a part of another object. For example, most of the labels assigned wheel are probably part of objects assigned to other labels like car or bicycle. These are called part labels. To determine if label P is a part label for label O:[4]

• Let ${\displaystyle \mathrm {I} _{\mathrm {O} }\,}$ denote the set of images containing an object (e.g. car)
• Let ${\displaystyle \mathrm {I} _{\mathrm {P} }\,}$ denote the set of images containing a part (e.g. wheel)
• Let the overlap score between object O and part P, ${\displaystyle \mathrm {S} _{\mathrm {O} ,\mathrm {P} }\,}$, be defined as the ratio of the intersection area to the area of the part polygon. (e.g. ${\displaystyle {\frac {\mathrm {A} (\mathrm {O} \cap \mathrm {P} )}{\mathrm {A} (\mathrm {P} )}}\,}$)
• Let ${\displaystyle \mathrm {I} _{\mathrm {O} ,\mathrm {P} }\subseteq \mathrm {I} _{\mathrm {P} }\,}$ denote the images where object and part polygons have ${\displaystyle \mathrm {S} _{\mathrm {O} ,\mathrm {P} }>\beta \,}$ where ${\displaystyle \beta \,}$ is some threshold value. The authors of LabelMe use ${\displaystyle \beta =0.5\,}$
• The object-part score for a candidate label is ${\displaystyle {\frac {\mathrm {N} _{\mathrm {O} ,\mathrm {P} }}{\mathrm {N} _{\mathrm {P} }+\alpha }}\,}$ where ${\displaystyle \mathrm {N} _{\mathrm {O} ,\mathrm {P} }\,}$ and ${\displaystyle \mathrm {N} _{\mathrm {P} }\,}$ are the number of images in ${\displaystyle \mathrm {I} _{\mathrm {O} ,\mathrm {P} }\,}$ and ${\displaystyle \mathrm {I} _{\mathrm {P} }\,}$, respectively, and ${\displaystyle \alpha \,}$ is a concentration parameter. The authors of LabelMe use ${\displaystyle \alpha =5\,}$.

This algorithm allows the automatic classification of parts of an object when the part objects are frequently contained within the outer object.

Another instance of object overlap is when one object is actually on top of the other. For example, an image might contain a person standing in front of a building. The person is not a part label as above since the person is not part of the building. Instead, they are two separate objects that happen to overlap. To automatically determine which object is the foreground and which is the background, the authors of LabelMe propose several options:[5]

• If an object is completely contained within another object, then the inner object must be in the foreground. Otherwise, it would not be visible in the image. The only exception is with transparent or translucent objects, but these occur rarely.
• One of the objects could be labeled as something that cannot be in the foreground. Examples are sky, ground, or road.
• The object with more polygon points inside the intersecting area is most likely the foreground. The authors tested this hypothesis and found it to be highly accurate.
• Histogram intersection[6] can be used. To do this, a color histogram in the intersecting areas is compared to the color histogram of the two objects. The object with the closer color histogram is assigned as the foreground. This method is less accurate than counting the polygon points.