Caltech 101

The Caltech 101 data set consists of a total of 9,146 images, split between 101 different object categories, as well as an additional background/clutter category.

Each object category contains between 40 and 800 images. Common and popular categories such as faces tend to have a larger number of images than others.

Each image is about 300x200 pixels. Images of oriented objects such as airplanes and motorcycles were mirrored to be left to right aligned and vertically oriented structures such as buildings were rotated to be off axis.

A set of annotations is provided for each image. Each set of annotations contains two pieces of information: the general bounding box in which the object is located and a detailed human-specified outline enclosing the object.

A Matlab script is provided with the annotations. It loads an image and its corresponding annotation file and displays them as a Matlab figure.

Caltech 101 has several advantages over other similar data sets:

• Uniform size and presentation:
  • Almost all the images within each category are uniform in image size and in the relative position of interest objects. Caltech 101 users generally do not need to crop or scale images before they can be used.
• Low level of clutter/occlusion:
  • Algorithms concerned with recognition usually function by storing features unique to the object. However, most images taken have varying degrees of background clutter, which means algorithms may build incorrectly.
• Detailed annotations

Weaknesses to the Caltech 101 data set[3][14] may be conscious trade-offs, but others are limitations of the data set. Papers that rely solely on Caltech 101 are frequently rejected.

Weaknesses include:

• The data set is too clean:
  • Images are very uniform in presentation, aligned from left to right, and usually not occluded. As a result, the images are not always representative of practical inputs that the algorithm might later expect to see. Under practical conditions, images are more cluttered, occluded and display greater variance in relative position and orientation of interest objects. The uniformity allows concepts to be derived using the average of a category, which is unrealistic.
• Limited number of categories:
  • The Caltech 101 data set represents only a small fraction of possible object categories.
• Some categories contain few images:
  • Certain categories are not represented as well as others, containing as few as 31 images.
  • This means that ${\displaystyle \mathrm {N} _{\mathrm {train} }\leq 30}$. The number of images used for training must be less than or equal to 30, which is not sufficient for all purposes.
• Aliasing and artifacts due to manipulation:
  • Some images have been rotated and scaled from their original orientation, and suffer from some amount of artifacts or aliasing.

• Caltech 256 is another image data set, created in 2007. It is a successor to Caltech 101. It is intended to address some of the weaknesses of Caltech 101. Overall, it is a more difficult data set than Caltech 101, but it suffers from comparable problems. It includes[3]
  • 30,607 images, covering a larger number of categories
  • Minimum number of images per category raised to 80
  • Images are not left-right aligned
  • More variation in image presentation
• LabelMe is an open, dynamic data set created at MIT Computer Science and Artificial Intelligence Laboratory (CSAIL). LabelMe takes a different approach to the problem of creating a large image data set, with different trade-offs.
  • 106,739 images, 41,724 annotated images, and 203,363 labeled objects.
  • Users may add images to the data set by upload, and add labels or annotations to existing images.
  • Due to its open nature, LabelMe has many more images covering a much wider scope than Caltech 101. However, since each person decides what images to upload, and how to label and annotate each image, the images are less consistent.
• VOC 2008 is a European effort to collect images for benchmarking visual categorization methods. Compared to Caltech 101/256, a smaller number of categories (about 20) are collected. The number of images in each category, however, is larger.
• Overhead Imagery Research Data Set (OIRDS) is an annotated library of imagery and tools.[15] OIRDS v1.0 is composed of passenger vehicle objects annotated in overhead imagery. Passenger vehicles in the OIRDS include cars, trucks, vans, etc. In addition to the object outlines, the OIRDS includes subjective and objective statistics that quantify the vehicle within the image's context. For example, subjective measures of image clutter, clarity, noise, and vehicle color are included along with more objective statistics such as ground sample distance (GSD), time of day, and day of year.
  • ~900 images, containing ~1800 annotated images
  • ~30 annotations per object
  • ~60 statistical measures per object
  • Wide variation in object context
  • Limited to passenger vehicles in overhead imagery
• MICC-Flickr 101 is an image data set created at the Media Integration and Communication Center (MICC), University of Florence, in 2012. It is based on Caltech 101 and is collected from Flickr. MICC-Flickr 101[16] corrects the main drawback of Caltech 101, i.e. its low inter-class variability and provides social annotations through user tags. It builds on a standard and widely used data set composed of a manageable number of categories (101) and therefore can be used to compare object categorization performance in a constrained scenario (Caltech 101) and object categorization "in the wild" (MICC-Flickr 101) on the same 101 categories.