Apache OODT

The Apache Object Oriented Data Technology (OODT) is an open source data management system framework that is managed by the Apache Software Foundation. OODT was originally developed at NASA Jet Propulsion Laboratory to support capturing, processing and sharing of data for NASA's scientific archives.

Apache OODT
Developer(s)Apache Software Foundation
Stable release
1.9.1 / October 3, 2021 (2021-10-03)[1]
RepositoryOODT Repository
Written inJava
Operating systemCross-platform
TypeSearch and index API
LicenseApache License 2.0
Websiteoodt.apache.org

History

The project started out as an internal NASA Jet Propulsion Laboratory project incepted by Daniel J. Crichton, Sean Kelly and Steve Hughes. The early focus of the effort was on information integration and search using XML as described in Crichton et al.'s paper in the CODATA meeting in 2000.[2]

After deploying OODT to the Planetary Data System and to the National Cancer Institute EDRN or Early Detection Research Network project, OODT in 2005 moved into the era of large scale data processing and management via NASA's Orbiting Carbon Observatory (OCO) project. OODT's role on OCO was to usher in a new data management processing framework that instead of tens of jobs per day and tens of gigabytes of data would handle 10,000 jobs per day and hundreds of terabytes of data. This required an overhaul of OODT to support these new requirements. Dr. Chris Mattmann at NASA JPL led a team of 3-4 developers between 2005-2009 and completely re-engineered OODT to support these new requirements.

Influenced by the emerging efforts in Apache Nutch and Hadoop which Mattmann participated in, OODT was given an overhaul making it more amenable towards Apache Software Foundation like projects. In addition, Mattmann had a close relationship with Dr. Justin Erenkrantz, who as the Apache Software Foundation President at the time, and the idea to bring OODT to the Apache Software Foundation emerged. In 2009, Mattmann and his team received approval from NASA and from JPL to bring OODT to Apache making it the first NASA project to be stewarded by the foundation. Seven years later, the project has released a version 1.0.

Features

OODT focuses on two canonical use cases: Big Data processing and on Information integration. Both were described in Mattmann's ICSE 2006[3] and SMC-IT 2009[4] papers. It provides three core services.

File Manager

A File Manager is responsible for tracking file locations, their metadata, and for transferring files from a staging area to controlled access storage.

Workflow Manager

A Workflow Manager captures control flow and data flow for complex processes, and allows for reproducibility and the construction of scientific pipelines.

Resource Manager

A Resource Manager handles allocation of Workflow Tasks and other jobs to underlying resources, e.g., Python jobs go to nodes with Python installed on them; jobs that require a large disk or CPU are properly sent to those nodes that fulfill those requirements.

In addition to the three core services, OODT provides three client-oriented frameworks that build on these services.

File Crawler

A file Crawler automatically extracts metadata and uses Apache Tika to identify file types and ingest the associated information into the File Manager.

Catalog and Archive Crawling Framework

A Push/Pull framework acquires remote files and makes them available to the system.

Catalog and Archive Service Production Generation Executive (CAS-PGE)

A scientific algorithm wrapper (called CAS-PGE, for Catalog and Archive Service Production Generation Executive) encapsulates scientific codes and allows for their execution independent of environment, and while doing so capturing provenance, and making the algorithms easily integrated into a production system.

CAS RESTful Services

A Set of RESTful APIs which exposes the capabilities of File Manager, Workflow Manager and Resource manager components.

OPSUI Monitor Dashboard

A web application for exposing services form the underlying OODT product / workflow / resource managing Control Systems via the JAX-RS[5] specification. At this stage it is built using Apache Wicket[6] components.

The overall motivation for OODT's re-architecting was described in a paper in Nature (journal) in 2013 by Mattmann called A Vision for Data Science.[7]

OODT is written in the Java, and through its REST API[8] used in other languages including Python (programming language).

Notable uses

OODT has been recently highlighted as contributing to NASA missions including Soil Moisture Active Passive[9] and New Horizons.[10] OODT also helps to power the Square Kilometre Array telescope[11] increasing the scope of its use from Earth science, Planetary science, radio astronomy, and to other sectors. OODT is also used within bioinformatics and is a part of the Knowledgent Big Data Platform.[12]

References
  1. "[ANNOUNCE] Apache OODT 1.9.1 released". Retrieved 27 September 2022.
  2. Crichton, Daniel; Hughes, John; Hyon, Jason; Kelly, Sean (2000). "Science Search and Retrieval using XML". The Second National Conference on Scientific and Technical Data, US National Committee for CODATA, National Research Council.
  3. Mattmann, Chris A.; Crichton, Daniel J.; Medvidovic, Nenad; Hughes, Steve (2006-01-01). "A software architecture-based framework for highly distributed and data intensive scientific applications". Proceedings of the 28th international conference on Software engineering. ICSE '06. New York, NY, USA: ACM. pp. 721–730. doi:10.1145/1134285.1134400. ISBN 978-1595933751. S2CID 7699385.
  4. Mattmann, C. A.; Freeborn, D.; Crichton, D.; Foster, B.; Hart, A.; Woollard, D.; Hardman, S.; Ramirez, P.; Kelly, S. (2009-07-01). "A Reusable Process Control System Framework for the Orbiting Carbon Observatory and NPP Sounder PEATE Missions". 2009 Third IEEE International Conference on Space Mission Challenges for Information Technology. pp. 165–172. doi:10.1109/SMC-IT.2009.27. ISBN 978-0-7695-3637-8. S2CID 705732.
  5. en:Java_API_for_RESTful_Web_Services, oldid 889016216
  6. en:Apache_Wicket, oldid 875122637
  7. Mattmann, Chris A. (2013-01-24). "Computing: A vision for data science". Nature. 493 (7433): 473–475. Bibcode:2013Natur.493..473M. doi:10.1038/493473a. ISSN 0028-0836. PMID 23344342.
  8. "Apache OODT APIs - OODT - Apache Software Foundation". cwiki.apache.org. Retrieved 2016-06-27.
  9. "Apache - The ASF on Twitter". Retrieved 2016-06-27.
  10. "Apache - The ASF on Twitter". Retrieved 2016-06-27.
  11. "Apache - The ASF on Twitter". Retrieved 2016-06-27.
  12. "Q&A on the Advantages of OODT - Object Oriented Data Technology - Knowledgent Perspectives". 2014-07-30. Archived from the original on 2015-04-14. Retrieved 2016-06-27.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.