rasdaman

Based on an array algebra[7] specifically developed for database purposes, rasdaman adds a new attribute type, array, to the relational model. As this array definition is parametrized it constitutes a second-order construct or template; this fact is reflected by the second-order functionals in the algebra and query language.

For historical reasons, tables are called collections, as initial design emphasized an embedding into the object-oriented database standard, ODMG. Anticipating a full integration with SQL, rasdaman collections represent a binary relation with the first attribute being an object identifier and the second being the array. This allows the establishment of foreign key references between arrays and regular relational tuples.

The rasdaman query language, rasql, embeds itself into standard SQL and its set-oriented processing. On the new attribute type, multi-dimensional arrays, a set of extra operations is provided which all are based on a minimal set of algebraically defined core operators, an array constructor (which establishes a new array and fills it with values) and an array condenser (which, similarly to SQL aggregates, derives scalar summary information from an array). The query language is declarative (and, hence, optimizable) and safe in evaluation - that is: every query is guaranteed to return after a finite number of processing steps.

The rasql query guide[8] provides details, here some examples may illustrate its use:

• "From all 4-D x/y/z/t climate simulation data cubes, a cutout which contains all in x, a y extract between 100 and 200, all available along z, and a slice at position 42 (effectively resulting in a 3-D x/y/z cube)":

select c[ *:*, 100:200, *:*, 42 ] 
from   ClimateSimulations as c

• "In all Landsat satellite images, suppress all non-green areas":

select img * (img.green > 130)
from   LandsatArchive as img

Note: this is a very naive phrasing of vegetation search; in practice one would use the NDVI formula, use null values for cloud masking, and several more techniques.

• "All MRI images where, in some region defined by the bit masks, intensity exceeds a threshold of 250":

select img
from   MRI as img, Masks as m
where  some_cells( img > 250 and m )

• "A 2-D x/y slice from all 4-D climate simulation data cubes, each one encoded in PNG format":

select png( c[ *:*, *:*, 100, 42 ] )
from   ClimateSimulations as c

Sample array tiling in rasdaman

Raster objects are maintained in a standard relational database, based on the partitioning of a raster object into tiles.[9] Aside from a regular subdivision, any user or system generated partitioning is possible. As tiles form the unit of disk access, it is of critical importance that the tiling pattern is adjusted to the query access patterns; several tiling strategies assist in establishing a well-performing tiling. A geo index is employed to quickly determine the tiles affected by a query. Optionally, tiles are compressed using one of various choices, including lossless and lossy (wavelet) algorithms; independently from that, query results can be compressed for transfer to the client. Both tiling strategy and compression comprise database tuning parameters.

Tiles and tile index are stored as BLOBs in a relational database which also holds the data dictionary needed by rasdaman's dynamic type system. Adapters are available for several relational systems, among them open-source PostgreSQL. For arrays larger than disk space, hierarchical storage management (HSM) support has been developed.

Queries are parsed, optimised, and executed in the rasdaman server. The parser receives the query string and generates the operation tree. Further, it applies algebraic optimisation rules to the query tree where applicable; of the 150 algebraic rewriting rules, 110 are actually optimising while the other 40 serve to transform the query into canonical form. Parsing and optimization together take less than a millisecond on a laptop.

Execution follows a tile streaming paradigm: whenever possible, array tiles addressed by a query are fetched sequentially, and each tile is discarded after processing. This leads to an architecture scalable to data volumes exceeding server main memory by orders of magnitude.

Query execution is parallelised. First, rasdaman offers inter-query parallelism: A dispatcher schedules requests into a pool of server processes on a per-transaction basis. Intra-query parallelism transparently distributes query subtrees across available cores, GPUs, or cloud nodes.

The primary interface to rasdaman is the query language. Embeddings into C++ and Java APIs allow invocation of queries, as well as client-side convenience functions for array handling. Arrays per se are delivered in the main memory format of the client language and processor architecture, ready for further processing. Data format codecs allow to retrieve arrays in common raster formats, such as CSV, PNG, and NetCDF.

A Web design toolkit, raswct, is provided which makes the creation of Web query frontends easy, including graphical widgets for parametrized query handling, such as sliders for thresholds in queries.

A Java servlet, petascope, running as a rasdaman client offers Web service interfaces specifically for geo data access, processing and filtering. The following OGC standards are supported: WMS, WCS, WCPS, and WPS.

For WCS and WCPS, rasdaman is the reference implementation.