Home · All Classes · Functions · Building · Tuning 

SVMT Developers Documentation

1.0.1

Introduction

This documentation concerns the inner workings of SVMT, the ESO Survey Visualization and Monitoring Tool. This document is not targeted at end users but rather at developers, maintainers or administrators who want to deploy a new instance of the server.

SVMT client is a plugin-in for the open source software Stellarium. As such, the developer should refer to the Stellarium developer's documentation available at http://stellarium.org/doc/head/, and especially to the section dedicated to plugins (http://stellarium.org/doc/head/plugins.html).

Architecture Overview

From version 1.0.0, SVMT is based on a client-server architecture. The client is a plugin running inside Stellarium essentially managing display and user interaction. At startup, the client connects to an HTTP server and download static files, such as definition of the GUI and configuration files. After that, all query operations needed by the client are sent and performed remotely on an instance of a SVMT query engine. The SVMT query engine is a cgi program running e.g. in an apache web server. Client and remote query engine communicate using a simple HTTP JSON-based stateless protocol.

In this documentation, all remote resources needed by the client (i.e. GUI and configuration files downloaded at startup as well as the remote query egine cgi) are commonly refered to as the SVMT Web Service. All elements of the SVMT Web Service are accessible under a unique root URL (e.g. http://svmt.hq.eso.org/1.0.1/).

architecture-overview.png

Query Engine API

The SVMT query engine is based on a large indexed table containing all the information relevant to the client. The table contains one row per item, each row containing a set of key/value pairs providing the information for each item. The cgi provides a JSON-based API exposing information about the content of this table. A typical query is composed of 3 steps: a filtering step where a subset of all the rows is filtered according to a given set of constraints (like the WHERE in an SQL query), a grouping step where the resulting rows are grouped according to a given set of conditions, and finally an aggregating step where information on each groups is computed. The result of the query is always a list of aggregation results, with one aggregation results set per group.

Note that because we only deal with one table, the query language can be kept much simpler than SQL (which has to deal with the complexity of joins).

Let's imagine we have to deal with a list of animals that are compiled in our table under the following structure. The table must have the following properties:

[svmt_id: "cow",        nbLegs: 4, biotop: "ground", skinType: "hair",    color: ["white", "black", "brown"]                   ]
[svmt_id: "snake",      nbLegs: 0, biotop: "ground", skinType: "scale",   color: ["yellow", "black", "brown", "green"]         ]
[svmt_id: "black bird", nbLegs: 2, biotop: "air",    skinType: "feather", color: ["black", "yellow"]                           ]
[svmt_id: "dog",        nbLegs: 4, biotop: "ground", skinType: "hair",    color: ["white", "black", "yellow", "brown"]         ]
[svmt_id: "octopus",    nbLegs: 8, biotop: "water",  skinType: "skin"                                                          ]

Don't forget that this is only a schematical representation of the table content, the real data is stored in the SVMT Web Service in a special binary format based on Tokyo Cabinet.

An example query is the following: filter animals between 0 and 5 legs living on the ground, group them all in one group, and for this group compute the number of results, the number of results per type of skin and a histogram of the number of legs. In our JSON format this can be translated into the following:

{
    "constraints": [
        {"fieldName": "nbLegs", "operation": "NUMBER_BETWEEN", "expression": [0, 5],   "negate": false}
        {"fieldName": "biotop", "operation": "STRING_EQUAL",   "expression": "ground", "negate": false}
    ],
    "groupingOptions": [
        {"fieldName": "", "operation": "GROUP_ALL", "expression": ""}
    ],
    "aggregateOptions": [
        {"fieldName": "",         "operation": "COUNT",            "expression": ""},
        {"fieldName": "skinType", "operation": "VALUE_AND_COUNT",  "expression": ""},
        {"fieldName": "nbLegs",   "operation": "NUMBER_HISTOGRAM", "expression": ""}
    ]
}

You can observe that concepts of filtering, grouping and aggregating are naturally translated into 3 JSON list of options. The constraints section is a list of contraints which should all be valid for a row to go through the filter. The groupingOptions section is a list of grouping options which all apply to discriminate groups, e.g. grouping by nbLegs and biotop will ensure that in each group will be only one number of legs and biotop value. Finally, the aggregateOptions section is a list of aggregation operations which are applied to each group, and returned in the result.

You can also observe that the type of the table "columns" is not predefined in the queryService, but are rather implicitely assumed by using the proper query operations. E.g. using a the NUMBER_BETWEEN constraint operation assumes that the value matching the corresponding fieldName (e.g. nbLegs in our example) is a number.

Appart from strings and numbers, the queryEngine also has support for spatial spherical regions. The underlying engine is based on the SphericalRegionP class from Stellarium's code. The details of the JSON serialization are explained in the documentation of the SphericalRegionP::loadFromJson method.

Please refer to the inline doc to see the detailed description of all possible ConstraintsSet::ConstraintOperation, QeQuery::GroupingOperation and QeQuery::AggregateOperation.

Sending this JSON query to the queryEngine using HTTP GET or POST (url/to/queryEngine?query=xxxx) will execute it and return the following result:

[[3, 
{
    "hair": 2,
    "scale": 1
}, 
{
    "histo": [1, 0, 0, 0, 0, 0, 0, 0, 0, 2],
    "max": 4,
    "min": 0
}
]]
svmt-logo.png
Generated on Fri Sep 10 15:36:04 2010 for SVMT by  doxygen 1.6.3