DAME Project - Astrophysics with SCoPE
Modern scientific data mainly consist of huge datasets that are gathered by a very large number of techniques and stored in very diversified and often incompatible data repositories. More in general, in the e-science environment, it is considered as a critical and urgent requirement to integrate services across distributed, heterogeneous, dynamic "virtual organizations" formed by different resources within a single enterprise and/or external resource sharing and service provider relationships.
The Astronomy and Astrophysics environment has become an immensely data rich field due basically to:
- Detector evolution
(plates to digital to mosaics)
- Telescope evolution
- Space instruments
One first and important step towards the definition of a Virtual Organization has already been undertaken within the Astrophysics community with a set of initiatives that have flourished under the generic name of Virtual Observatory (VObs). The VObs, organized worldwide by means of the International Virtual Observatory Alliance (IVOA), has defined a set of standards to allow interoperability among different archives and databases in the astrophysical domain, and keeps them updated through the activity of dedicated working groups. One of the first main goals of the IVOA is the federation under common standards of all astronomical archives available worldwide. The concept is that having this meta-archive completed, its exploitation allows a new type of multi-wavelength, multi-epoch science which can only be barely imagined, but also poses unprecedented computing problems. So far, up to now most of the implementation effort for the VObs has concerned the storage, standardization and interoperability of the data together with the computational infrastructures. In particular it has focused on the realization of the low-level tools and on the definition of standards.
The astrophysical project inserted into the domain of SCoPE, (DAME, DAta Mining & Exploration), extends this fundamental target, by integrating it in a service oriented infrastructure, including the implementation of advanced tools for Massive Data Sets (MDS) exploration, soft computing, data mining (DM) and Knowledge Discovery in Databases (KDD).
The DAME project, run jointly by the Department of Physics of the University Federico II, INAF (National Institute of Astrophysics) Astronomical Observatory of Napoli, and the California Institute of Technology, aims at creating a single, sustainable, distributed e-infrastructure for data mining and exploration in massive data sets, to be offered to the astronomical (but not only) community as a web application. The framework makes use of distributed computing environments (e.g. S.Co.P.E. infrastructure) and matches the international IVOA standards and requirements. The integration process is technically challenging due to the need of achieving a specific quality of service when running on top of different native platforms. In these terms, the result of the DAME project effort will be a service-oriented architecture, obtained by using appropriate standards and incorporating GRID paradigms and restful Web services frameworks where needed, that will have as main target the integration of interdisciplinary distributed systems within and across organizational domains.
Currently, the DAME project, already completed in terms of the engineering design aspects, is under implementation of the first official version of the complete data mining & processing tool package, deployed on SCoPE GRID Infrastructure. But a prototype is already available (please visit project website) and succesfully tested on significative science cases, as well demonstrated by related scientific production.
Links