Accueil Accueil

LSST school workshop: Getting ready to do science with LSST data
Exploring Spark and MongoDb for LSST (9016) - Jeudi 15 juin 2017 12:20 - 12:50

Exploring Spark and MongoDb for LSST

Spark is a very promising technology offering distributed data and computing mechanisms.

At LAL(Orsay) we have started to look at how the typical computing workflows used in LSST could use the Spark eco-system:

How to distribute algorithms in a map-reduce approach How to format various data structures to partition them in a distributed file system Thus, a OpenStack based cluster has been configured at LAL with Spark and its various associated components, and several models are experienced to evaluate the performance and configuration parameters (memory, CPU, …)

In the same context, in the process in exploring various technologies related with QServ or the catalog access techniques, we are working on two promissing technologies: MongoDB and Spark DataFrames, both offering a natural data or processing distribution approach.

The method is similar for both: we exploit one limited dataset (2To) (sources and objects) and try and apply the benchmarking queries that used to be applied to QServ.

The concepts, the ingestion, and the querying methods are explored, in particular looking at possible functional or performance limitations for both systems.

Several platforms are used for this study:

The Galactica cluster at Clermont (Petasky context) OpenStack at LAL (VirtualData context) A test cluster CCIN2P3.

Mr. Christian ARNAULT (CNRS)

Toutes les vidéos de l'évenement LSST school workshop: Getting ready to do science with LSST data (35 videos)

00:57:52.63

2017-06-12

09:05 - 10:30 : Point sources and astrometry

00:19:33.76

2017-06-12

11:00 - 11:30 : Point sources and astrometry (cont.)

00:53:58.55

2017-06-12

11:30 - 12:30 : Extended sources and galaxy photometry

00:42:27.54

2017-06-12

14:00 - 15:00 : Extended sources and galaxy photometry (cont.)

00:40:20.86

2017-06-12

15:00 - 15:30 : Image differencing

01:21:04.24

2017-06-12

16:00 - 17:30 : Image differencing (cont.)

00:47:16.06

2017-06-13

09:20 - 10:00 : PSF parameter estimation

00:39:50.93

2017-06-13

09:20 - 10:00 : Efficient multi-band deblending

00:36:08.77

2017-06-13

11:10 - 11:50 : LSST DM Processing of Crowded Fields

00:24:52.80

2017-06-13

11:50 - 12:15 : Improved Point Source Detection in Crowded Fields using Probabili...

00:35:12.33

2017-06-13

14:00 - 14:45 : Hyper Suprime-Cam Subaru Strategic Program

00:47:18.80

2017-06-13

14:45 - 15:30 : HSC data meets LSST code

00:30:32.70

2017-06-13

16:00 - 16:40 : Photometry extraction and validation using the HSC pipeline of HS...

00:49:15.71

2017-06-13

16:40 - 17:20 : Reprocessing CFHT data with the LSST DM software stack

00:39:19.51

2017-06-13

17:20 - 18:00 : Reprocessing CFHT Deep fields with the stack

00:33:20.11

2017-06-14

09:00 - 09:30 : Joint astrometry

00:29:58.54

2017-06-14

09:30 - 10:00 : Creating DCR-matched templates for image differencing

00:32:01.48

2017-06-14

10:00 - 10:30 : Advances in astronomical image processing - Solving the problems ...

00:32:11.88

2017-06-14

11:00 - 11:30 : LSST DM stack image differencing on CFHT images

00:34:11.44

2017-06-14

14:00 - 14:35 : Dark Energy Survey - Status, Science, and Algorithmic Advancement...

00:32:37.68

2017-06-14

14:35 - 15:10 : Forward Global Calibration of the Dark Energy Survey [remote pres...

00:36:19.72

2017-06-14

15:10 - 15:45 : Photometric Calibration: lessons from CFHTLS

00:24:44.83

2017-06-14

16:10 - 16:40 : PHOTOMETRYPIPELINE: An automated pipeline for calibrated photomet...

00:27:32.14

2017-06-15

09:00 - 09:30 : LSST Data Management Overview

00:41:45.78

2017-06-15

09:30 - 10:10 : What LSST Will Deliver: Images, Catalogs, Alerts, Services

00:33:17.41

2017-06-15

10:10 - 10:40 : Qserv: A distributed shared-nothing database for the LSST catalog

00:31:42.47

2017-06-15

11:10 - 11:40 : Scaling cloud for LSST catalog at IN2P3

00:44:12.37

2017-06-15

11:40 - 12:20 : LHC experiments : Petabytes to papers

00:34:58.27

2017-06-15

12:20 - 12:50 : Exploring Spark and MongoDb for LSST

00:40:06.94

2017-06-15

14:00 - 14:40 : Euclid - An update on the mission and its processing plan

00:32:55.31

2017-06-15

14:40 - 15:20 : The Dark Energy Survey Data Management System [remote presentatio...

00:33:43.47

2017-06-15

15:50 - 16:20 : Weak Lensing Data processing for the CFHTLenS and KiDS surveys [r...

00:35:11.10

2017-06-15

16:20 - 17:00 : Lessons from a Large Survey: The First Decade of Pan-STARRS Obser...

00:16:24.21

2017-06-15

17:00 - 17:10 : CosmoHub demonstration [spontaneous contribution]

00:04:11.44

2017-06-15

17:10 - 17:20 : Workshop: Wrap up

Contact Webcast

La cellule webcast du CCIN2P3 vous propose de diffuser en direct et/ou en différé sur internet vos manifestations, colloques, conférences. Attention, ce service est réservé au domaine public dans le domaine de la Recherche Scientifique.


Cellule Webcast
Centre de Calcul IN2P3/CNRS
21 Avenue Pierre de Coubertin
CS70202
69627 VILLEURBANNE Cedex

Tél. :
+33(0) 4.78.93.08.80

Fax. :
+33(0) 4.72.69.41.70



Email :

Voir les mentions légales du site