Introduction

Scope

This document describes in details the EDPS dashboard (Graphic User Interface), the recommended infrastructure to reduce data from the ESO telescope, using the new ESO Data Processing System (EDPS). Links to tutorials for general or instrument specific reduction with EDPS can be found in Section target-info

What is EDPS?

The ESO Data Processing System (EDPS) is a framework to run ESO’s data processing pipelines and it is meant to eventually replace the previous EsoReflex environment. The general principles of EDPS have been described by Freudling, Zampieri, Coccato et al. 2024, A&A, 681, A93. Please refer to that paper if you have used EDPS for research resulting in a scientific publication.

Each of ESO’s data processing pipeline consist of a series of standalone programs called “recipes”. Each recipe is designed to process certain type(s) of input data. The processing of these input data typically requires a range of auxiliary files such as calibration files. EDPS is designed to select appropriate input data for the different recipes of a pipeline, and execute them in sequence. This is done by specifying for each pipeline the workflow for organizing data and executing the recipes. This workflow can the used to process a set of data fully automatically.

Main concepts

EDPS is an environment designed to execute the recipes of an instrument pipeline according to a series of instructions. The main concepts in EDPS are:

  • Workflow and reduction cascades. A workflow is a series of instructions designed to reduce data with an instrument pipeline in potentially multiple ways, by carrying on a sequence of tasks. Each workflow can define multiple reduction cascades, depending on the scientific needs. Foe example, the same workflow can be used to process data following different strategies that trigger different reduction steps (e.g. in one strategy flux calibration can be omitted) or different end-points (e.g., combine different science exposures, or stop after the reduction of individual exposures without combining them). Each of these “strategies” defines a “reduction cascade”.

  • Task, jobs, and recipes. A task is an element in the workflow that performs a given step of the data reduction cascade. Tasks are often associated to a recipe of the underlying instrument pipeline. A jobs is a work unit in a processing environment, that runs a recipe on a set of input data with a set of recipe parameters. A single task can generate several jobs: for example, a “bias” task, can generate multiple jobs, each of the running the bias recipe on a different set of input files.

  • Dataset. A dataset is a collection of files, that are needed to perform the data reduction as specified by the workflow. It consists, for example, of one or more science files plus the calibrations needed to process them. In EDPS, the dataset has a hierarchical structure, which highlights the connections between the various files and tasks (e.g., task A is an input to task B).

  • Target and Target category. The “target”, or the “target task” is the end point of the reduction cascade. When specifying a target, EDPS will process all and only the files needed to execute it. For example, if my target is “science”, and the science files need the bias files, EDPS will process only the biases that have been selected to process those science files; then it processes the science using the product of the bias reduction. However, if my target is bias, then EDPS will process all and only the bias files, regardless they are not used by any science. In this case, EDPS does not processes the science, as it has reached already the end reduction point (e.g., process all biases). The “Target category” is a group of target that have similar purposes. For example, the target category “science”, includes all the tasks that deliver final scientific products, the target category “qc1calib” includes all and only the tasks that processes calibrations (e.g., bias, standard stars).

Go to edps-gui documentation index