Fast online reconstruction and online calibration in the ALICE High Level Trigger

ALICE (A Large Heavy Ion Experiment) is one of four major experiments at the Large Hadron Collider (LHC) at CERN. The ALICE High Level Trigger (HLT) is a cluster of 200 nodes, which reconstructs collisions as recorded by the ALICE detector in real-time. It employs a custom online data-transport framework to distribute data and workload among the compute nodes. ALICE employs subdetectors sensitive to environmental conditions such as pressure and temperature, e.g. the Time Projection Chamber (TPC). A precise reconstruction of particle trajectories requires the calibration of these detectors. Performing the calibration in real time in the HLT improves the online reconstructions and renders certain offline calibration steps obsolete speeding up offline physics analysis. For LHC Run 3, starting in 2020 when data reduction will rely on reconstructed data, online calibration becomes a necessity. Reconstructed particle trajectories build the basis for the calibration making a fast online-tracking mandatory. The main detectors used for this purpose are the TPC and ITS. Reconstructing the trajectories in the TPC is the most compute-intense step. We present several components of the ALICE High Level Trigger used for fast event reconstruction and then focus on newly developed components for online calibration. The TPC tracker employs GPUs to speed up the processing and is based on a Cellular Automaton and the Kalman filter. It has been used successfully in proton-proton, lead-lead, and proton-lead runs between 2011 and 2015. We have implemented a wrapper to run ALICE offline analysis and calibration software inside the HLT. Normally, the HLT works in an event-synchronous mode. We have added asynchronous processing capabilities to support long-running calibration tasks. In order to improve the resiliency, an isolated process performs the asynchronous operations such that even a fatal error does not disturb data taking. We have complemented the original loop-free HLT chain with ZeroMQ data-transfer components. The ZeroMQ components facilitate a feedback loop, that after a short delay inserts the calibration result created at the end of the chain back into tracking components at the beginning of the chain. On top of that, these components are used to ship QA histograms to the Data Quality Monitoring (DQM) and to obtain information of pressure and temperature sensors needed for calibration. All these new features are implemented in a general way, such that they have use-cases aside from online calibration. In order to gather sufficient statistics for the calibration, the asynchronous calibration component must process enough events per time interval. Since the calibration is only valid for a certain time period, the delay until the feedback loop provides updated calibration data must not be too long. A first full-scale test of the online calibration functionality was performed during the 2015 heavy-ion run under real conditions. We present a timing analysis of this first online-calibration test, which indicates that the HLT is capable of online TPC drift time calibration fast enough to calibrate the tracking via the feedback loop.