Trigger EDM

The Trigger Event Data Model (EDM) defines the trigger reconstructed content that is written out to RAW files and data or MC AODs. The Trigger EDM content for Run 3 is defined in TrigEDMConfig.TriggerEDMRun3 (the same folder contains similar files for other periods). Each EDM entry is a tuple containing all EDM details for a specific interface or Aux container. An example of two (interface + aux) EDM entries:

('xAOD::CaloClusterContainer#HLT_CaloEMClusters_Electron',               'BS ESD AODFULL AODSLIM', 'Egamma', [InViews('precisionCaloElectronViews')]),
('xAOD::CaloClusterTrigAuxContainer#HLT_CaloEMClusters_ElectronAux.',    'BS ESD AODFULL AODSLIM', 'Egamma'),

('xAOD::CaloClusterContainer#HLT_TopoCaloClustersLC',                             'BS ESD AODFULL', 'Tau', [InViews('tauCaloMVAViews'), allowTruncation]),   
('xAOD::CaloClusterTrigAuxContainer#HLT_TopoCaloClustersLCAux.nCells.CENTER_MAG', 'BS ESD AODFULL', 'Tau', [allowTruncation]),

Every EDM entry will at minimum define the container type and container name, the "EDM targets" and trigger signature. Any additional dynamic Aux variables to be recorded are specified as part of the Aux container name string. The EDM targets define the output type to which the containers will be recorded in the case of RAW, RDO_TRIG, ESD and AOD files, as well as PHYSVAL DAOD files. The ESD level is used for MC RDO_TRIG (equivalent to data RAW files for MC) production. We use these to restrict the HLT content when not all HLT content is needed, e.g. in MC AODs for physics analysis derivations. The trigger signature is used to categorise total EDM size by signature in the Trigger EDM monitoring.

EDM targets

The EDM target labels (2nd entry in an EDM entry) are mapped to the following output:

Label	Output	Purpose
BS	ByteStream - Any content that is to make it out of Point 1 and be recorded to RAW file.	Required for recording to data AOD after or kept in BS for debugging in case of data taking issues.
ESD	ESD files, RDO_TRIG files, "Sample A" productions	Low-level reconstructed information such as HLT calorimeter clusters for specialised reconstruction studies. Also used for RDO_TRIG (MC equivalent of a RAW file) output.
AODFULL	data AOD at Tier0, trigger developer AOD production requests, HLT Reprocessing AODs.	Required for regular trigger efficiency studies in data or validation during commissioning.
AODSLIM	bulk MC production AODs for physics analysis, "Sample T" productions.	Required for physics analysis. Examples: Final precision HLT leptons/b-jets needed for trigger matching and trigger scale factors.
PhysicsTLA/EgammaPEBTLA/DarkJetPEBTLA/FTagPEBTLA	respective RAW output of a Trigger-Level Analysis (TLA) or TLA Partial Event Building stream	Content required for a trigger-level physics analysis or TLA with partial event building (PEB) information.

Important: Think hard about what targets you need, as the HLT content in total has been taking up a significant fraction of disk space.

Valid reasons to keep trigger content in AODFULL: If needed for regular trigger efficiency checks in recorded data AOD or if it is a new implementation that needs a few months of commissioning.

Valid reasons to keep trigger content in AODSLIM: If needed for physics analysis, namely trigger matching and scale factor derivations.

Note on AODSLIM: Further reductions

One can specify Run 3 container dynamic variables that are not needed in AODSLIM, i.e. variables that are not needed by physics analyses. This is set via the varToRemoveFromAODSLIM list in TrigEDMConfig.TriggerEDMRun3.

Optional EDM details

There are several additional EDM details that can be specified as a list constituting the last entry in the tuple.

InViews

In the case of reconstruction in a limited region of the detector, this names the event view in which this collection is reconstructed, so that additional information allowing the items to be accessed in event views is recorded. For more details, see the section on Event Views.

allowTruncation

This flag is only valid for data taking and is used to indicate that this container can be dropped from an event if the event data is larger than some threshold that would have resulted in the event data going to the debug stream. A developer need not worry about setting this EDM detail. However, for technical reasons, "allowTruncation" entries need to appear at the end of the EDM list. Therefore, a developer should add any new EDM content before the last "allowTruncation" entries and, more specifically, within the relevant signature section.

Aliases

A name with suffix ShallowCopy is set for all ShallowAuxContainer type containers to indicate that they need special treatment by the HLTEDMCreator given that the Aux store name differs from the interface name.

unit tests

Unit tests exist that check the validity, ordering and manipulation of EDM entries.

EDM manipulation

The following configuration flag setting will override the "output level" for any AOD output file:

flags.Trigger.AODEDMSet='ESD'

i.e. this will record all ESD labelled containers to the output AOD file.

The following configuration flag settings will override or add a new EDM entry if not already found:

flags.Trigger.ExtraEDMList = [
    ('xAOD::CaloClusterContainer#HLT_TopoCaloClustersFS',                  'BS ESD AODFULL', 'Calo'),
    ('xAOD::CaloClusterTrigAuxContainer#HLT_TopoCaloClustersFSAux.nCells', 'BS ESD AODFULL', 'Calo'),
    ('xAOD::ElectronContainer#HLT_NewParticles',                                'BS ESD AODFULL', 'EGamma', [InViews('NewViews')]),
    ('xAOD::ElectronAuxContainer#HLT_NewParticlesAux.',                'BS ESD AODFULL', 'EGamma'),
]

This command will update the targets for the already existing HLT_TopoCaloClustersFS and HLT_TopoCaloClustersFSAux containers, as well as add new HLT_NewParticles and HLT_NewParticlesAux containers.

Procedure for modifying trigger EDM

Adding content

The following steps should be followed if you would like to add new trigger EDM content:

1. Do the necessary software modifications to record the new content as described in section Reconstruction Algorithms: Saving EDM objects.
2. Test and validate: Check content is recorded in the expected output (data AOD, MC AOD etc) and the content is correct. See the subsection below!
3. Estimate the size increase. See the subsection below!
4. Contact the trigger EDM coordinator found on the trigger Core s/w twiki, providing:
   1. (likely already existing) JIRA ticket motivating the new implementation, in which is posted the motivation for the new EDM content and EDM target and the results of the 2 previous steps.
   2. (if ready) a link to the draft Merge Request.
5. The trigger EDM coordinator will confirm whether the EDM targets are appropriate and sufficiently motivated.
6. Submit MR/mark MR as ready
   1. If an MR changes the EDM during a data taking period (e.g. Run 3) to the current Tier0 branch (24.0), the MR will require "PROC" sign-off (Prompt Reconstruction Operation Coordinator) due to the Frozen Tier0 Policy which tries to prevent EDM changes during data taking periods or major MC campaigns -- see details on the FrozenTier0 twiki. There will be failing CI tests due to this policy, namely CITest_RecoRun3Data_Checks-test and CITest_RecoRun3MC-test:

      ERROR    Py:diff-root        ERROR files differ!
      ERROR    Your change breaks the frozen tier0 policy in test q449.
      ERROR    Please make sure this has been discussed in the correct meeting (RIG or Simulation) meeting and approved by the relevant experts.
      

      along with details of the diff. Double-check that the differences are expected and tag the current PROC in your MR, providing the JIRA link with motivation (see ATLAS Data Preparation organisation page or twiki for current coordinators). The MR will only be merged once the AOD/ESD references have been updated with your changes by PROC (once all tests are green).
7. When requested to give a presentation of your new implementation at the Trigger General Meeting, provide 1-2 slides motivating the trigger EDM changes and size estimates.

Note

Notifying the Trigger EDM coordinator is very important. Besides reviewing size increases, the trigger EDM coordinator may also need to, post-MR, update the bytestream schema, so that new container types can always be serialised, as well as ensure that Athena releases for P1/Tier0 and for GlobalMonitoring are synchronised!

Removing content

If you would like to remove trigger EDM content, then get in touch with the trigger EDM coordinator directly, so they can make sure that the content is not used by anyone. Note A trigger EDM entry should never be completely removed; the EDM targets should be removed instead.

Test and validate

The following should be cross-checked for new EDM content:

1. New EDM containers appear in RAW or relevant AOD output.
2. New EDM containers are populated by your trigger reconstruction (Note: Containers will be filled with the online reconstruction for testing your trigger hypothesis, even if your trigger didn't pass).
3. Dynamic values can be accessed and make sense: Are not default values, etc.
4. The size increase is as expected and not exorbitant.

The results should be documented in JIRA.

Useful trigger tests

The following tests will output AODs after rerunning the HLT on a given menu:

test	data/MC	EDM target	Menu
test_trigAna_RDOtoAOD_v1MC_build	ttbar MC	AODSLIM	"Physics_pp_run3"
test_trigAna_RDOtoAOD_v1Dev_build	ttbar MC	AODFULL	"Dev_pp_run3"
test_trigP1_v1PhysP1_T0Mon_build	data	AODFULL	"Physics_pp_run3"

Each test will output AOD.pool.root.checkFile and AOD.pool.root.checkFileTrigSize files, produced from checkFile.py AOD.pool.root and checkFileTrigSize.py AOD.pool.root, respectively: checkFile output provides sizes per container element, checkFileTrigSize provides EDM sizes for the total trigger EDM as well as individual trigger signatures.

Tips on running own trigger tests

When you run tests like test_trigAna_RDOtoAOD_v1MC_build.py or test_trigP1_v1PhysP1_T0Mon_build.py, files are automatically created containing the executables: commands.json, as well as runargs.RAWtoALL.py for the latter. Alternatively, you copy and edit the test directly. You can use this output as a reference for running your own trigger test with modified settings (different menu, number of events, EDM target etc).

Cross-check: Container recording

Your container should appear in AOD.pool.root.checkFile for a relevant ("AODFULL", "AODSLIM" ...) AOD file.

Warning

All EDM containers listed in TriggerEDMRunX will appear in the relevant output specified by the EDM target, regardless if they were reconstructed at the HLT: We run a "GapFiller" each time we produce AODs, which creates empty HLT containers for an event if they don't already exist to ensure a consistent EDM throughout all events. Thus, seeing a container listed doesn't mean it's populated!

Cross-check: Container content

You should verify that new containers are being filled and that they are populated by your chains. A very useful cross-check is to run a trigger test on a limited menu containing your chains only by setting flags.Trigger.selectedChains=["my_new_chain"]. This ensures that all reconstruction and hence container content is the result of your chains (especially relevant if you are adding new chains that write to already existing containers). Ideally, you run the trigger task on an input MC RDO file containing a good fraction of events that pass your triggers. The ttbar MC RDO used in test_trigAna_RDOtoAOD_v1MC_build.py may be sufficient.

Use an analysis script to retrieve and plot EDM variables to histogram or open the AOD or RDO_TRIG file in interactive ROOT or ROOT browser to check that variable distributions look correct.

Estimate size increase

The size increase that primarily matters to us are for:

• test_trigAna_RDOtoAOD_v1MC_build AOD output ("AODSLIM" content), run on the MC menu
• test_trigP1_v1PhysP1_T0Mon_build RAW and AOD output ("AODFULL" content), run on the Physics menu.

It is useful to have the size increase quoted as percentage relative to: total file size, total trigger EDM size (from checkFileTrigSize) and relevant trigger signature size (from checkFileTrigSize).

One can measure the size increase by:

• Comparing output file sizes (e.g. AOD.pool.root, *.physics_Main*._athenaHLT*.data) between release 24.0 and development branch with your trigger chains included in the physics menu.
• Comparing numbers printed in checkFile and checkFileTrigSize output: e.g. AOD.pool.root.checkFile provides total file size + size for individual EDM elements, AOD.pool.root.checkFileTrigSize provides sizes for total trigger EDM and for individual signatures.

Example terminal output comparing total file, trigger EDM and individual trigger signature sizes:

du 24.0/AOD.pool.root
4554    24.0/AOD.pool.root
du my_dev_branch/AOD.pool.root
4573    my_dev_branch/AOD.pool.root
...
head 24.0/AOD.pool.root.checkFile
PyRoot: Styles set by rootlogin.C
## opening file [AOD.pool.root]...
## importing ROOT...
## importing ROOT... [DONE]
## opening file [OK]
File:AOD.pool.root
Size:     4553.341 kb
Nbr Events: 20

head my_dev_branch/AOD.pool.root.checkFile
PyRoot: Styles set by rootlogin.C
## opening file [AOD.pool.root]...
## importing ROOT...
## importing ROOT... [DONE]
## opening file [OK]
File:AOD.pool.root
Size:     4572.145 kb
Nbr Events: 20

...
grep triggerTau 24.0/AOD.pool.root.checkFileTrigSize
triggerTau                       3.184
grep triggerTau my_dev_branch/AOD.pool.root.checkFileTrigSize
triggerTau                       4.067
...
grep "Total Trigger Size" 24.0/AOD.pool.root.checkFileTrigSize
Total Trigger Size      51.414
grep "Total Trigger Size" my_dev_branch/AOD.pool.root.checkFileTrigSize
Total Trigger Size      52.311

Example terminal output looking at the "Size/Evt" of individual EDM additions in AOD.pool.root.checkFileand comparing to other similar containers - the "Size/Evt" for other containers can be retrieved from the "checkFile" output if run on the full MC/Physics menu or from the trigger EDM size monitoring webpage for the MC and PhysP1 tests mentioned above:

================================================================================
     Mem Size       Disk Size        Size/Evt      MissZip/Mem  items  (X) Container Name (X=Tree|Branch)
================================================================================
grep HLT_TrigTauRecMerged_MVA my_dev_branch/AOD.pool.root.checkFile
       1.049 kb        0.256 kb        0.013 kb        0.000       20  (B) HLT_TrigTauRecMerged_MVA
       1.121 kb        0.316 kb        0.016 kb        0.000       20  (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_Loose
       1.121 kb        0.316 kb        0.016 kb        0.000       20  (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_Tight
       1.126 kb        0.317 kb        0.016 kb        0.000       20  (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_Medium
       1.141 kb        0.324 kb        0.016 kb        0.000       20  (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_VeryLoose
       1.939 kb        0.338 kb        0.017 kb        0.000       20  (B) HLT_TrigTauRecMerged_MVAAuxDyn.viewIndex
       1.426 kb        0.480 kb        0.024 kb        0.000       20  (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_Score
       1.465 kb        0.484 kb        0.024 kb        0.000       20  (B) HLT_TrigTauRecMerged_MVAAuxDyn.GNTau_ScoreSigTrans
     144.458 kb       20.145 kb        1.007 kb        0.000       20  (B) HLT_TrigTauRecMerged_MVAAux.
grep HLT_TrigTauRecMerged_LLP my_dev_branch/AOD.pool.root.checkFile
       1.049 kb        0.256 kb        0.013 kb        0.000       20  (B) HLT_TrigTauRecMerged_LLP
       1.541 kb        0.350 kb        0.017 kb        0.000       20  (B) HLT_TrigTauRecMerged_LLPAuxDyn.viewIndex
      88.093 kb       12.688 kb        0.634 kb        0.000       20  (B) HLT_TrigTauRecMerged_LLPAux.
grep HLT_TrigTauRecMerged_MVA my_dev_branch/AOD.pool.root.checkFile | awk '{SIZE=$5;SUM+=SIZE}END{print SUM}'
1.149
grep HLT_TrigTauRecMerged_LLP my_dev_branch/AOD.pool.root.checkFile | awk '{SIZE=$5;SUM+=SIZE}END{print SUM}'
0.664
grep triggerTau my_dev_branch/AOD.pool.root.checkFileTrigSize
triggerTau                       4.067

Note

It can be difficult obtaining a good size estimate if your triggers have a low selection rate. However, once your EDM changes are merged to the official Athena branch, ART (nightly) tests monitor the trigger EDM size on a larger amount (~1000) of MC & data events, which will give us a more accurate idea of the size increase. It is good to reference these numbers in your JIRA ticket once available (absolute "Size/Evt" values as well as relative size increase). See the section on trigger EDM size monitoring on obtaining these size estimates.