Skip to content

Tools and tips

Browsing and comparing configurations

The configuration stored in a ComponentAccumulator can be saved to a file with: 

acc = ComponentAccumulator()
...
# at the end of config file
with open("config.pkl", "wb") as f:
    acc.store(f)
or via a postExec: 
--postExec 'with open("config.pkl", "wb") as f: cfg.store(f)'
This is a binary file however it can be browsed with the confTool.py and iconfTool utilities: 
> confTool.py -p config.pkl
# or for a more interactive and visual browsing
> iconfTool -i config.pkl
The confTool.py has multiple options to constrain or expand the amount of details in the output (see -h option). For actions available in the iconfTool hit the keyword h.

Both tools can be used for comparing two configurations: 

> confTool.py --diff old-config.pkl config.pkl
The output will indicate missing as well as different settings.

For comparison with the legacy configuration the option --config-only="old-config.pkl" can be added to the athena command line. The confTool.py has several handy options to suppress the differences that are false-positives due to the differences in how the old and new configuration systems work:

  • --ignoreDefaults ignores those settings that are set in python yet are identical to the defaults from C++
  • --shortenDefaultComponents changes component names of the form AComp/AComp to AComp
  • --knownDifferencesFile loads a file with known differences
  • --renameCompsFile loads file with directives for comparing component names

Discovering component properties

Component properties are also inherited from base classes. A quickest way to discover all of them is by instantiating the python configuration class and list the class description as in the following example: 

> python
>>> from AthenaConfiguration.ComponentFactory import CompFactory
>>> help(CompFactory.HelloAlg)
This will result in the output like below with the details of all properties enlisted: 
Help on class HelloAlg in module GaudiConfig2.Configurables:

class HelloAlg(GaudiConfig2._configurables.Configurable)
 |  HelloAlg(name=None, **kwargs)
 |
 |  Properties
 |  ----------
 |  - ExtraInputs: std::unordered_set<DataObjID,DataObjID_Hasher,std::equal_to<DataObjID>,std::allocator<DataObjID> > ([])
 |     [DataHandleHolderBase<PropertyHolder<CommonMessaging<implements<IAlgorithm,IDataHandleHolder,IProperty,IStateful> > > >]
 |
 |  - ExtraOutputs: std::unordered_set<DataObjID,DataObjID_Hasher,std::equal_to<DataObjID>,std::allocator<DataObjID> > ([])
 |     [DataHandleHolderBase<PropertyHolder<CommonMessaging<implements<IAlgorithm,IDataHandleHolder,IProperty,IStateful> > > >]
 |
 |  - OutputLevel: int (0)
 |    output level [Gaudi::Algorithm]
 |
 |  - Enable: bool (True)
 |    should the algorithm be executed or not [Gaudi::Algorithm]
 |
 |  - ErrorMax: unsigned int (1)
 |    [[deprecated]] max number of errors [Gaudi::Algorithm]
 .....

Debugging the configuration process

Sometimes it is necessary to interrupt the configuration process and be able to enter interactive mode somewhere deep in the call stack. The code python module can be used for this.

In place where you intend the configuration process to be interrupted add the following: 

import code; code.interact(local=locals())
With this the configuration process will be interrupted and control given to you. You can inspect local scope variables as needed. Use Ctrl-D to exit the debugging session and resume the configuration process.

Another option could be to invoke the python debugger pdb in the location where you want to start debugging: 

import pdb; pdb.set_trace()

Finding the source of merge conflicts

When merging CA instances, conflicts can arise but it is not always evident where the origin of the conflict is. To track the context of where each CA or component was created, set: 

ComponentAccumulator.debugMode = "trackCA track[EventAlgo|CondAlgo|PublicTool|PrivateTool|Service|Sequence] ..."
Since context collection slows down the configuration process, it can be enabled for specific component types only. With this, the merge conflict exception is extended by information as to where the conflicting components are defined: 
Traceback (most recent call last):
 File "/srv/build/x86_64-centos7-gcc8-opt/python/AthenaConfiguration/ComponentAccumulatorTest.py", line 98, in test_conflict_in_cond_alg
    acc.addCondAlgo(ExampleAlg1("Cond1", MyInt=8))
.....
ValueError: conflicting settings for property MyInt of ExampleAlg1: cannot merge values 8 and 7
with the context
 Adding Conditions Algorithm to the ComponentAccumulator that has instance of it already added in this context
 >>> runpy.py:193(_run_module_as_main) case.py:624(run) ComponentAccumulatorTest.py:57(setUp)
The last line contains an abbreviated sequence of calls that lead to the creation of the component conflicting with the one attempted to be added.

Profiling and optimising configuration

The configuration process can take a significant amount of time and it can be useful to profile it to find the bottlenecks.

The cProfile module can be used for this. The following example shows how to profile the configuration process of MuonCombinedReconstructionConfig: 

python -m cProfile -o profile -m MuonCombinedConfig.MuonCombinedReconstructionConfig --threads=1

If you want to profile a more complex transform, the easiest approach is probably to modify the runargs 

# Import skeleton and execute it
from RecJobTransforms.RAWtoALL_Skeleton import fromRunArgs
import cProfile
cProfile.run('fromRunArgs(runArgs)', filename='RAWtoALL.prof')

The profiling output can be analysed using the pstats module or graphically with the snakeviz tool.

Once you have identified the bottlenecks, you can try to optimise the configuration process. For example, an AccumulatorCache class can be used to cache the results of the configuration functions (this was discussed earlier in "Caching of configuration results"). See atlas/athena!57202 for an example of a MR showing some snakeviz screenshots and optimisations using AccumulatorCache.

Setting properties while debugging

During development or debugging, it might be useful to set individual component properties without having to modify the CA configuration functions themselves. This can be achieved using the foreach_component method of the ComponentAccumulator instance. To ensure such code does not enter production, an ERROR message is printed when this feature is used.

# Example: setting property for all extrapolators
ca.foreach_component("*/Extrapolator/*").UseMuonMatApproximation = False
# Example: setting property for all HLT hypothesis algorithms
ca.foreach_component("*/HLT/*/*").RuntimeValidation = False
The path to the component is constructed as a UNIX-like path using sequence names, class name and instance name, e.g. the path to MyTool might be: 
AthMasterSeq/AthAlgEvtSeq/AthAllAlgSeq/AthAlgSeq/MyAlgClass/MyAlg/MyToolClass/MyTool
In detail, the following convention for paths of nested components is used: 
Sequence     : only the name is used in the path
Algorithm    : "type/name" is used
Private Tool : ToolHandle property name plus "type/name" is used
Public Tool  : located under "ToolSvc/" and "type/name" is used
Service      : located under "SvcMgr/" and "type/name" is used

The match is performed using the fnmatch syntax, e.g.: 

'*/ToolInstance'      : all tools that have matching instance name
'*/MyTool/*'          : all instances of type MyTool
'*/MyAlgo/MyInstance' : specific algorithm instance
'*/MyAlgo/*'          : all instances of the specific algorithm class
'*/AthAlgSeq/*'       : all algorithms of the given sequence
'ToolSvc/My*/*'       : all public tools with instance name starting with "My"
To see which components were matched, search for the PropSetterProxy

Modifying the OutputLevel

For debugging, the OutputLevel of individual components can be set on the command line using the Exec.[LEVEL]MessageComponents flags. The flag value can be:

  • a (list of) string(s) using the same component wildcard syntax of foreach_component introduced above
  • a plain component name (e.g. "MyAlg")

For example, to enable DEBUG output for all instances of MyAlgClass, use: 

athena.py ... Exec.DebugMessageComponents="*/MyAlgClass/*"

Tip

For this to work the main driver script needs to call AthenaConfiguration.Utils.setupLoggingLevels before executing the CA. This should be the case for all "official" workflows.