Skip to content

Frequently Asked Questions

Wouldn't XYZ be a much cleaner columnar design?

Probably, maybe. If you made a complete greenfield design for purely columnar tools, then you would likely choose a different design. Actually you hopefully wouldn't develop something new, but use an existing design like correctionlib.

However, this is not a greenfield design, we are migrating the existing CP tools that implement finely tuned CP recommendations to run on the already existing PHYSLITE files. None of that was developed with columnar analysis in mind. And while we need updates to all of them, we want to minimize the (mandatory) effort needed for migration.

There is also the need (or at least desire) for these tools to integrate well with the xAOD environment. The columnar infrastructure largely achieves that, keeping a similar syntax in many cases, working in a largely xAOD native way in xAOD mode, and allowing xAOD only code paths within an otherwise columnar tool.

Another important constraint is that we have a very complex input file format, in particular the various element links that can point to multiple target containers. The columnar infrastructure can handle and hide the corresponding complexity fairly well. It's not something that is common in native columnar design, and a number of the proposed alternatives did not address it at all.

That is not to say that individual tools and data structures couldn't be made more columnar, or that that wouldn't be a good thing. It is simply not a requirement with the current design.

This all feels really complicated. Shouldn't columnar code automatically become incredible simple?

Short answer: No, if you do something complicated it will maybe become somewhat simpler, but generally still be complicated.

In particular we have a couple of drivers of complexity:

  • We want the code to run and integrate well in multiple environments: EventLoop/Athena, uproot, and RDataFrame
  • The PHYSLITE data format is non-trivial, including variant links, data that needs special unpacking, etc.
  • The CP recommendations do some non-trivial things, that need adequate support.
  • We try to minimize the need to rewrite existing CP Tools.

While there may be ways to simplify some of the aspects of the columnar infrastructure, in many cases that would either move the complexity elsewhere (as opposed to eliminating it) or would require disproportionate changes to the CP Tools. There is the hope that eventually we can simplify CP recommendations and PHYSLITE, but we don't want to require that upfront.

That is not to say that we don't have simplifications compared to the xAOD infrastructure:

  • The columnar infrastructure doesn't have an intrinsic memory model, allowing to offload that to the xAODs, the columnar framework, etc.
  • A lot of the special features are not intrinsic to the core of the columnar infrastructure, but more add-on and orthogonal features. So while there are occasionally some special hooks or limitations from those features, their added complexity mostly affects their users and could be removed if it is no longer used.

Couldn't we implement columnar tools as a regular xAOD tool with a custom aux store?

The idea sounds simple enough: Since the aux store internally represents each decoration as a simple data column, can't we just use a custom aux store implementation that redirects to the input columns and stick with the existing xAOD tools.

One big constraint is performance: We try for columnar tools to be orders of magnitude faster than the existing xAOD-based tools. And switching from xAOD-based operations to columnar mode often already buys 1-2 orders of magnitude. Not in all cases, but the more we speed up other aspects of the tools the more important this will become.

Another big constraint is memory layout: While for some columns the memory layout is the same as for the xAOD aux store, for everything except simple POD types (int, float, etc.) it is not and would have to be reformatted to be used within a custom aux store, putting a further drag on performance.

There are also a lot of minor things that tools can do that don't work well with such a model, and would have to be changed, or we would have to essentially build an entire event store from the input columns.

There is also the whole issue of data dependencies: For columnar environments you will typically need column level data dependencies, which xAOD code doesn't provide. While the dependency lists can be maintained manually, they also often depend on the exact properties set on the tool.

Why is columnar mode so much faster than xAOD mode?

Short answer: We don't know for sure. We can guess, and it is generally consistent with other profiling and benchmarking tests.

The xAOD code is generally doing fairly lightweight things and is carefully tuned by experienced developers, so we expect it to be fast. At the same time, the alternative to an aux-store lookup in xAOD mode is a simple array lookup in columnar mode that can be fully inlined and optimized by the compiler. And most CP tools are computationally light, they look up a number of variables and often only do one or two operations on each variable.

There are also minor factors that help at the margins: For columnar code we usually quote numbers for running on batches on 1000 events, which is realistic for how they may be called and can be a factor 1-3 faster than single event processing. Again, it is not quite understood why, but it is consistent with general expectations for columnar code.

Conversely for xAOD mode numbers include event store operations as they would be done in the CP Algorithms. This adds 10-30% to overall execution time, but is also hard/impossible to avoid in realistic jobs. It also doesn't qualitatively change the performance difference.