diff --git a/doc/frame.md b/doc/frame.md new file mode 100644 index 000000000..cab0a9910 --- /dev/null +++ b/doc/frame.md @@ -0,0 +1,157 @@ +# The `Frame` concept +The `podio::Frame` is a general data container for collection data of podio generated EDMs. +Additionally, it offers the functionality to store some (limited) data outside of an EDM. +The basic idea of the `Frame` is to give users of podio the possibility to organize EDM data into logical units and to potentially build a hierarchy of different `Frame`s. +Common examples would be the organisation of data into *Events* and *Runs*. +However, it is important to note that podio does really not impose any meaning on any `Frame` and each `Frame` is essentially defined by its contents. + +## Basic functionality of a `Frame` +The main functionality of a `Frame` is to store and aggregate EDM collection data and it also offers the possibility to store some generic data alongside. +To ensure thread safety and const-correctness a `Frame` takes ownership of any data that is put into it and only gives read access to immutable data. +This is mandated by the interface for collection data (simplified here for better readability): +```cpp +struct Frame { +template +const CollT& put(CollT&& coll, const std::string& name); + +void put(std::unique_ptr coll, const std::string& name); + +template +const CollT& get(const std::string& name) const; + +template +void putParameter(const std::string& name, T value); + +template +const T& getParameter(const std::string); +}; +``` +In this case there are two ways to get collection data into the `Frame` +1. By passing a concrete collection (of type `CollT`) into the `Frame` as an [`rvalue`](https://en.cppreference.com/w/cpp/language/value_category). There are two ways to achieve this, either by passing the return value of a function directly into `Frame::put` or by explicitly moving it in the call via `std::move` if you are using a named variable. +2. By passing a `std::unique_ptr` to a collection. Similar to the first case, this can either be the return value of a function call, or has to be done via `std::move` (as mandated by the `std::unique_ptr` interface). + +In both cases, if you passed in a named variable, the user is left with a *moved-from object*, which has to be in a *valid but indefinite* state, and cannot be used afterwards. +Some compilers and static code analysis tools are able to detect the accidental usage of *moved-from* objects. + +For putting in parameters the basic principle is very similar, with the major difference being, that for *trivial* types `getParameter` will actually return by value. + +For all use cases there is some `enable_if` machinery in place to ensure that only valid collections and valid parameter types can actually be used. +These checks also make sure that it is impossible to put in collections without handing over ownership to the `Frame`. + +### Usage examples for collection data +These are a few very basic usage examples that highlight the main functionality (and potential pitfalls). + +#### Putting collection data into the `Frame` +In all of the following examples, the following basic setup is assumed: +```cpp +#include "podio/Frame.h" + +#include "edm4hep/MCParticleCollection.h" // just to have a concrete example + +// create an empty Frame +auto frame = podio::Frame(); +``` + +Assuming there is a function that creates an `MCParticleCollection` putting the return value into the `Frame` is very simple +```cpp +edm4hep::MCParticleCollection createMCParticles(); // implemented somewhere else + +// put the return value of a function into the Frame +frame.put(createMCParticles(), "particles"); + +// put the return value into the Frame but keep the const reference +auto& particles = frame.put(createMCParticles(), "moreParticles"); +``` + +If working with named variables it is necessary to use `std::move` to put collections into the `Frame`. +The `Frame` will refuse to compile in case a named variable is not moved. +Assuming the same `createMCParticles` function as above, this looks like the following + +```cpp +auto coll = createMCParticles(); +// potentially still modify the collection + +// Need to use std::move now that the collection has a name +frame.put(std::move(coll), "particles"); + +// Keeping a const reference is also possible +// NOTE: We are explicitly using a new variable name here +auto coll2 = createMCParticles(); +auto& particles = frame.put(std::move(coll2), "MCParticles"); +``` +At this point only `particles` is in a valid and **defined** state. + +#### Getting collection (references) from the `Frame` +Obtaining immutable (`const`) references to collections stored in the `Frame` is trivial. +Here we are assuming that the collections are actually present in the `Frame`. +```cpp +auto& particles = frame.get("particles"); +``` + +### Usage for Parameters +Parameters are using the `podio::GenericParameters` class behind the scene. +Hence, the types that can be used are `int`, `float`, and `std::string` as well as as `std::vectors` of those. +For better usability, some overloads for `putParameter` exist to allow for an *in-place* construction, like, e.g. +```cpp +// Passing in a const char* for a std::string +frame.putParameter("aString", "a string value"); + +// Creating a vector of ints on the fly +frame.putParameter("ints", {1, 2, 3, 4}); +``` + +## I/O basics and philosophy +podio offers all the necessary functionality to read and write `Frame`s. +However, it is not in the scope of podio to organize them into a hierarchy, nor +to maintain such a hierarchy. When writing data to file `Frame`s are written to +the file in the order they are passed to the writer. For reading them back podio +offers random access to stored `Frame`s, which should make it possible to +restore any hierarchy again. The Writers and Readers of podio are supposed to be +run on and accessed by only one single thread. + +### Writing a `Frame` +For writing a `Frame` the writers can ask each `Frame` for `CollectionWriteBuffers` for each collection that should be written. +In these buffers the underlying data is still owned by the collection, and by extension the `Frame`. +This makes it possible to write the same collection with several different writers. +Writers can access a `Frame` from several different threads, even though each writer is assumed to be on only one thread. +For writing the `GenericParameters` that are stored in the `Frame` and for other necessary data, similar access functionality is offered by the `Frame`. + +### Reading a `Frame` +When reading a `Frame` readers do not have to return a complete `Frame`. +Instead they return a more or less arbitrary type of `FrameData` that simply has to provide the following public interface. +```cpp +struct FrameData { + /// Get a (copy) of the internal collection id table + podio::CollectionIDTable getIDTable() const; + + /// Get the buffers to construct the collection with the given name + std::optional getCollectionBuffers(const std::string& name); + + /// Get the still available, i.e. yet unpacked, collections from the raw data + std::vector getAvailableCollections() const; + + /// Get the parameters that are stored in the raw data + std::unique_ptr getParameters(); +}; +``` +A `Frame` is constructed with a (`unique_ptr` of such) `FrameData` and handles everything from there. +Note that the `FrameData` type of any I/O backend is a free type without inheritance as the `Frame` constructor is templated on this. +This splitting of reading data from file and constructing a `Frame` from it later has some advantages: +- Since podio assumes that reading is done single threaded the amount of time that is actually spent in a reader is minimized, as only the file operations need to be done on a single thread. All further processing (potential decompression, unpacking, etc.) can be done on a different thread where the `Frame` is actually constructed. +- It gives different backends the necessary freedom to exploit different optimization strategies and does not force them to conform to an implementation that is potentially detrimental to performance. +- It also makes it possible to pass around data from which a `Frame` can be constructed without having to actually construct one. +- Readers do not have to know how to construct collections from the buffers, as they are only required to provide the buffers themselves. + +### Schema evolution +Schema evolution happens on the `CollectionReadBuffers` when they are requested from the `FrameData` inside the `Frame`. +It is possible for the I/O backend to handle schema evolution before the `Frame` sees the buffers for the first time. +In that case podio schema evolution becomes a simple check. + +# Frame implementation and design +One of the main concerns of the `Frame` is to offer one common, non-templated, interface while still supporting different I/O backends and potentially different *policies*. +The "classic" approach would be to have an abstract `IFrame` interface with several implementations that offer the desired functionality (and their small differences). +One problem with that approach is that a purely abstract interface cannot have templated member functions. Hence, the desired type-safe behavior of `get` and `put` would be very hard to implement. +Additionally, policies ideally affect orthogonal aspects of the `Frame` behavior. +Implementing all possible combinations of behaviors through implementations of an abstract interface would lead to quite a bit of code duplication and cannot take advantage of the factorization of the problem. +To solve these problems, we chose to implement the `Frame` via [*Type Erasure*](https://en.wikibooks.org/wiki/More_C%2B%2B_Idioms/Type_Erasure). +This also has the advantage that the `Frame` also has *value semantics* in line with the design of podio. diff --git a/include/podio/CollectionBase.h b/include/podio/CollectionBase.h index fbd67ec99..670291ba3 100644 --- a/include/podio/CollectionBase.h +++ b/include/podio/CollectionBase.h @@ -44,7 +44,10 @@ class CollectionBase { virtual unsigned getID() const = 0; /// Get the collection buffers for this collection - virtual podio::CollectionBuffers getBuffers() = 0; + virtual podio::CollectionWriteBuffers getBuffers() = 0; + + /// Create (empty) collection buffers from which a collection can be constructed + virtual podio::CollectionReadBuffers createBuffers() /*const*/ = 0; /// check for validity of the container after read virtual bool isValid() const = 0; diff --git a/include/podio/CollectionBuffers.h b/include/podio/CollectionBuffers.h index 87d2dce71..d69ff0288 100644 --- a/include/podio/CollectionBuffers.h +++ b/include/podio/CollectionBuffers.h @@ -3,6 +3,7 @@ #include "podio/ObjectID.h" +#include #include #include #include @@ -10,6 +11,8 @@ namespace podio { +class CollectionBase; + template using UVecPtr = std::unique_ptr>; @@ -20,16 +23,81 @@ using VectorMembersInfo = std::vector>; * Simple helper struct that bundles all the potentially necessary buffers that * are necessary to represent a collection for I/O purposes. */ -struct CollectionBuffers { +struct CollectionWriteBuffers { void* data{nullptr}; CollRefCollection* references{nullptr}; VectorMembersInfo* vectorMembers{nullptr}; template std::vector* dataAsVector() { + return asVector(data); + } + + template + static std::vector* asVector(void* raw) { // Are we at a beach? I can almost smell the C... - return *static_cast**>(data); + return *static_cast**>(raw); + } +}; + +struct CollectionReadBuffers { + void* data{nullptr}; + CollRefCollection* references{nullptr}; + VectorMembersInfo* vectorMembers{nullptr}; + + using CreateFuncT = std::function(podio::CollectionReadBuffers, bool)>; + using RecastFuncT = std::function; + + CollectionReadBuffers(void* d, CollRefCollection* ref, VectorMembersInfo* vec, CreateFuncT&& createFunc, + RecastFuncT&& recastFunc) : + data(d), + references(ref), + vectorMembers(vec), + createCollection(std::move(createFunc)), + recast(std::move(recastFunc)) { } + + CollectionReadBuffers() = default; + CollectionReadBuffers(const CollectionReadBuffers&) = default; + CollectionReadBuffers& operator=(const CollectionReadBuffers&) = default; + + CollectionReadBuffers(CollectionWriteBuffers buffers) : + data(buffers.data), references(buffers.references), vectorMembers(buffers.vectorMembers) { + } + + template + std::vector* dataAsVector() { + return asVector(data); + } + + template + static std::vector* asVector(void* raw) { + // Are we at a beach? I can almost smell the C... + return static_cast*>(raw); + } + + CreateFuncT createCollection{}; + + // This is a hacky workaround for the ROOT backend at the moment. There is + // probably a better solution, but I haven't found it yet. The problem is the + // following: + // + // When creating a pointer to a vector, either via new or via + // TClass::New(), we get a void*, that can be cast back to a vector with + // + // static_cast*>(raw); + // + // However, as soon as we pass that same void* to TBranch::SetAddress this no + // longer works and the actual cast has to be + // + // *static_cast**>(raw); + // + // To make it possible to always use the first form, after we leave the Root + // parts of reading, this function is populated in the createBuffers call of each + // datatype where we have the necessary type information (from code + // generation) to do the second cast and assign the result of that to the data + // field again. + RecastFuncT recast{}; }; } // namespace podio diff --git a/include/podio/CollectionIDTable.h b/include/podio/CollectionIDTable.h index 5573b3e2c..c639a904c 100644 --- a/include/podio/CollectionIDTable.h +++ b/include/podio/CollectionIDTable.h @@ -1,6 +1,7 @@ #ifndef PODIO_COLLECTIONIDTABLE_H #define PODIO_COLLECTIONIDTABLE_H +#include #include #include #include @@ -12,11 +13,20 @@ class CollectionIDTable { public: /// default constructor CollectionIDTable() = default; + ~CollectionIDTable() = default; + + CollectionIDTable(const CollectionIDTable&) = delete; + CollectionIDTable& operator=(const CollectionIDTable&) = delete; + CollectionIDTable(CollectionIDTable&&) = default; + CollectionIDTable& operator=(CollectionIDTable&&) = default; /// constructor from existing ID:name mapping CollectionIDTable(std::vector&& ids, std::vector&& names) : m_collectionIDs(std::move(ids)), m_names(std::move(names)){}; + CollectionIDTable(const std::vector& ids, const std::vector& names) : + m_collectionIDs(ids), m_names(names){}; + /// return collection ID for given name int collectionID(const std::string& name) const; @@ -43,10 +53,15 @@ class CollectionIDTable { /// Prints collection information void print() const; + /// Does this table hold any information? + bool empty() const { + return m_names.empty(); + } + private: std::vector m_collectionIDs{}; std::vector m_names{}; - mutable std::mutex m_mutex{}; + mutable std::unique_ptr m_mutex{std::make_unique()}; }; } // namespace podio diff --git a/include/podio/Frame.h b/include/podio/Frame.h new file mode 100644 index 000000000..557eddceb --- /dev/null +++ b/include/podio/Frame.h @@ -0,0 +1,414 @@ +#ifndef PODIO_FRAME_H +#define PODIO_FRAME_H + +#include "podio/CollectionBase.h" +#include "podio/CollectionIDTable.h" +#include "podio/GenericParameters.h" +#include "podio/ICollectionProvider.h" +#include "podio/utilities/TypeHelpers.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace podio { + +/// Alias template for enabling overloads only for Collections +template +using EnableIfCollection = typename std::enable_if_t>; + +/// Alias template for enabling overloads only for Collection r-values +template +using EnableIfCollectionRValue = typename std::enable_if_t && !std::is_lvalue_reference_v>; + +namespace detail { + /** The minimal interface for raw data types + */ + struct EmptyFrameData { + podio::CollectionIDTable getIDTable() const { + return {}; + } + + std::optional getCollectionBuffers(const std::string&) { + return std::nullopt; + } + + /** Get the still available, i.e. yet unpacked, collections from the raw data + */ + std::vector getAvailableCollections() const { + return {}; + } + + /** Get the parameters that are stored in the raw data + */ + std::unique_ptr getParameters() { + return std::make_unique(); + } + }; +} // namespace detail + +template +std::optional unpack(FrameDataT* data, const std::string& name) { + return data->getCollectionBuffers(name); +} + +/** + * Frame class that serves as a container of collection and meta data. + */ +class Frame { + /** + * Internal abstract interface for the type-erased implementation of the Frame + * class + */ + struct FrameConcept { + virtual ~FrameConcept() = default; + virtual const podio::CollectionBase* get(const std::string& name) const = 0; + virtual const podio::CollectionBase* put(std::unique_ptr coll, const std::string& name) = 0; + virtual podio::GenericParameters& parameters() = 0; + virtual const podio::GenericParameters& parameters() const = 0; + + virtual std::vector availableCollections() const = 0; + + // Writing interface. Need this to be able to store all necessary information + // TODO: Figure out whether this can be "hidden" somehow + virtual podio::CollectionIDTable getIDTable() const = 0; + }; + + /** + * The interface implementation of the abstract FrameConcept that is necessary + * for a type-erased implementation of the Frame class + */ + template + struct FrameModel final : FrameConcept, public ICollectionProvider { + + FrameModel(std::unique_ptr data); + ~FrameModel() = default; + FrameModel(const FrameModel&) = delete; + FrameModel& operator=(const FrameModel&) = delete; + FrameModel(FrameModel&&) = default; + FrameModel& operator=(FrameModel&&) = default; + + /** Try and get the collection from the internal storage and return a + * pointer to it if found. Otherwise return a nullptr + */ + const podio::CollectionBase* get(const std::string& name) const final; + + /** Try and place the collection into the internal storage and return a + * pointer to it. If a collection already exists or insertion fails, return + * a nullptr + */ + const podio::CollectionBase* put(std::unique_ptr coll, const std::string& name) final; + + /** Get a reference to the internally used GenericParameters + */ + podio::GenericParameters& parameters() override { + return *m_parameters; + } + /** Get a const reference to the internally used GenericParameters + */ + const podio::GenericParameters& parameters() const override { + return *m_parameters; + }; + + bool get(int collectionID, podio::CollectionBase*& collection) const override; + + podio::CollectionIDTable getIDTable() const override { + // Make a copy + return {m_idTable.ids(), m_idTable.names()}; + } + + std::vector availableCollections() const override; + + private: + podio::CollectionBase* doGet(const std::string& name, bool setReferences = true) const; + + using CollectionMapT = std::unordered_map>; + + mutable CollectionMapT m_collections{}; ///< The internal map for storing unpacked collections + mutable std::unique_ptr m_mapMtx{nullptr}; ///< The mutex for guarding the internal collection map + std::unique_ptr m_data{nullptr}; ///< The raw data read from file + mutable std::unique_ptr m_dataMtx{nullptr}; ///< The mutex for guarding the raw data + podio::CollectionIDTable m_idTable{}; ///< The collection ID table + std::unique_ptr m_parameters{nullptr}; ///< The generic parameter store for this frame + mutable std::set m_retrievedIDs{}; ///< The IDs of the collections that we have already read (but not yet put + ///< into the map) + }; + + std::unique_ptr m_self; ///< The internal concept pointer through which all the work is done + +public: + /** Empty Frame constructor + */ + Frame(); + + /** Frame constructor from (almost) arbitrary raw data + */ + template + Frame(std::unique_ptr); + + // The frame is a non-copyable type + Frame(const Frame&) = delete; + Frame& operator=(const Frame&) = delete; + + Frame(Frame&&) = default; + Frame& operator=(Frame&&) = default; + + /** Frame destructor */ + ~Frame() = default; + + /** Get a collection from the Frame + */ + template > + const CollT& get(const std::string& name) const; + + /** (Destructively) move a collection into the Frame and get a const reference + * back for further use + */ + template > + const CollT& put(CollT&& coll, const std::string& name); + + /** Move a collection into the Frame handing over ownership to the Frame + */ + void put(std::unique_ptr coll, const std::string& name); + + /** Add a value to the parameters of the Frame (if the type is supported). + * Copy the value into the internal store + */ + template > + void putParameter(const std::string& key, T value) { + m_self->parameters().setValue(key, value); + } + + /** Add a string value to the parameters of the Frame by copying it. Dedicated + * overload for enabling the on-the-fly conversion on the string literals. + */ + void putParameter(const std::string& key, std::string value) { + putParameter(key, std::move(value)); + } + + /** Add a vector of strings to the parameters of the Frame (via copy). + * Dedicated overload for enabling on-the-fly conversions of initializer_list + * of string literals. + */ + void putParameter(const std::string& key, std::vector values) { + putParameter>(key, std::move(values)); + } + + /** Add a vector of values into the parameters of the Frame. Overload for + * catching on-the-fly conversions of initializer_lists of values. + */ + template >> + void putParameter(const std::string& key, std::initializer_list&& values) { + putParameter>(key, std::move(values)); + } + + /** Retrieve parameters via key from the internal store. Return type will + * either by a const reference or a value depending on the desired type. + */ + template > + podio::GenericDataReturnType getParameter(const std::string& key) const { + return m_self->parameters().getValue(key); + } + + /** Get all **currently** available collections (including potentially + * unpacked ones from raw data) + */ + std::vector getAvailableCollections() const { + return m_self->availableCollections(); + } + + // Interfaces for writing below + // TODO: Hide this from the public interface somehow? + + /** + * Get the GenericParameters for writing + */ + const podio::GenericParameters& getGenericParametersForWrite() const { + return m_self->parameters(); + } + + /** + * Get a collection for writing (in a prepared and "ready-to-write" state) + */ + const podio::CollectionBase* getCollectionForWrite(const std::string& name) const { + const auto* coll = m_self->get(name); + if (coll) { + coll->prepareForWrite(); + } + + return coll; + } + + podio::CollectionIDTable getCollectionIDTableForWrite() const { + return m_self->getIDTable(); + } +}; + +// implementations below + +Frame::Frame() : Frame(std::make_unique()) { +} + +template +Frame::Frame(std::unique_ptr data) : m_self(std::make_unique>(std::move(data))) { +} + +template +const CollT& Frame::get(const std::string& name) const { + const auto* coll = dynamic_cast(m_self->get(name)); + if (coll) { + return *coll; + } + // TODO: Handle non-existing collections + static const auto emptyColl = CollT(); + return emptyColl; +} + +void Frame::put(std::unique_ptr coll, const std::string& name) { + const auto* retColl = m_self->put(std::move(coll), name); + if (!retColl) { + // TODO: Handle collisions + } +} + +template +const CollT& Frame::put(CollT&& coll, const std::string& name) { + const auto* retColl = static_cast(m_self->put(std::make_unique(std::move(coll)), name)); + if (retColl) { + return *retColl; + } + // TODO: Handle collision case + static const auto emptyColl = CollT(); + return emptyColl; +} + +template +Frame::FrameModel::FrameModel(std::unique_ptr data) : + m_mapMtx(std::make_unique()), + m_data(std::move(data)), + m_dataMtx(std::make_unique()), + m_idTable(std::move(m_data->getIDTable())), + m_parameters(std::move(m_data->getParameters())) { +} + +template +const podio::CollectionBase* Frame::FrameModel::get(const std::string& name) const { + return doGet(name); +} + +template +podio::CollectionBase* Frame::FrameModel::doGet(const std::string& name, bool setReferences) const { + { + // First check whether the collection is in the map already + // + // Collections only land here if they are fully unpacked, i.e. + // prepareAfterRead has been called or it has been put into the Frame + std::lock_guard lock{*m_mapMtx}; + if (const auto it = m_collections.find(name); it != m_collections.end()) { + return it->second.get(); + } + } + + podio::CollectionBase* retColl = nullptr; + + // Now try to get it from the raw data if we have the possibility + if (m_data) { + // Have the buffers in the outer scope here to hold the raw data lock as + // briefly as possible + auto buffers = std::optional{std::nullopt}; + { + std::lock_guard lock{*m_dataMtx}; + buffers = unpack(m_data.get(), name); + } + if (buffers) { + auto coll = buffers->createCollection(buffers.value(), buffers->data == nullptr); + coll->prepareAfterRead(); + coll->setID(m_idTable.collectionID(name)); + { + std::lock_guard mapLock{*m_mapMtx}; + auto [it, success] = m_collections.emplace(name, std::move(coll)); + // TODO: Check success? Or simply assume that everything is fine at this point? + // TODO: Collision handling? + retColl = it->second.get(); + } + + if (setReferences) { + retColl->setReferences(this); + } + } + } + + return retColl; +} + +template +bool Frame::FrameModel::get(int collectionID, CollectionBase*& collection) const { + const auto& name = m_idTable.name(collectionID); + const auto& [_, inserted] = m_retrievedIDs.insert(collectionID); + + if (!inserted) { + auto coll = doGet(name); + if (coll) { + collection = coll; + return true; + } + } else { + auto coll = doGet(name, false); + if (coll) { + collection = coll; + return true; + } + } + + return false; +} + +template +const podio::CollectionBase* Frame::FrameModel::put(std::unique_ptr coll, + const std::string& name) { + { + std::lock_guard lock{*m_mapMtx}; + auto [it, success] = m_collections.try_emplace(name, std::move(coll)); + if (success) { + // TODO: Check whether this collection is already known to the idTable + // -> What to do on collision? + // -> Check before we emplace it into the internal map to prevent possible + // collisions from collections that are potentially present from rawdata? + it->second->setID(m_idTable.add(name)); + return it->second.get(); + } + } + + return nullptr; +} + +template +std::vector Frame::FrameModel::availableCollections() const { + // TODO: Check if there is a more efficient way to do this. Currently this is + // done very conservatively, but in a way that should always work, regardless + // of assumptions. It might be possible to simply return what is in the + // idTable here, because that should in principle encompass everything that is + // in the raw data as well as things that have been put into the frame + + // Lock both the internal map and the rawdata for this + std::scoped_lock lock{*m_mapMtx, *m_dataMtx}; + + auto collections = m_data->getAvailableCollections(); + collections.reserve(collections.size() + m_collections.size()); + + for (const auto& [name, _] : m_collections) { + collections.push_back(name); + } + + return collections; +} + +} // namespace podio + +#endif // PODIO_FRAME_H diff --git a/include/podio/GenericParameters.h b/include/podio/GenericParameters.h index 08f2caac6..5da2c22ef 100644 --- a/include/podio/GenericParameters.h +++ b/include/podio/GenericParameters.h @@ -7,6 +7,8 @@ #include #include #include +#include +#include #include #include @@ -78,8 +80,24 @@ class GenericParameters { using IntMap = MapType; using FloatMap = MapType; using StringMap = MapType; + // need mutex pointers for having the possibility to copy/move GenericParameters + using MutexPtr = std::unique_ptr; public: + GenericParameters() = default; + + /// GenericParameters are copyable + /// NOTE: This is currently mainly done to keep the ROOT I/O happy, because + /// that needs a copy constructor + GenericParameters(const GenericParameters&); + GenericParameters& operator=(const GenericParameters&) = delete; + + /// GenericParameters are default moveable + GenericParameters(GenericParameters&&) = default; + GenericParameters& operator=(GenericParameters&&) = default; + + ~GenericParameters() = default; + /// Get the value that is stored under the given key, by const reference or by /// value depending on the desired type template > @@ -240,15 +258,32 @@ class GenericParameters { } } + /// Get the mutex that guards the map for the given type + template + std::mutex& getMutex() const { + if constexpr (std::is_same_v, int>) { + return *(m_intMtx.get()); + } else if constexpr (std::is_same_v, float>) { + return *(m_floatMtx.get()); + } else { + return *(m_stringMtx.get()); + } + } + private: - IntMap _intMap{}; - FloatMap _floatMap{}; - StringMap _stringMap{}; + IntMap _intMap{}; ///< The map storing the integer values + mutable MutexPtr m_intMtx{std::make_unique()}; ///< The mutex guarding the integer map + FloatMap _floatMap{}; ///< The map storing the float values + mutable MutexPtr m_floatMtx{std::make_unique()}; ///< The mutex guarding the float map + StringMap _stringMap{}; ///< The map storing the double values + mutable MutexPtr m_stringMtx{std::make_unique()}; ///< The mutex guarding the float map }; template GenericDataReturnType GenericParameters::getValue(const std::string& key) const { const auto& map = getMap(); + auto& mtx = getMutex(); + std::lock_guard lock{mtx}; const auto it = map.find(key); // If there is no entry to the key, we just return an empty default // TODO: make this case detectable from the outside @@ -269,11 +304,15 @@ GenericDataReturnType GenericParameters::getValue(const std::string& key) con template void GenericParameters::setValue(const std::string& key, T value) { auto& map = getMap(); + auto& mtx = getMutex(); + if constexpr (detail::isVector) { + std::lock_guard lock{mtx}; map.insert_or_assign(key, std::move(value)); } else { // Wrap the value into a vector with exactly one entry and store that std::vector v = {value}; + std::lock_guard lock{mtx}; map.insert_or_assign(key, std::move(v)); } } @@ -281,6 +320,8 @@ void GenericParameters::setValue(const std::string& key, T value) { template size_t GenericParameters::getN(const std::string& key) const { const auto& map = getMap(); + auto& mtx = getMutex(); + std::lock_guard lock{mtx}; if (const auto it = map.find(key); it != map.end()) { return it->second.size(); } @@ -292,7 +333,11 @@ std::vector GenericParameters::getKeys() const { std::vector keys; const auto& map = getMap(); keys.reserve(map.size()); - std::transform(map.begin(), map.end(), std::back_inserter(keys), [](const auto& pair) { return pair.first; }); + { + auto& mtx = getMutex(); + std::lock_guard lock{mtx}; + std::transform(map.begin(), map.end(), std::back_inserter(keys), [](const auto& pair) { return pair.first; }); + } return keys; } diff --git a/include/podio/ROOTFrameData.h b/include/podio/ROOTFrameData.h new file mode 100644 index 000000000..157e3fbea --- /dev/null +++ b/include/podio/ROOTFrameData.h @@ -0,0 +1,70 @@ +#ifndef PODIO_ROOTFRAMEDATA_H +#define PODIO_ROOTFRAMEDATA_H + +#include "podio/CollectionBuffers.h" +#include "podio/CollectionIDTable.h" +#include "podio/GenericParameters.h" + +#include +#include +#include +#include + +namespace podio { + +class ROOTFrameData { + using CollIDPtr = std::shared_ptr; + +public: + using BufferMap = std::unordered_map; + + ROOTFrameData() = delete; + ~ROOTFrameData() = default; + ROOTFrameData(ROOTFrameData&&) = default; + ROOTFrameData& operator=(ROOTFrameData&&) = default; + ROOTFrameData(const ROOTFrameData&) = delete; + ROOTFrameData& operator=(const ROOTFrameData&) = delete; + + ROOTFrameData(BufferMap&& buffers, CollIDPtr&& idTable, podio::GenericParameters&& params) : + m_buffers(std::move(buffers)), m_idTable(idTable), m_parameters(std::move(params)) { + } + + std::optional getCollectionBuffers(const std::string& name) { + const auto bufferHandle = m_buffers.extract(name); + if (bufferHandle.empty()) { + return std::nullopt; + } + + return {bufferHandle.mapped()}; + } + + podio::CollectionIDTable getIDTable() const { + // Construct a copy of the internal table + return {m_idTable->ids(), m_idTable->names()}; + } + + std::unique_ptr getParameters() { + return std::make_unique(std::move(m_parameters)); + } + + std::vector getAvailableCollections() const { + std::vector collections; + collections.reserve(m_buffers.size()); + for (const auto& [name, _] : m_buffers) { + collections.push_back(name); + } + + return collections; + } + +private: + // TODO: switch to something more elegant once the basic functionality and + // interface is better defined + BufferMap m_buffers{}; + // This is co-owned by each FrameData and the original reader. (for now at least) + CollIDPtr m_idTable{nullptr}; + podio::GenericParameters m_parameters{}; +}; +} // namespace podio + +#endif // PODIO_ROOTFRAMEDATA_H diff --git a/include/podio/ROOTFrameReader.h b/include/podio/ROOTFrameReader.h new file mode 100644 index 000000000..b82f44a87 --- /dev/null +++ b/include/podio/ROOTFrameReader.h @@ -0,0 +1,120 @@ +#ifndef PODIO_ROOTFRAMEREADER_H +#define PODIO_ROOTFRAMEREADER_H + +#include "podio/CollectionBranches.h" +#include "podio/ROOTFrameData.h" +#include "podio/podioVersion.h" + +#include "TChain.h" + +#include +#include +#include +#include +#include +#include + +// forward declarations +class TClass; +// class TChain; +class TFile; +class TTree; + +namespace podio { + +namespace detail { + // Information about the data vector as wall as the collection class type + // and the index in the collection branches cache vector + using CollectionInfo = std::tuple; + +} // namespace detail + +class EventStore; +class CollectionBase; +class CollectionIDTable; +class GenericParameters; +struct CollectionReadBuffers; + +/** + * This class has the function to read available data from disk + * and to prepare collections and buffers. + **/ +class ROOTFrameReader { + +public: + ROOTFrameReader() = default; + ~ROOTFrameReader() = default; + + // non-copyable + ROOTFrameReader(const ROOTFrameReader&) = delete; + ROOTFrameReader& operator=(const ROOTFrameReader&) = delete; + + void openFile(const std::string& filename); + + void openFiles(const std::vector& filenames); + + /** + * Read the next data entry from which a Frame can be constructed for the + * given name. In case there are no more entries left for this name or in + * case there is no data for this name, this returns a nullptr. + */ + std::unique_ptr readNextEntry(const std::string& name); + + /// Returns number of entries for the given name + unsigned getEntries(const std::string& name) const; + + podio::version::Version currentFileVersion() const { + return m_fileVersion; + } + +private: + /** + * Helper struct to group together all the necessary state to read / process a + * given category. A "category" in this case describes all frames with the + * same name which are constrained by the ROOT file structure that we use to + * have the same contents. It encapsulates all state that is necessary for + * reading from a TTree / TChain (i.e. collection infos, branches, ...) + */ + struct CategoryInfo { + /// constructor from chain for more convenient map insertion + CategoryInfo(std::unique_ptr&& c) : chain(std::move(c)) { + } + std::unique_ptr chain{nullptr}; ///< The TChain with the data + unsigned entry{0}; ///< The next entry to read + std::vector> storedClasses{}; ///< The stored collections in this + ///< category + std::vector branches{}; ///< The branches for this category + std::shared_ptr table{nullptr}; ///< The collection ID table for this category + }; + + /** + * Initialze the passed CategoryInfo by setting up the necessary branches, + * collection infos and all necessary meta data to be able to read entries + * with this name + */ + void initCategory(CategoryInfo& catInfo, const std::string& name); + + /** + * Get the category information for the given name. In case there is no TTree + * with contents for the given name this will return a CategoryInfo with an + * uninitialized chain (nullptr) member + */ + CategoryInfo& getCategoryInfo(const std::string& name); + + GenericParameters readEventMetaData(CategoryInfo& catInfo); + + /** + * Get / read the buffers at index iColl in the passed category information + */ + podio::CollectionReadBuffers getCollectionBuffers(CategoryInfo& catInfo, size_t iColl); + + std::unique_ptr m_metaChain{nullptr}; ///< The metadata tree + std::unordered_map m_categories{}; ///< All categories + std::vector m_availCategories{}; ///< All available categories from this file + + podio::version::Version m_fileVersion{0, 0, 0}; +}; + +} // namespace podio + +#endif // PODIO_ROOTFRAMEREADER_H diff --git a/include/podio/ROOTFrameWriter.h b/include/podio/ROOTFrameWriter.h new file mode 100644 index 000000000..9428ed929 --- /dev/null +++ b/include/podio/ROOTFrameWriter.h @@ -0,0 +1,87 @@ +#ifndef PODIO_ROOTFRAMEWRITER_H +#define PODIO_ROOTFRAMEWRITER_H + +#include "podio/CollectionBranches.h" +#include "podio/CollectionIDTable.h" + +#include "TFile.h" + +#include +#include +#include +#include +#include + +// forward declarations +class TTree; + +namespace podio { +class Frame; +class CollectionBase; +class GenericParameters; + +class ROOTFrameWriter { +public: + ROOTFrameWriter(const std::string& filename); + ~ROOTFrameWriter() = default; + + ROOTFrameWriter(const ROOTFrameWriter&) = delete; + ROOTFrameWriter& operator=(const ROOTFrameWriter&) = delete; + + /** Store the given frame with the given category. Store all available + * collections from the Frame. + * + * NOTE: The contents of the first Frame that is written in this way + * determines the contents that will be written for all subsequent Frames. + */ + void writeFrame(const podio::Frame& frame, const std::string& category); + + /** Store the given Frame with the given category. Store only the + * collections that are passed. + * + * NOTE: The contents of the first Frame that is written in this way + * determines the contents that will be written for all subsequent Frames. + */ + void writeFrame(const podio::Frame& frame, const std::string& category, const std::vector& collsToWrite); + + /** Write the current file, including all the necessary metadata to read it again. + */ + void finish(); + +private: + using StoreCollection = std::pair; + + // collectionID, collectionType, subsetCollection + // NOTE: same as in rootUtils.h private header! + using CollectionInfoT = std::tuple; + + /** + * Helper struct to group together all necessary state to write / process a + * given category. Created during the first writing of a category + */ + struct CategoryInfo { + TTree* tree{nullptr}; ///< The TTree to which this category is written + std::vector branches{}; ///< The branches for this category + std::vector collInfo{}; ///< Collection info for this category + podio::CollectionIDTable idTable{}; ///< The collection id table for this category + std::vector collsToWrite{}; ///< The collections to write for this category + }; + + /// Initialize the branches for this category + void initBranches(CategoryInfo& catInfo, const std::vector& collections, + /*const*/ podio::GenericParameters& parameters); + + /// Get the (potentially uninitialized category information for this category) + CategoryInfo& getCategoryInfo(const std::string& category); + + static void resetBranches(std::vector& branches, + const std::vector& collections, + /*const*/ podio::GenericParameters* parameters); + + std::unique_ptr m_file{nullptr}; ///< The storage file + std::unordered_map m_categories{}; ///< All categories +}; + +} // namespace podio + +#endif // PODIO_ROOTFRAMEWRITER_H diff --git a/include/podio/ROOTReader.h b/include/podio/ROOTReader.h index 09dfa646d..ba33cae16 100644 --- a/include/podio/ROOTReader.h +++ b/include/podio/ROOTReader.h @@ -2,7 +2,6 @@ #define PODIO_ROOTREADER_H #include "podio/CollectionBranches.h" -#include "podio/ICollectionProvider.h" #include "podio/IReader.h" #include @@ -26,6 +25,7 @@ class CollectionBase; class Registry; class CollectionIDTable; class GenericParameters; + /** This class has the function to read available data from disk and to prepare collections and buffers. diff --git a/include/podio/SIOBlock.h b/include/podio/SIOBlock.h index ce7ee9937..fdc4bd8a8 100644 --- a/include/podio/SIOBlock.h +++ b/include/podio/SIOBlock.h @@ -35,7 +35,11 @@ class SIOBlock : public sio::block { SIOBlock& operator=(const SIOBlock&) = delete; podio::CollectionBase* getCollection() { - return _col; + return m_buffers.createCollection(m_buffers, m_subsetColl).release(); + } + + podio::CollectionReadBuffers getBuffers() const { + return m_buffers; } std::string name() { @@ -43,16 +47,18 @@ class SIOBlock : public sio::block { } void setCollection(podio::CollectionBase* col) { - _col = col; + m_subsetColl = col->isSubsetCollection(); + m_buffers = col->getBuffers(); } virtual SIOBlock* create(const std::string& name) const = 0; // create a new collection for this block - virtual void createCollection(const bool subsetCollection = false) = 0; + virtual void createBuffers(const bool subsetCollection = false) = 0; protected: - podio::CollectionBase* _col{}; + bool m_subsetColl{false}; + podio::CollectionReadBuffers m_buffers{}; }; /** @@ -65,6 +71,15 @@ class SIOCollectionIDTableBlock : public sio::block { SIOCollectionIDTableBlock(podio::EventStore* store); + SIOCollectionIDTableBlock(std::vector&& names, std::vector&& ids, std::vector&& types, + std::vector&& isSubsetColl) : + sio::block("CollectionIDs", sio::version::encode_version(0, 3)), + _names(std::move(names)), + _ids(std::move(ids)), + _types(std::move(types)), + _isSubsetColl(std::move(isSubsetColl)) { + } + SIOCollectionIDTableBlock(const SIOCollectionIDTableBlock&) = delete; SIOCollectionIDTableBlock& operator=(const SIOCollectionIDTableBlock&) = delete; @@ -208,6 +223,13 @@ class SIOFileTOCRecord { size_t getNRecords(const std::string& name) const; + /** Get the position of the iEntry-th record with the given name. If no entry + * with the given name is recorded, return 0. Note there is no internal check + * on whether the given name actually has iEntry records. Use getNRecords to + * check for that if necessary. + */ + PositionType getPosition(const std::string& name, unsigned iEntry = 0) const; + private: friend struct SIOFileTOCRecordBlock; @@ -221,6 +243,10 @@ struct SIOFileTOCRecordBlock : public sio::block { SIOFileTOCRecordBlock() : sio::block(sio_helpers::SIOTocRecordName, sio::version::encode_version(0, 1)) { } + SIOFileTOCRecordBlock(SIOFileTOCRecord* r) : + sio::block(sio_helpers::SIOTocRecordName, sio::version::encode_version(0, 1)), record(r) { + } + SIOFileTOCRecordBlock(const SIOFileTOCRecordBlock&) = delete; SIOFileTOCRecordBlock& operator=(const SIOFileTOCRecordBlock&) = delete; diff --git a/include/podio/SIOBlockUserData.h b/include/podio/SIOBlockUserData.h index 208a8e59f..7ce28cd8e 100644 --- a/include/podio/SIOBlockUserData.h +++ b/include/podio/SIOBlockUserData.h @@ -1,6 +1,7 @@ #ifndef PODIO_SIOBLOCKUSERDATA_H #define PODIO_SIOBLOCKUSERDATA_H +#include "podio/CollectionBuffers.h" #include "podio/SIOBlock.h" #include "podio/UserDataCollection.h" @@ -37,30 +38,37 @@ class SIOBlockUserData : public podio::SIOBlock { } void read(sio::read_device& device, sio::version_type /*version*/) override { - auto collBuffers = _col->getBuffers(); - auto* dataVec = collBuffers.dataAsVector(); + auto* dataVec = new std::vector(); unsigned size(0); device.data(size); dataVec->resize(size); podio::handlePODDataSIO(device, &(*dataVec)[0], size); + m_buffers.data = dataVec; } void write(sio::write_device& device) override { - _col->prepareForWrite(); - auto collBuffers = _col->getBuffers(); - auto* dataVec = collBuffers.dataAsVector(); + auto* dataVec = podio::CollectionWriteBuffers::asVector(m_buffers.data); unsigned size = dataVec->size(); device.data(size); podio::handlePODDataSIO(device, &(*dataVec)[0], size); } - void createCollection(const bool) override { - setCollection(new podio::UserDataCollection); + void createBuffers(bool) override { + + m_buffers.references = new podio::CollRefCollection(); + m_buffers.vectorMembers = new podio::VectorMembersInfo(); + + // Nothing to do here since UserDataCollections cannot be subset collections + m_buffers.createCollection = [](podio::CollectionReadBuffers buffers, bool) { + return std::make_unique>(std::move(*buffers.dataAsVector())); + }; } SIOBlock* create(const std::string& name) const override { return new SIOBlockUserData(name); } + +private: }; } // namespace podio diff --git a/include/podio/SIOFrameData.h b/include/podio/SIOFrameData.h new file mode 100644 index 000000000..9cbc8a724 --- /dev/null +++ b/include/podio/SIOFrameData.h @@ -0,0 +1,89 @@ +#ifndef PODIO_SIOFRAMEDATA_H +#define PODIO_SIOFRAMEDATA_H + +#include "podio/CollectionBuffers.h" +#include "podio/CollectionIDTable.h" +#include "podio/GenericParameters.h" +#include "podio/SIOBlock.h" + +#include +#include + +#include +#include +#include +#include +#include + +namespace podio { +/** + * The Frame data container for the SIO backend. It is constructed from the + * compressed sio::buffers that is read from file and does all the necessary + * unpacking and decompressing internally after construction. + */ +class SIOFrameData { + +public: + SIOFrameData() = delete; + ~SIOFrameData() = default; + + SIOFrameData(const SIOFrameData&) = delete; + SIOFrameData& operator=(const SIOFrameData&) = delete; + + SIOFrameData(SIOFrameData&&) = default; + SIOFrameData& operator=(SIOFrameData&&) = default; + + /** + * Constructor from the collBuffers containing the collection data and a + * tableBuffer containing the necessary information for unpacking the + * collections. The two size parameters denote the uncompressed size of the + * respective buffers. + */ + SIOFrameData(sio::buffer&& collBuffers, std::size_t dataSize, sio::buffer&& tableBuffer, std::size_t tableSize) : + m_recBuffer(std::move(collBuffers)), + m_tableBuffer(std::move(tableBuffer)), + m_dataSize(dataSize), + m_tableSize(tableSize) { + } + + std::optional getCollectionBuffers(const std::string& name); + + podio::CollectionIDTable getIDTable() { + if (m_idTable.empty()) { + readIdTable(); + } + return {m_idTable.ids(), m_idTable.names()}; + } + + std::unique_ptr getParameters(); + + std::vector getAvailableCollections(); + +private: + void unpackBuffers(); + + void readIdTable(); + + void createBlocks(); + + // Default initialization doesn't really matter here, because they are made + // the correct size on construction + sio::buffer m_recBuffer{sio::kbyte}; ///< The compressed record (data) buffer + sio::buffer m_tableBuffer{sio::kbyte}; ///< The compressed collection id table buffer + + std::size_t m_dataSize{}; ///< Uncompressed data buffer size + std::size_t m_tableSize{}; ///< Uncompressed table size + + std::vector m_availableBlocks{}; ///< The blocks that have already been retrieved + + sio::block_list m_blocks{}; + + podio::CollectionIDTable m_idTable{}; + std::vector m_typeNames{}; + std::vector m_subsetCollectionBits{}; + + podio::GenericParameters m_parameters{}; +}; +} // namespace podio + +#endif // PODIO_SIOFRAMEDATA_H diff --git a/include/podio/SIOFrameReader.h b/include/podio/SIOFrameReader.h new file mode 100644 index 000000000..241289ee7 --- /dev/null +++ b/include/podio/SIOFrameReader.h @@ -0,0 +1,63 @@ +#ifndef PODIO_SIOFRAMEREADER_H +#define PODIO_SIOFRAMEREADER_H + +#include "podio/SIOBlock.h" +#include "podio/SIOFrameData.h" +#include "podio/podioVersion.h" + +#include + +#include +#include +#include + +namespace podio { + +class CollectionIDTable; + +class SIOFrameReader { + +public: + SIOFrameReader(); + ~SIOFrameReader() = default; + + // non copyable + SIOFrameReader(const SIOFrameReader&) = delete; + SIOFrameReader& operator=(const SIOFrameReader&) = delete; + + /** + * Read the next data entry from which a Frame can be constructed for the + * given name. In case there are no more entries left for this name or in + * case there is no data for this name, this returns a nullptr. + */ + std::unique_ptr readNextEntry(const std::string& name); + + /// Returns number of entries for the given name + unsigned getEntries(const std::string& name) const; + + void openFile(const std::string& filename); + + podio::version::Version currentFileVersion() const { + return m_fileVersion; + } + +private: + void readPodioHeader(); + + /// read the TOC record + bool readFileTOCRecord(); + + sio::ifstream m_stream{}; ///< The stream from which we read + + /// Count how many times each an entry of this name has been read already + std::unordered_map m_nameCtr{}; + + /// Table of content record where starting points of named entries can be read from + SIOFileTOCRecord m_tocRecord{}; + /// The podio version that has been used to write the file + podio::version::Version m_fileVersion{0}; +}; + +} // namespace podio + +#endif // PODIO_SIOFRAMEREADER_H diff --git a/include/podio/SIOFrameWriter.h b/include/podio/SIOFrameWriter.h new file mode 100644 index 000000000..1ccc7a2e8 --- /dev/null +++ b/include/podio/SIOFrameWriter.h @@ -0,0 +1,41 @@ +#ifndef PODIO_SIOFRAMEWRITER_H +#define PODIO_SIOFRAMEWRITER_H + +#include "podio/SIOBlock.h" + +#include + +#include +#include +#include + +namespace podio { + +class Frame; + +class SIOFrameWriter { +public: + SIOFrameWriter(const std::string& filename); + ~SIOFrameWriter() = default; + + SIOFrameWriter(const SIOFrameWriter&) = delete; + SIOFrameWriter& operator=(const SIOFrameWriter&) = delete; + + /** Write the given Frame with the given category + */ + void writeFrame(const podio::Frame& frame, const std::string& category); + + /** Write the given Frame with the given category only storing the collections + * that are desired via collsToWrite + */ + void writeFrame(const podio::Frame& frame, const std::string& category, const std::vector& collsToWrite); + + void finish(); + +private: + sio::ofstream m_stream{}; ///< The output file stream + SIOFileTOCRecord m_tocRecord{}; ///< The "table of contents" of the written file +}; +} // namespace podio + +#endif // PODIO_SIOFRAMEWRITER_H diff --git a/include/podio/UserDataCollection.h b/include/podio/UserDataCollection.h index 461004e80..2365c5094 100644 --- a/include/podio/UserDataCollection.h +++ b/include/podio/UserDataCollection.h @@ -2,6 +2,7 @@ #define PODIO_USERDATACOLLECTION_H #include "podio/CollectionBase.h" +#include "podio/CollectionBuffers.h" #include "podio/utilities/TypeHelpers.h" #include @@ -68,6 +69,9 @@ class UserDataCollection : public CollectionBase { public: UserDataCollection() = default; + /// Constructor from an existing vector (wich will be moved from!) + UserDataCollection(std::vector&& vec) : _vec(std::move(vec)) { + } UserDataCollection(const UserDataCollection&) = delete; UserDataCollection& operator=(const UserDataCollection&) = delete; UserDataCollection(UserDataCollection&&) = default; @@ -98,11 +102,21 @@ class UserDataCollection : public CollectionBase { } /// Get the collection buffers for this collection - podio::CollectionBuffers getBuffers() override { + podio::CollectionWriteBuffers getBuffers() override { _vecPtr = &_vec; // Set the pointer to the correct internal vector return {&_vecPtr, &m_refCollections, &m_vecmem_info}; } + podio::CollectionReadBuffers createBuffers() /*const*/ final { + return {nullptr, nullptr, nullptr, + [](podio::CollectionReadBuffers buffers, bool) { + return std::make_unique>(std::move(*buffers.dataAsVector())); + }, + [](podio::CollectionReadBuffers& buffers) { + buffers.data = podio::CollectionWriteBuffers::asVector(buffers.data); + }}; + } + /// check for validity of the container after read bool isValid() const override { return true; diff --git a/include/podio/utilities/TypeHelpers.h b/include/podio/utilities/TypeHelpers.h index 452600026..b351e2118 100644 --- a/include/podio/utilities/TypeHelpers.h +++ b/include/podio/utilities/TypeHelpers.h @@ -101,6 +101,16 @@ namespace detail { static constexpr bool isVector = IsVectorHelper::value; } // namespace detail + +// forward declaration to be able to use it below +class CollectionBase; + +/** + * Alias template for checking whether a passed type T inherits from podio::CollectionBase + */ +template +static constexpr bool isCollection = std::is_base_of_v; + } // namespace podio #endif // PODIO_UTILITIES_TYPEHELPERS_H diff --git a/python/templates/Collection.cc.jinja2 b/python/templates/Collection.cc.jinja2 index e9f40ce2d..bf7fd9e0d 100644 --- a/python/templates/Collection.cc.jinja2 +++ b/python/templates/Collection.cc.jinja2 @@ -19,6 +19,9 @@ {{ collection_type }}::{{ collection_type }}() : m_isValid(false), m_isPrepared(false), m_isSubsetColl(false), m_collectionID(0), m_storage() {} +{{ collection_type }}::{{ collection_type }}({{ collection_type }}Data&& data, bool isSubsetColl) : + m_isValid(false), m_isPrepared(false), m_isSubsetColl(isSubsetColl), m_collectionID(0), m_storage(std::move(data)) {} + {{ collection_type }}::~{{ collection_type }}() { // Need to tell the storage how to clean-up m_storage.clear(m_isSubsetColl); @@ -142,10 +145,35 @@ void {{ collection_type }}::push_back({{ class.bare_type }} object) { } } -podio::CollectionBuffers {{ collection_type }}::getBuffers() { +podio::CollectionWriteBuffers {{ collection_type }}::getBuffers() { return m_storage.getCollectionBuffers(m_isSubsetColl); } +podio::CollectionReadBuffers {{ collection_type }}::createBuffers() /*const*/ { + // Very cumbersome way at the moment. We get the actual buffers to have the + // references and vector members sized appropriately (we will use this + // information to create new buffers outside) + auto collBuffers = m_storage.getCollectionBuffers(m_isSubsetColl); + auto readBuffers = podio::CollectionReadBuffers{}; + readBuffers.references = collBuffers.references; + readBuffers.vectorMembers = collBuffers.vectorMembers; + readBuffers.createCollection = [](podio::CollectionReadBuffers buffers, bool isSubsetColl) { + {{ collection_type }}Data data(buffers, isSubsetColl); + return std::make_unique<{{ collection_type }}>(std::move(data), isSubsetColl); + }; + readBuffers.recast = [](podio::CollectionReadBuffers& buffers) { + if (buffers.data) { + buffers.data = podio::CollectionWriteBuffers::asVector<{{ class.full_type }}Data>(buffers.data); + } +{% if VectorMembers %} +{% for member in VectorMembers %} + (*buffers.vectorMembers)[{{ loop.index0 }}].second = podio::CollectionWriteBuffers::asVector<{{ member.full_type }}>((*buffers.vectorMembers)[{{ loop.index0 }}].second); +{% endfor %} +{% endif %} + }; + return readBuffers; +} + #ifdef PODIO_JSON_OUTPUT void to_json(nlohmann::json& j, const {{ collection_type }}& collection) { j = nlohmann::json::array(); diff --git a/python/templates/Collection.h.jinja2 b/python/templates/Collection.h.jinja2 index 80b963ccb..b6f444e49 100644 --- a/python/templates/Collection.h.jinja2 +++ b/python/templates/Collection.h.jinja2 @@ -47,6 +47,7 @@ public: using iterator = {{ class.bare_type }}MutableCollectionIterator; {{ class.bare_type }}Collection(); + {{ class.bare_type }}Collection({{ class.bare_type }}CollectionData&& data, bool isSubsetColl); // This is a move-only type {{ class.bare_type }}Collection(const {{ class.bare_type}}Collection& ) = delete; {{ class.bare_type }}Collection& operator=(const {{ class.bare_type}}Collection& ) = delete; @@ -106,7 +107,10 @@ public: bool setReferences(const podio::ICollectionProvider* collectionProvider) final; /// Get the collection buffers for this collection - podio::CollectionBuffers getBuffers() final; + podio::CollectionWriteBuffers getBuffers() final; + + /// Create (empty) collection buffers from which a collection can be constructed + podio::CollectionReadBuffers createBuffers() /*const*/ final; void setID(unsigned ID) final { m_collectionID = ID; diff --git a/python/templates/CollectionData.cc.jinja2 b/python/templates/CollectionData.cc.jinja2 index 5ea6342b2..ddc6f29dd 100644 --- a/python/templates/CollectionData.cc.jinja2 +++ b/python/templates/CollectionData.cc.jinja2 @@ -28,6 +28,23 @@ {% endfor %} } +{{ class_type }}::{{ class_type }}(podio::CollectionReadBuffers buffers, bool isSubsetColl) : +{% for relation in OneToManyRelations + OneToOneRelations %} + m_rel_{{ relation.name }}(new std::vector<{{ relation.namespace }}::{{ relation.bare_type }}>()), +{% endfor %} + m_refCollections(std::move(*buffers.references)), + m_vecmem_info(std::move(*buffers.vectorMembers)) { + // For subset collections we are done, for proper collections we still have to + // populate the data and vector members + if (!isSubsetColl) { + m_data.reset(buffers.dataAsVector<{{ class.full_type }}Data>()); + +{% for member in VectorMembers %} + m_vec_{{ member.name }}.reset(podio::CollectionReadBuffers::asVector<{{ member.full_type }}>(m_vecmem_info[{{ loop.index0 }}].second)); +{% endfor %} + } +} + void {{ class_type }}::clear(bool isSubsetColl) { if (isSubsetColl) { // We don't own the objects so no cleanup to do here @@ -64,7 +81,7 @@ void {{ class_type }}::clear(bool isSubsetColl) { entries.clear(); } -podio::CollectionBuffers {{ class_type }}::getCollectionBuffers(bool isSubsetColl) { +podio::CollectionWriteBuffers {{ class_type }}::getCollectionBuffers(bool isSubsetColl) { {% if VectorMembers %} // Make sure these point to the right place, even if a collection has been // moved since it has been created diff --git a/python/templates/CollectionData.h.jinja2 b/python/templates/CollectionData.h.jinja2 index 43faaa331..50ae8dd02 100644 --- a/python/templates/CollectionData.h.jinja2 +++ b/python/templates/CollectionData.h.jinja2 @@ -41,6 +41,11 @@ public: */ {{ class_type }}(); + /** + * Constructor from existing I/O buffers + */ + {{ class_type }}(podio::CollectionReadBuffers buffers, bool isSubsetColl); + /** * Non copy-able, move-only class */ @@ -56,7 +61,7 @@ public: void clear(bool isSubsetColl); - podio::CollectionBuffers getCollectionBuffers(bool isSubsetColl); + podio::CollectionWriteBuffers getCollectionBuffers(bool isSubsetColl); void prepareForWrite(bool isSubsetColl); @@ -77,12 +82,12 @@ private: std::vector> m_rel_{{ relation.name }}_tmp{}; ///< Relation buffer for internal book-keeping {% endfor %} {% for relation in OneToOneRelations %} - podio::UVecPtr<{{ relation.namespace }}::{{ relation.bare_type }}> m_rel_{{ relation.name }}; ///< Relation buffer for read / write + podio::UVecPtr<{{ relation.namespace }}::{{ relation.bare_type }}> m_rel_{{ relation.name }}{nullptr}; ///< Relation buffer for read / write {% endfor %} // members to handle vector members {% for member in VectorMembers %} - podio::UVecPtr<{{ member.full_type }}> m_vec_{{ member.name }}; /// combined vector of all objects in collection + podio::UVecPtr<{{ member.full_type }}> m_vec_{{ member.name }}{nullptr}; /// combined vector of all objects in collection std::vector> m_vecs_{{ member.name }}{}; /// pointers to individual member vectors {% endfor %} diff --git a/python/templates/SIOBlock.cc.jinja2 b/python/templates/SIOBlock.cc.jinja2 index 10282d014..a33febb8c 100644 --- a/python/templates/SIOBlock.cc.jinja2 +++ b/python/templates/SIOBlock.cc.jinja2 @@ -5,6 +5,8 @@ #include "{{ incfolder }}{{ class.bare_type }}SIOBlock.h" #include "{{ incfolder }}{{ class.bare_type }}Collection.h" +#include "podio/CollectionBuffers.h" + #include #include #include @@ -13,17 +15,25 @@ {% with block_class = class.bare_type + 'SIOBlock' %} void {{ block_class }}::read(sio::read_device& device, sio::version_type) { - auto collBuffers = _col->getBuffers(); - if (not _col->isSubsetCollection()) { - auto* dataVec = collBuffers.dataAsVector<{{ class.full_type }}Data>(); + if (m_subsetColl) { + m_buffers.references->emplace_back(std::make_unique>()); + } else { +{% for relation in OneToManyRelations + OneToOneRelations %} + m_buffers.references->emplace_back(std::make_unique>()); +{% endfor %} + } + + if (not m_subsetColl) { unsigned size(0); device.data( size ); - dataVec->resize(size); + m_buffers.data = new std::vector<{{ class.full_type }}Data>(size); + auto* dataVec = m_buffers.dataAsVector<{{ class.full_type }}Data>(); podio::handlePODDataSIO(device, dataVec->data(), size); + // m_buffers.data = dataVec; } //---- read ref collections ----- - auto* refCols = collBuffers.references; + auto* refCols = m_buffers.references; for( auto& refC : *refCols ){ unsigned size{0}; device.data( size ) ; @@ -32,8 +42,13 @@ void {{ block_class }}::read(sio::read_device& device, sio::version_type) { } {% if VectorMembers %} +{% for member in VectorMembers %} + // auto {{ member.name }}Buffers = new std::vector<{{ member.full_type }}>(); + // m_buffers.vectorMembers->emplace_back("{{ member.full_type }}", &{{ member.name }}Buffers); + m_buffers.vectorMembers->emplace_back("{{ member.full_type }}", new std::vector<{{ member.full_type }}>()); +{% endfor %} //---- read vector members - auto* vecMemInfo = collBuffers.vectorMembers; + auto* vecMemInfo = m_buffers.vectorMembers; unsigned size{0}; {% for member in VectorMembers %} @@ -43,17 +58,15 @@ void {{ block_class }}::read(sio::read_device& device, sio::version_type) { } void {{ block_class }}::write(sio::write_device& device) { - _col->prepareForWrite() ; - auto collBuffers = _col->getBuffers(); - if (not _col->isSubsetCollection()) { - auto* dataVec = collBuffers.dataAsVector<{{ class.full_type }}Data>(); + if (not m_subsetColl) { + auto* dataVec = podio::CollectionWriteBuffers::asVector<{{ class.full_type }}Data>(m_buffers.data); unsigned size = dataVec->size() ; device.data( size ) ; podio::handlePODDataSIO( device , dataVec->data(), size ) ; } //---- write ref collections ----- - auto* refCols = collBuffers.references; + auto* refCols = m_buffers.references; for( auto& refC : *refCols ){ unsigned size = refC->size() ; device.data( size ) ; @@ -62,7 +75,7 @@ void {{ block_class }}::write(sio::write_device& device) { {% if VectorMembers %} //---- write vector members - auto* vecMemInfo = collBuffers.vectorMembers; + auto* vecMemInfo = m_buffers.vectorMembers; unsigned size{0}; {% for member in VectorMembers %} @@ -71,9 +84,18 @@ void {{ block_class }}::write(sio::write_device& device) { {% endif %} } -void {{ block_class }}::createCollection(const bool subsetCollection) { - setCollection(new {{ class.bare_type }}Collection); - _col->setSubsetCollection(subsetCollection); +void {{ block_class }}::createBuffers(bool subsetColl) { + m_subsetColl = subsetColl; + + + + m_buffers.references = new podio::CollRefCollection(); + m_buffers.vectorMembers = new podio::VectorMembersInfo(); + + m_buffers.createCollection = [](podio::CollectionReadBuffers buffers, bool isSubsetColl) { + {{ class.bare_type }}CollectionData data(buffers, isSubsetColl); + return std::make_unique<{{ class.bare_type }}Collection>(std::move(data), isSubsetColl); + }; } {% endwith %} diff --git a/python/templates/SIOBlock.h.jinja2 b/python/templates/SIOBlock.h.jinja2 index 1ea55d69e..62d210852 100644 --- a/python/templates/SIOBlock.h.jinja2 +++ b/python/templates/SIOBlock.h.jinja2 @@ -13,6 +13,10 @@ #include #include +namespace podio { + struct CollectionReadBuffers; +} + {{ utils.namespace_open(class.namespace) }} {% with block_class = class.bare_type + 'SIOBlock' %} @@ -27,15 +31,18 @@ public: // SIOBlock(name + "__{{ class.bare_type }}", sio::version::encode_version(0, 1)) {} SIOBlock(name, sio::version::encode_version(0, 1)) {} - // Read the particle data from the device - virtual void read(sio::read_device& device, sio::version_type version) override; + // Read the collection data from the device + void read(sio::read_device& device, sio::version_type version) override; - // Write the particle data to the device - virtual void write(sio::write_device& device) override; + // Write the collection data to the device + void write(sio::write_device& device) override; - virtual void createCollection(const bool subsetCollection=false) override; + void createBuffers(bool isSubsetColl) override; SIOBlock* create(const std::string& name) const override { return new {{ block_class }}(name); } + +private: + podio::CollectionReadBuffers createBuffers() const; }; static {{ block_class }} _dummy{{ block_class }}; diff --git a/python/templates/macros/sioblocks.jinja2 b/python/templates/macros/sioblocks.jinja2 index 379aceae7..b3b903f8b 100644 --- a/python/templates/macros/sioblocks.jinja2 +++ b/python/templates/macros/sioblocks.jinja2 @@ -8,9 +8,9 @@ {% macro vector_member_read(member, index) %} - auto* vec{{ index }} = *reinterpret_cast**>(vecMemInfo->at({{ index }}).second); + auto* vec{{ index }} = reinterpret_cast*>(vecMemInfo->at({{ index }}).second); size = 0u; device.data(size); vec{{ index }}->resize(size); - podio::handlePODDataSIO(device, &(*vec{{ index }})[0], size); + podio::handlePODDataSIO(device, vec{{ index }}->data(), size); {% endmacro %} diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 9c4b5bd87..696831613 100755 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -11,6 +11,8 @@ SET(root_sources rootUtils.h ROOTWriter.cc ROOTReader.cc + ROOTFrameWriter.cc + ROOTFrameReader.cc ) SET(sio_sources @@ -18,6 +20,9 @@ SET(sio_sources SIOWriter.cc SIOBlockUserData.cc SIOBlock.cc + SIOFrameWriter.cc + SIOFrameReader.cc + SIOFrameData.cc ) SET(python_sources diff --git a/src/CollectionIDTable.cc b/src/CollectionIDTable.cc index 7d25f63d4..f999af077 100644 --- a/src/CollectionIDTable.cc +++ b/src/CollectionIDTable.cc @@ -6,21 +6,21 @@ namespace podio { const std::string CollectionIDTable::name(int ID) const { - std::lock_guard lock(m_mutex); + std::lock_guard lock(*m_mutex); const auto result = std::find(begin(m_collectionIDs), end(m_collectionIDs), ID); const auto index = std::distance(m_collectionIDs.begin(), result); return m_names[index]; } int CollectionIDTable::collectionID(const std::string& name) const { - std::lock_guard lock(m_mutex); + std::lock_guard lock(*m_mutex); const auto result = std::find(begin(m_names), end(m_names), name); const auto index = std::distance(m_names.begin(), result); return m_collectionIDs[index]; } void CollectionIDTable::print() const { - std::lock_guard lock(m_mutex); + std::lock_guard lock(*m_mutex); std::cout << "CollectionIDTable" << std::endl; for (unsigned i = 0; i < m_names.size(); ++i) { std::cout << "\t" << m_names[i] << " : " << m_collectionIDs[i] << std::endl; @@ -28,13 +28,13 @@ void CollectionIDTable::print() const { } bool CollectionIDTable::present(const std::string& name) const { - std::lock_guard lock(m_mutex); + std::lock_guard lock(*m_mutex); const auto result = std::find(begin(m_names), end(m_names), name); return result != end(m_names); } int CollectionIDTable::add(const std::string& name) { - std::lock_guard lock(m_mutex); + std::lock_guard lock(*m_mutex); const auto result = std::find(begin(m_names), end(m_names), name); int ID = 0; if (result == m_names.end()) { diff --git a/src/GenericParameters.cc b/src/GenericParameters.cc index 654f4dd61..4cf48d550 100644 --- a/src/GenericParameters.cc +++ b/src/GenericParameters.cc @@ -5,6 +5,23 @@ namespace podio { +GenericParameters::GenericParameters(const GenericParameters& other) : + m_intMtx(std::make_unique()), + m_floatMtx(std::make_unique()), + m_stringMtx(std::make_unique()) { + { + // acquire all three locks at once to make sure all three internal maps are + // copied at the same "state" of the GenericParameters + auto& intMtx = other.getMutex(); + auto& floatMtx = other.getMutex(); + auto& stringMtx = other.getMutex(); + std::scoped_lock lock(intMtx, floatMtx, stringMtx); + _intMap = other._intMap; + _floatMap = other._floatMap; + _stringMap = other._stringMap; + } +} + int GenericParameters::getIntVal(const std::string& key) const { return getValue(key); } diff --git a/src/ROOTFrameReader.cc b/src/ROOTFrameReader.cc new file mode 100644 index 000000000..30cbda383 --- /dev/null +++ b/src/ROOTFrameReader.cc @@ -0,0 +1,275 @@ +#include "podio/ROOTFrameReader.h" +#include "podio/CollectionBase.h" +#include "podio/CollectionBuffers.h" +#include "podio/CollectionIDTable.h" +#include "podio/GenericParameters.h" +#include "rootUtils.h" + +// ROOT specific includes +#include "TChain.h" +#include "TClass.h" +#include "TFile.h" +#include "TTree.h" +#include "TTreeCache.h" + +#include + +namespace podio { + +std::tuple, std::vector>> +createCollectionBranches(TChain* chain, const podio::CollectionIDTable& idTable, + const std::vector& collInfo); + +GenericParameters ROOTFrameReader::readEventMetaData(ROOTFrameReader::CategoryInfo& catInfo) { + // Parameter branch is always the last one + auto& paramBranches = catInfo.branches.back(); + auto* branch = paramBranches.data; + + GenericParameters params; + auto* emd = ¶ms; + branch->SetAddress(&emd); + branch->GetEntry(catInfo.entry); + return params; +} + +std::unique_ptr ROOTFrameReader::readNextEntry(const std::string& name) { + auto& catInfo = getCategoryInfo(name); + if (!catInfo.chain) { + return nullptr; + } + if (catInfo.entry >= catInfo.chain->GetEntries()) { + return nullptr; + } + + ROOTFrameData::BufferMap buffers; + for (size_t i = 0; i < catInfo.storedClasses.size(); ++i) { + buffers.emplace(catInfo.storedClasses[i].first, getCollectionBuffers(catInfo, i)); + } + + auto parameters = readEventMetaData(catInfo); + + catInfo.entry++; + return std::make_unique(std::move(buffers), catInfo.table, std::move(parameters)); +} + +podio::CollectionReadBuffers ROOTFrameReader::getCollectionBuffers(ROOTFrameReader::CategoryInfo& catInfo, + size_t iColl) { + const auto& name = catInfo.storedClasses[iColl].first; + const auto& [theClass, collectionClass, index] = catInfo.storedClasses[iColl].second; + auto& branches = catInfo.branches[index]; + + // Create empty collection buffers, and connect them to the right branches + auto collBuffers = podio::CollectionReadBuffers(); + // If we have a valid data buffer class we know that have to read data, + // otherwise we are handling a subset collection + const bool isSubsetColl = theClass == nullptr; + if (!isSubsetColl) { + collBuffers.data = theClass->New(); + } + + { + auto collection = + std::unique_ptr(static_cast(collectionClass->New())); + collection->setSubsetCollection(isSubsetColl); + + auto tmpBuffers = collection->createBuffers(); + collBuffers.createCollection = std::move(tmpBuffers.createCollection); + collBuffers.recast = std::move(tmpBuffers.recast); + + if (auto* refs = tmpBuffers.references) { + collBuffers.references = new podio::CollRefCollection(refs->size()); + } + if (auto* vminfo = tmpBuffers.vectorMembers) { + collBuffers.vectorMembers = new podio::VectorMembersInfo(); + collBuffers.vectorMembers->reserve(vminfo->size()); + + for (const auto& [type, _] : (*vminfo)) { + const auto* vecClass = TClass::GetClass(("vector<" + type + ">").c_str()); + collBuffers.vectorMembers->emplace_back(type, vecClass->New()); + } + } + } + + const auto localEntry = catInfo.chain->LoadTree(catInfo.entry); + // After switching trees in the chain, branch pointers get invalidated so + // they need to be reassigned. + // NOTE: root 6.22/06 requires that we get completely new branches here, + // with 6.20/04 we could just re-set them + if (localEntry == 0) { + branches.data = root_utils::getBranch(catInfo.chain.get(), name.c_str()); + + // reference collections + if (auto* refCollections = collBuffers.references) { + for (size_t i = 0; i < refCollections->size(); ++i) { + const auto brName = root_utils::refBranch(name, i); + branches.refs[i] = root_utils::getBranch(catInfo.chain.get(), brName.c_str()); + } + } + + // vector members + if (auto* vecMembers = collBuffers.vectorMembers) { + for (size_t i = 0; i < vecMembers->size(); ++i) { + const auto brName = root_utils::vecBranch(name, i); + branches.vecs[i] = root_utils::getBranch(catInfo.chain.get(), brName.c_str()); + } + } + } + + // set the addresses and read the data + root_utils::setCollectionAddresses(collBuffers, branches); + root_utils::readBranchesData(branches, localEntry); + + collBuffers.recast(collBuffers); + + return collBuffers; +} + +ROOTFrameReader::CategoryInfo& ROOTFrameReader::getCategoryInfo(const std::string& category) { + if (auto it = m_categories.find(category); it != m_categories.end()) { + // Use the id table as proxy to check whether this category has been + // initialized alrready + if (it->second.table == nullptr) { + initCategory(it->second, category); + } + return it->second; + } + + // Use a nullptr TChain to signify an invalid category request + // TODO: Warn / log + static auto invalidCategory = CategoryInfo{nullptr}; + + return invalidCategory; +} + +void ROOTFrameReader::initCategory(CategoryInfo& catInfo, const std::string& category) { + catInfo.table = std::make_shared(); + auto* table = catInfo.table.get(); + auto* tableBranch = root_utils::getBranch(m_metaChain.get(), root_utils::idTableName(category)); + tableBranch->SetAddress(&table); + tableBranch->GetEntry(0); + + auto* collInfoBranch = root_utils::getBranch(m_metaChain.get(), root_utils::collInfoName(category)); + auto collInfo = new std::vector(); + collInfoBranch->SetAddress(&collInfo); + collInfoBranch->GetEntry(0); + + std::tie(catInfo.branches, catInfo.storedClasses) = + createCollectionBranches(catInfo.chain.get(), *catInfo.table, *collInfo); + + delete collInfo; + + // Finaly set up the branches for the paramters + root_utils::CollectionBranches paramBranches{}; + paramBranches.data = root_utils::getBranch(catInfo.chain.get(), root_utils::paramBranchName); + catInfo.branches.push_back(paramBranches); +} + +std::vector getAvailableCategories(TChain* metaChain) { + auto* branches = metaChain->GetListOfBranches(); + std::vector brNames; + brNames.reserve(branches->GetEntries()); + for (int i = 0; i < branches->GetEntries(); ++i) { + const std::string name = branches->At(i)->GetName(); + const auto fUnder = name.find("___"); + if (fUnder != std::string::npos) { + brNames.emplace_back(name.substr(0, fUnder)); + } + } + + std::sort(brNames.begin(), brNames.end()); + brNames.erase(std::unique(brNames.begin(), brNames.end()), brNames.end()); + + return brNames; +} + +void ROOTFrameReader::openFile(const std::string& filename) { + openFiles({filename}); +} + +void ROOTFrameReader::openFiles(const std::vector& filenames) { + m_metaChain = std::make_unique(root_utils::metaTreeName); + // NOTE: We simply assume that the meta data doesn't change throughout the + // chain! This essentially boils down to the assumption that all files that + // are read this way were written with the same settings. + m_metaChain->Add(filenames[0].c_str()); + + podio::version::Version* versionPtr{nullptr}; + if (auto* versionBranch = root_utils::getBranch(m_metaChain.get(), root_utils::versionBranchName)) { + versionBranch->SetAddress(&versionPtr); + versionBranch->GetEntry(0); + } + m_fileVersion = versionPtr ? *versionPtr : podio::version::Version{0, 0, 0}; + delete versionPtr; + + // Do some work up front for setting up categories and setup all the chains + // and record the available categories. The rest of the setup follows on + // demand when the category is first read + m_availCategories = getAvailableCategories(m_metaChain.get()); + for (const auto& cat : m_availCategories) { + auto [it, _] = m_categories.try_emplace(cat, std::make_unique(cat.c_str())); + for (const auto& fn : filenames) { + it->second.chain->Add(fn.c_str()); + } + } +} + +unsigned ROOTFrameReader::getEntries(const std::string& name) const { + if (auto it = m_categories.find(name); it != m_categories.end()) { + return it->second.chain->GetEntries(); + } + + return 0; +} + +std::tuple, std::vector>> +createCollectionBranches(TChain* chain, const podio::CollectionIDTable& idTable, + const std::vector& collInfo) { + + size_t collectionIndex{0}; + std::vector collBranches; + collBranches.reserve(collInfo.size() + 1); + std::vector> storedClasses; + storedClasses.reserve(collInfo.size()); + + for (const auto& [collID, collType, isSubsetColl] : collInfo) { + // We only write collections that are in the collectionIDTable, so no need + // to check here + const auto name = idTable.name(collID); + + root_utils::CollectionBranches branches{}; + const auto collectionClass = TClass::GetClass(collType.c_str()); + + // Need the collection here to setup all the branches. Have to manage the + // temporary collection ourselves + auto collection = + std::unique_ptr(static_cast(collectionClass->New())); + collection->setSubsetCollection(isSubsetColl); + + if (!isSubsetColl) { + // This branch is guaranteed to exist since only collections that are + // also written to file are in the info metadata that we work with here + branches.data = root_utils::getBranch(chain, name.c_str()); + } + + const auto buffers = collection->getBuffers(); + for (size_t i = 0; i < buffers.references->size(); ++i) { + const auto brName = root_utils::refBranch(name, i); + branches.refs.push_back(root_utils::getBranch(chain, brName.c_str())); + } + + for (size_t i = 0; i < buffers.vectorMembers->size(); ++i) { + const auto brName = root_utils::vecBranch(name, i); + branches.vecs.push_back(root_utils::getBranch(chain, brName.c_str())); + } + + const std::string bufferClassName = "std::vector<" + collection->getDataTypeName() + ">"; + const auto bufferClass = isSubsetColl ? nullptr : TClass::GetClass(bufferClassName.c_str()); + + storedClasses.emplace_back(name, std::make_tuple(bufferClass, collectionClass, collectionIndex++)); + collBranches.push_back(branches); + } + + return {collBranches, storedClasses}; +} + +} // namespace podio diff --git a/src/ROOTFrameWriter.cc b/src/ROOTFrameWriter.cc new file mode 100644 index 000000000..39708a096 --- /dev/null +++ b/src/ROOTFrameWriter.cc @@ -0,0 +1,139 @@ +#include "podio/ROOTFrameWriter.h" +#include "podio/CollectionBase.h" +#include "podio/Frame.h" +#include "podio/GenericParameters.h" +#include "podio/podioVersion.h" + +#include "rootUtils.h" + +#include "TTree.h" + +namespace podio { + +ROOTFrameWriter::ROOTFrameWriter(const std::string& filename) { + m_file = std::make_unique(filename.c_str(), "recreate"); +} + +void ROOTFrameWriter::writeFrame(const podio::Frame& frame, const std::string& category) { + writeFrame(frame, category, frame.getAvailableCollections()); +} + +void ROOTFrameWriter::writeFrame(const podio::Frame& frame, const std::string& category, + const std::vector& collsToWrite) { + auto& catInfo = getCategoryInfo(category); + // Use the TTree as proxy here to decide whether this category has already + // been initialized + if (catInfo.tree == nullptr) { + catInfo.idTable = frame.getCollectionIDTableForWrite(); + catInfo.collsToWrite = collsToWrite; + catInfo.tree = new TTree(category.c_str(), (category + " data tree").c_str()); + catInfo.tree->SetDirectory(m_file.get()); + } + + std::vector collections; + collections.reserve(catInfo.collsToWrite.size()); + for (const auto& name : catInfo.collsToWrite) { + auto* coll = frame.getCollectionForWrite(name); + collections.emplace_back(name, const_cast(coll)); + } + + // We will at least have a parameters branch, even if there are no + // collections + if (catInfo.branches.empty()) { + initBranches(catInfo, collections, const_cast(frame.getGenericParametersForWrite())); + + } else { + resetBranches(catInfo.branches, collections, + &const_cast(frame.getGenericParametersForWrite())); + } + + catInfo.tree->Fill(); +} + +ROOTFrameWriter::CategoryInfo& ROOTFrameWriter::getCategoryInfo(const std::string& category) { + if (auto it = m_categories.find(category); it != m_categories.end()) { + return it->second; + } + + auto [it, _] = m_categories.try_emplace(category, CategoryInfo{}); + return it->second; +} + +void ROOTFrameWriter::initBranches(CategoryInfo& catInfo, const std::vector& collections, + /*const*/ podio::GenericParameters& parameters) { + catInfo.branches.reserve(collections.size() + 1); // collections + parameters + + // First collections + for (auto& [name, coll] : collections) { + root_utils::CollectionBranches branches; + const auto buffers = coll->getBuffers(); + + // data buffer branch, only for non-subset collections + if (buffers.data) { + auto bufferDataType = "vector<" + coll->getDataTypeName() + ">"; + branches.data = catInfo.tree->Branch(name.c_str(), bufferDataType.c_str(), buffers.data); + } + + // reference collections + if (auto refColls = buffers.references) { + int i = 0; + for (auto& c : (*refColls)) { + const auto brName = root_utils::refBranch(name, i++); + branches.refs.push_back(catInfo.tree->Branch(brName.c_str(), c.get())); + } + } + + // vector members + if (auto vmInfo = buffers.vectorMembers) { + int i = 0; + for (auto& [type, vec] : (*vmInfo)) { + const auto typeName = "vector<" + type + ">"; + const auto brName = root_utils::vecBranch(name, i++); + branches.vecs.push_back(catInfo.tree->Branch(brName.c_str(), typeName.c_str(), vec)); + } + } + + catInfo.branches.push_back(branches); + catInfo.collInfo.emplace_back(catInfo.idTable.collectionID(name), coll->getTypeName(), coll->isSubsetCollection()); + } + + // Also make branches for the parameters + root_utils::CollectionBranches branches; + branches.data = catInfo.tree->Branch(root_utils::paramBranchName, ¶meters); + catInfo.branches.push_back(branches); +} + +void ROOTFrameWriter::resetBranches(std::vector& branches, + const std::vector& collections, + /*const*/ podio::GenericParameters* parameters) { + size_t iColl = 0; + for (auto& coll : collections) { + const auto& collBranches = branches[iColl]; + root_utils::setCollectionAddresses(coll.second->getBuffers(), collBranches); + iColl++; + } + + branches.back().data->SetAddress(¶meters); +} + +void ROOTFrameWriter::finish() { + auto* metaTree = new TTree(root_utils::metaTreeName, "metadata tree for podio I/O functionality"); + metaTree->SetDirectory(m_file.get()); + + // Store the collection id table and collection info for reading in the meta tree + for (/*const*/ auto& [category, info] : m_categories) { + metaTree->Branch(root_utils::idTableName(category).c_str(), &info.idTable); + metaTree->Branch(root_utils::collInfoName(category).c_str(), &info.collInfo); + } + + // Store the current podio build version into the meta data tree + auto podioVersion = podio::version::build_version; + metaTree->Branch(root_utils::versionBranchName, &podioVersion); + + metaTree->Fill(); + + m_file->Write(); + m_file->Close(); +} + +} // namespace podio diff --git a/src/ROOTReader.cc b/src/ROOTReader.cc index d5ae89775..c34cef9c0 100644 --- a/src/ROOTReader.cc +++ b/src/ROOTReader.cc @@ -109,7 +109,7 @@ CollectionBase* ROOTReader::getCollection(const std::pairgetBuffers(), branches); return readCollectionData(branches, collection, localEntry, name); } diff --git a/src/ROOTWriter.cc b/src/ROOTWriter.cc index 2a8b09bfd..46842b370 100644 --- a/src/ROOTWriter.cc +++ b/src/ROOTWriter.cc @@ -90,7 +90,7 @@ void ROOTWriter::setBranches(const std::vector& collections) { size_t iCollection = 0; for (auto& coll : collections) { const auto& branches = m_collectionBranches[iCollection]; - root_utils::setCollectionAddresses(coll.second, branches); + root_utils::setCollectionAddresses(coll.second->getBuffers(), branches); iCollection++; } @@ -111,6 +111,7 @@ void ROOTWriter::finish() { // No check necessary, only registered collections possible m_store->get(name, coll); const auto collType = coll->getTypeName(); + // const auto collType = "std::vector<" + coll->getDataTypeName() + ">"; collectionInfo.emplace_back(collID, std::move(collType), coll->isSubsetCollection()); } diff --git a/src/SIOBlock.cc b/src/SIOBlock.cc index db2b777af..c521b694c 100644 --- a/src/SIOBlock.cc +++ b/src/SIOBlock.cc @@ -113,12 +113,12 @@ void SIONumberedMetaDataBlock::write(sio::write_device& device) { } std::shared_ptr SIOBlockFactory::createBlock(const std::string& typeStr, const std::string& name, - const bool isRefColl) const { + const bool isSubsetColl) const { const auto it = _map.find(typeStr); if (it != _map.end()) { auto blk = std::shared_ptr(it->second->create(name)); - blk->createCollection(isRefColl); + blk->createBuffers(isSubsetColl); return blk; } else { return nullptr; @@ -210,6 +210,18 @@ size_t SIOFileTOCRecord::getNRecords(const std::string& name) const { return 0; } +SIOFileTOCRecord::PositionType SIOFileTOCRecord::getPosition(const std::string& name, unsigned iEntry) const { + const auto it = std::find_if(m_recordMap.cbegin(), m_recordMap.cend(), + [&name](const auto& keyVal) { return keyVal.first == name; }); + if (it != m_recordMap.end()) { + if (iEntry < it->second.size()) { + return it->second[iEntry]; + } + } + + return 0; +} + void SIOFileTOCRecordBlock::read(sio::read_device& device, sio::version_type) { int size; device.data(size); diff --git a/src/SIOFrameData.cc b/src/SIOFrameData.cc new file mode 100644 index 000000000..e7c87e9e6 --- /dev/null +++ b/src/SIOFrameData.cc @@ -0,0 +1,98 @@ +#include "podio/SIOFrameData.h" +#include "podio/SIOBlock.h" + +#include + +#include +#include + +namespace podio { +std::optional SIOFrameData::getCollectionBuffers(const std::string& name) { + unpackBuffers(); + + if (m_idTable.present(name)) { + // The collections that we read are not necessarily in the same order as + // they are in the collection id table. Hence, we cannot simply use the + // collection ID to index into the blocks + const auto& names = m_idTable.names(); + const auto nameIt = std::find(std::begin(names), std::end(names), name); + // collection indices start at 1! + const auto index = std::distance(std::begin(names), nameIt) + 1; + + m_availableBlocks[index] = 1; + return {dynamic_cast(m_blocks[index].get())->getBuffers()}; + } + + return std::nullopt; +} + +std::unique_ptr SIOFrameData::getParameters() { + unpackBuffers(); + m_availableBlocks[0] = 0; + return std::make_unique(std::move(m_parameters)); +} + +std::vector SIOFrameData::getAvailableCollections() { + unpackBuffers(); + std::vector collections; + for (size_t i = 0; i < m_blocks.size(); ++i) { + if (m_availableBlocks[i]) { + collections.push_back(m_idTable.name(i)); + } + } + + return collections; +} + +void SIOFrameData::unpackBuffers() { + // Only do the unpacking once. Use the block as proxy for deciding whether + // we have already unpacked things, since that is the main thing we do in + // here: create blocks and read the data into them + if (!m_blocks.empty()) { + return; + } + + if (m_idTable.empty()) { + readIdTable(); + } + + createBlocks(); + + sio::zlib_compression compressor; + sio::buffer uncBuffer{m_dataSize}; + compressor.uncompress(m_recBuffer.span(), uncBuffer); + sio::api::read_blocks(uncBuffer.span(), m_blocks); +} + +void SIOFrameData::createBlocks() { + m_blocks.reserve(m_typeNames.size() + 1); + // First block during writing is parameters / metadata, then collections + auto parameters = std::make_shared(); + parameters->metadata = &m_parameters; + m_blocks.push_back(parameters); + + for (size_t i = 0; i < m_typeNames.size(); ++i) { + const bool subsetColl = !m_subsetCollectionBits.empty() && m_subsetCollectionBits[i]; + auto blk = podio::SIOBlockFactory::instance().createBlock(m_typeNames[i], m_idTable.names()[i], subsetColl); + m_blocks.push_back(blk); + } + + m_availableBlocks.resize(m_blocks.size(), 1); +} + +void SIOFrameData::readIdTable() { + sio::buffer uncBuffer{m_tableSize}; + sio::zlib_compression compressor; + compressor.uncompress(m_tableBuffer.span(), uncBuffer); + + sio::block_list blocks; + blocks.emplace_back(std::make_shared()); + sio::api::read_blocks(uncBuffer.span(), blocks); + + auto* idTableBlock = static_cast(blocks[0].get()); + m_idTable = std::move(*idTableBlock->getTable()); + m_typeNames = idTableBlock->getTypeNames(); + m_subsetCollectionBits = idTableBlock->getSubsetCollectionBits(); +} + +} // namespace podio diff --git a/src/SIOFrameReader.cc b/src/SIOFrameReader.cc new file mode 100644 index 000000000..5b82f216d --- /dev/null +++ b/src/SIOFrameReader.cc @@ -0,0 +1,113 @@ +#include "podio/SIOFrameReader.h" +#include "podio/SIOBlock.h" + +#include +#include +#include + +#include + +namespace podio { + +namespace sio_utils { + // Read the record into a buffer and potentially uncompress it + std::pair readRecord(sio::ifstream& stream, bool decompress = true, + std::size_t initBufferSize = sio::mbyte) { + sio::record_info recInfo; + sio::buffer infoBuffer{sio::max_record_info_len}; + sio::buffer recBuffer{initBufferSize}; + sio::api::read_record_info(stream, recInfo, infoBuffer); + sio::api::read_record_data(stream, recInfo, recBuffer); + + if (decompress) { + sio::buffer uncBuffer{recInfo._uncompressed_length}; + sio::zlib_compression compressor; + compressor.uncompress(recBuffer.span(), uncBuffer); + return std::make_pair(std::move(uncBuffer), recInfo); + } + + return std::make_pair(std::move(recBuffer), recInfo); + } +} // namespace sio_utils + +SIOFrameReader::SIOFrameReader() { + auto& libLoader [[maybe_unused]] = SIOBlockLibraryLoader::instance(); +} + +void SIOFrameReader::openFile(const std::string& filename) { + m_stream.open(filename, std::ios::binary); + if (!m_stream.is_open()) { + SIO_THROW(sio::error_code::not_open, "Cannot open input file '" + filename + "' for reading"); + } + + // NOTE: reading TOC record first because that jumps back to the start of the file! + readFileTOCRecord(); + readPodioHeader(); +} + +std::unique_ptr SIOFrameReader::readNextEntry(const std::string& name) { + // Skip to where the next record of this name starts in the file, based on + // how many times we have already read this name + // + // NOTE: exploiting the fact that the operator[] of a map will create a + // default initialized entry for us if not present yet + const auto recordPos = m_tocRecord.getPosition(name, m_nameCtr[name]); + if (recordPos == 0) { + return nullptr; + } + m_stream.seekg(recordPos); + + auto [tableBuffer, tableInfo] = sio_utils::readRecord(m_stream, false); + auto [dataBuffer, dataInfo] = sio_utils::readRecord(m_stream, false); + + m_nameCtr[name]++; + + return std::make_unique(std::move(dataBuffer), dataInfo._uncompressed_length, std::move(tableBuffer), + tableInfo._uncompressed_length); +} + +unsigned SIOFrameReader::getEntries(const std::string& name) const { + return m_tocRecord.getNRecords(name); +} + +bool SIOFrameReader::readFileTOCRecord() { + // Check if there is a dedicated marker at the end of the file that tells us + // where the TOC actually starts + m_stream.seekg(-sio_helpers::SIOTocInfoSize, std::ios_base::end); + uint64_t firstWords{0}; + m_stream.read(reinterpret_cast(&firstWords), sizeof(firstWords)); + + const uint32_t marker = (firstWords >> 32) & 0xffffffff; + if (marker == sio_helpers::SIOTocMarker) { + const uint32_t position = firstWords & 0xffffffff; + m_stream.seekg(position); + + const auto& [uncBuffer, _] = sio_utils::readRecord(m_stream); + + sio::block_list blocks; + auto tocBlock = std::make_shared(); + tocBlock->record = &m_tocRecord; + blocks.push_back(tocBlock); + + sio::api::read_blocks(uncBuffer.span(), blocks); + + m_stream.seekg(0); + return true; + } + + m_stream.clear(); + m_stream.seekg(0); + return false; +} + +void SIOFrameReader::readPodioHeader() { + const auto& [buffer, _] = sio_utils::readRecord(m_stream, false, sizeof(podio::version::Version)); + + sio::block_list blocks; + blocks.emplace_back(std::make_shared()); + sio::api::read_blocks(buffer.span(), blocks); + + m_fileVersion = static_cast(blocks[0].get())->version; +} + +} // namespace podio diff --git a/src/SIOFrameWriter.cc b/src/SIOFrameWriter.cc new file mode 100644 index 000000000..b1c975037 --- /dev/null +++ b/src/SIOFrameWriter.cc @@ -0,0 +1,136 @@ +#include "podio/SIOFrameWriter.h" +#include "podio/CollectionBase.h" +#include "podio/CollectionIDTable.h" +#include "podio/Frame.h" +#include "podio/GenericParameters.h" +#include "podio/SIOBlock.h" + +#include +#include +#include +#include + +#include + +namespace podio { + +namespace sio_utils { + using StoreCollection = std::pair; + + std::shared_ptr createCollIDBlock(const std::vector& collections, + const podio::CollectionIDTable& collIdTable) { + // Need to make sure that the type names and subset collection bits are in + // the same order here! + std::vector types; + types.reserve(collections.size()); + std::vector subsetColl; + subsetColl.reserve(collections.size()); + std::vector names; + names.reserve(collections.size()); + std::vector ids; + ids.reserve(collections.size()); + + for (const auto& [name, coll] : collections) { + names.emplace_back(name); + ids.emplace_back(collIdTable.collectionID(name)); + types.emplace_back(coll->getValueTypeName()); + subsetColl.emplace_back(coll->isSubsetCollection()); + } + + return std::make_shared(std::move(names), std::move(ids), std::move(types), + std::move(subsetColl)); + } + + sio::block_list createBlocks(const std::vector& collections, + const podio::GenericParameters& parameters) { + sio::block_list blocks; + blocks.reserve(collections.size() + 1); // parameters + collections + + auto paramBlock = std::make_shared(); + // TODO: get rid of const_cast + paramBlock->metadata = const_cast(¶meters); + blocks.emplace_back(std::move(paramBlock)); + + for (const auto& [name, col] : collections) { + blocks.emplace_back(podio::SIOBlockFactory::instance().createBlock(col, name)); + } + + return blocks; + } + + // Write the passed record and return where it starts in the file + sio::ifstream::pos_type writeRecord(const sio::block_list& blocks, const std::string& recordName, + sio::ofstream& stream, std::size_t initBufferSize = sio::mbyte, + bool compress = true) { + auto buffer = sio::buffer{initBufferSize}; + auto recInfo = sio::api::write_record(recordName, buffer, blocks, 0); + + if (compress) { + // use zlib to compress the record into another buffer + sio::zlib_compression compressor; + compressor.set_level(6); // Z_DEFAULT_COMPRESSION==6 + auto comBuffer = sio::buffer{initBufferSize}; + sio::api::compress_record(recInfo, buffer, comBuffer, compressor); + + sio::api::write_record(stream, buffer.span(0, recInfo._header_length), comBuffer.span(), recInfo); + } else { + sio::api::write_record(stream, buffer.span(), recInfo); + } + + return recInfo._file_start; + } +} // namespace sio_utils + +SIOFrameWriter::SIOFrameWriter(const std::string& filename) { + m_stream.open(filename, std::ios::binary); + if (!m_stream.is_open()) { + SIO_THROW(sio::error_code::not_open, "Couldn't open output stream '" + filename + "'"); + } + + auto& libLoader [[maybe_unused]] = SIOBlockLibraryLoader::instance(); + + sio::block_list blocks; + blocks.emplace_back(std::make_shared(podio::version::build_version)); + // write the version uncompressed + sio_utils::writeRecord(blocks, "podio_header_info", m_stream, sizeof(podio::version::Version), false); +} + +void SIOFrameWriter::writeFrame(const podio::Frame& frame, const std::string& category) { + writeFrame(frame, category, frame.getAvailableCollections()); +} + +void SIOFrameWriter::writeFrame(const podio::Frame& frame, const std::string& category, + const std::vector& collsToWrite) { + std::vector collections; + collections.reserve(collsToWrite.size()); + for (const auto& name : collsToWrite) { + collections.emplace_back(name, frame.getCollectionForWrite(name)); + } + + // Write necessary metadata and the actual data into two different records. + // Otherwise we cannot easily unpack the data record, because necessary + // information is contained within the record. + sio::block_list tableBlocks; + tableBlocks.emplace_back(sio_utils::createCollIDBlock(collections, frame.getCollectionIDTableForWrite())); + m_tocRecord.addRecord(category, sio_utils::writeRecord(tableBlocks, category + "_HEADER", m_stream)); + + const auto blocks = sio_utils::createBlocks(collections, frame.getGenericParametersForWrite()); + sio_utils::writeRecord(blocks, category, m_stream); +} + +void SIOFrameWriter::finish() { + sio::block_list blocks; + blocks.emplace_back(std::make_shared(&m_tocRecord)); + + auto tocStartPos = sio_utils::writeRecord(blocks, sio_helpers::SIOTocRecordName, m_stream); + + // Now that we know the position of the TOC Record, put this information + // into a final marker that can be identified and interpreted when reading + // again + uint64_t finalWords = (((uint64_t)sio_helpers::SIOTocMarker) << 32) | ((uint64_t)tocStartPos & 0xffffffff); + m_stream.write(reinterpret_cast(&finalWords), sizeof(finalWords)); + + m_stream.close(); +} + +} // namespace podio diff --git a/src/SIOReader.cc b/src/SIOReader.cc index 39d43cb32..65430a3a2 100644 --- a/src/SIOReader.cc +++ b/src/SIOReader.cc @@ -25,10 +25,12 @@ CollectionBase* SIOReader::readCollection(const std::string& name) { readEvent(); } + // Have we unpacked this already? auto p = std::find_if(begin(m_inputs), end(m_inputs), [&name](const SIOReader::Input& t) { return t.second == name; }); if (p != end(m_inputs)) { + p->first->setID(m_table->collectionID(name)); p->first->prepareAfterRead(); return p->first; } @@ -87,8 +89,9 @@ void SIOReader::readEvent() { compressor.uncompress(m_rec_buffer.span(), m_unc_buffer); sio::api::read_blocks(m_unc_buffer.span(), m_blocks); - for (auto& [collection, name] : m_inputs) { - collection->setID(m_table->collectionID(name)); + for (size_t i = 1; i < m_blocks.size(); ++i) { + auto* blk = static_cast(m_blocks[i].get()); + m_inputs.emplace_back(blk->getCollection(), m_table->names()[i - 1]); } m_lastEventRead = m_eventNumber; @@ -132,7 +135,6 @@ void SIOReader::createBlocks() { const bool subsetColl = !m_subsetCollectionBits.empty() && m_subsetCollectionBits[i]; auto blk = podio::SIOBlockFactory::instance().createBlock(m_typeNames[i], m_table->names()[i], subsetColl); m_blocks.push_back(blk); - m_inputs.emplace_back(blk->getCollection(), m_table->names()[i]); } } diff --git a/src/SIOWriter.cc b/src/SIOWriter.cc index 3ac9d4258..782ddbd58 100644 --- a/src/SIOWriter.cc +++ b/src/SIOWriter.cc @@ -67,6 +67,7 @@ sio::block_list SIOWriter::createBlocks() const { for (const auto& name : m_collectionsToWrite) { const podio::CollectionBase* col{nullptr}; m_store->get(name, col); + col->prepareForWrite(); blocks.emplace_back(podio::SIOBlockFactory::instance().createBlock(col, name)); } diff --git a/src/rootUtils.h b/src/rootUtils.h index e8f87a5f9..5bce3d702 100644 --- a/src/rootUtils.h +++ b/src/rootUtils.h @@ -3,6 +3,7 @@ #include "podio/CollectionBase.h" #include "podio/CollectionBranches.h" +#include "podio/CollectionBuffers.h" #include "podio/CollectionIDTable.h" #include "TBranch.h" @@ -16,6 +17,42 @@ #include namespace podio::root_utils { +/** + * The name of the meta data tree in podio ROOT files. This tree mainly stores + * meta data that is necessary for ROOT based I/O. + */ +constexpr static auto metaTreeName = "podio_metadata"; + +/** + * The name of the branch in the TTree for each frame for storing the + * GenericParameters + */ +constexpr static auto paramBranchName = "PARAMETERS"; + +/** + * The name of the branch into which we store the build version of podio at the + * time of writing the file + */ +constexpr static auto versionBranchName = "PodioBuildVersion"; + +/** + * Name of the branch for storing the idTable for a given category in the meta + * data tree + */ +inline std::string idTableName(const std::string& category) { + constexpr static auto suffix = "___idTable"; + return category + suffix; +} + +/** + * Name of the branch for storing the collection info for a given category in + * the meta data tree + */ +inline std::string collInfoName(const std::string& category) { + constexpr static auto suffix = "___CollectionTypeInfo"; + return category + suffix; +} + // Workaround slow branch retrieval for 6.22/06 performance degradation // see: https://root-forum.cern.ch/t/serious-degradation-of-i-o-performance-from-6-20-04-to-6-22-06/43584/10 template @@ -23,6 +60,11 @@ TBranch* getBranch(Tree* chain, const char* name) { return static_cast(chain->GetListOfBranches()->FindObject(name)); } +template +TBranch* getBranch(Tree* chain, const std::string& name) { + return getBranch(chain, name.c_str()); +} + inline std::string refBranch(const std::string& name, size_t index) { return name + "#" + std::to_string(index); } @@ -31,8 +73,8 @@ inline std::string vecBranch(const std::string& name, size_t index) { return name + "_" + std::to_string(index); } -inline void setCollectionAddresses(podio::CollectionBase* collection, const CollectionBranches& branches) { - const auto collBuffers = collection->getBuffers(); +template +inline void setCollectionAddresses(const BufferT& collBuffers, const CollectionBranches& branches) { if (auto buffer = collBuffers.data) { branches.data->SetAddress(buffer); @@ -56,6 +98,19 @@ inline void setCollectionAddresses(podio::CollectionBase* collection, const Coll // collection using CollectionInfoT = std::tuple; +inline void readBranchesData(const CollectionBranches& branches, Long64_t entry) { + // Read all data + if (branches.data) { + branches.data->GetEntry(entry); + } + for (auto* br : branches.refs) { + br->GetEntry(entry); + } + for (auto* br : branches.vecs) { + br->GetEntry(entry); + } +} + /** * reconstruct the collection info from information that is available from other * trees in the file. diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index cde097bcc..ce31b0b19 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -35,7 +35,17 @@ function(CREATE_PODIO_TEST sourcefile additional_libs) ) endfunction() -set(root_dependent_tests write.cpp read.cpp read-multiple.cpp relation_range.cpp read_and_write.cpp read_and_write_associated.cpp write_timed.cpp read_timed.cpp) +set(root_dependent_tests + write.cpp + read.cpp + read-multiple.cpp + relation_range.cpp + read_and_write.cpp + read_and_write_associated.cpp + write_timed.cpp + read_timed.cpp + read_frame.cpp + write_frame_root.cpp) set(root_libs TestDataModelDict podio::podioRootIO) foreach( sourcefile ${root_dependent_tests} ) CREATE_PODIO_TEST(${sourcefile} "${root_libs}") @@ -111,7 +121,14 @@ endif() if (TARGET TestDataModelSioBlocks) - set(sio_dependent_tests write_sio.cpp read_sio.cpp read_and_write_sio.cpp write_timed_sio.cpp read_timed_sio.cpp) + set(sio_dependent_tests + write_sio.cpp + read_sio.cpp + read_and_write_sio.cpp + write_timed_sio.cpp + read_timed_sio.cpp + read_frame_sio.cpp + write_frame_sio.cpp) set(sio_libs podio::podioSioIO) foreach( sourcefile ${sio_dependent_tests} ) CREATE_PODIO_TEST(${sourcefile} "${sio_libs}") @@ -129,6 +146,7 @@ set_property(TEST read PROPERTY DEPENDS write) set_property(TEST read-multiple PROPERTY DEPENDS write) set_property(TEST read_and_write PROPERTY DEPENDS write) set_property(TEST read_timed PROPERTY DEPENDS write_timed) +set_property(TEST read_frame PROPERTY DEPENDS write_frame_root) add_executable(check_benchmark_outputs check_benchmark_outputs.cpp) target_link_libraries(check_benchmark_outputs ROOT::Tree) @@ -140,6 +158,7 @@ if (TARGET read_sio) set_property(TEST read_sio PROPERTY DEPENDS write_sio) set_property(TEST read_and_write_sio PROPERTY DEPENDS write_sio) set_property(TEST read_timed_sio PROPERTY DEPENDS write_timed_sio) + set_property(TEST read_frame_sio PROPERTY DEPENDS write_frame_sio) add_test(NAME check_benchmark_outputs_sio COMMAND check_benchmark_outputs write_benchmark_sio.root read_benchmark_sio.root) set_property(TEST check_benchmark_outputs_sio PROPERTY DEPENDS read_timed_sio write_timed_sio) @@ -159,7 +178,7 @@ set_property(TEST pyunittest PROPERTY DEPENDS write) configure_file(CTestCustom.cmake ${CMAKE_BINARY_DIR}/CTestCustom.cmake) find_package(Threads REQUIRED) -add_executable(unittest unittest.cpp) +add_executable(unittest unittest.cpp frame.cpp) target_link_libraries(unittest PUBLIC TestDataModel PRIVATE Catch2::Catch2WithMain Threads::Threads) # The unittests are a bit better and they are labelled so we can put together a @@ -184,6 +203,10 @@ if (USE_SANITIZER MATCHES "Memory(WithOrigin)?") # it fails to succesfully launch the executable and execute any test. Here # we just include them in order to have them show up as failing add_test(NAME unittest COMMAND unittest ${filter_tests}) + set_property(TEST unittest + PROPERTY ENVIRONMENT + LD_LIBRARY_PATH=${CMAKE_CURRENT_BINARY_DIR}:${CMAKE_BINARY_DIR}/src:$:$ENV{LD_LIBRARY_PATH} + ) endif() else() include(Catch) @@ -191,6 +214,9 @@ else() WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR} TEST_PREFIX "UT_" # make it possible to filter easily with -R ^UT TEST_SPEC ${filter_tests} # discover only tests that are known to not fail + PROPERTIES + ENVIRONMENT + LD_LIBRARY_PATH=${CMAKE_CURRENT_BINARY_DIR}:${CMAKE_BINARY_DIR}/src:$:$ENV{LD_LIBRARY_PATH} ) endif() diff --git a/tests/CTestCustom.cmake b/tests/CTestCustom.cmake index 79a9296d8..c0c3b04f5 100644 --- a/tests/CTestCustom.cmake +++ b/tests/CTestCustom.cmake @@ -21,12 +21,17 @@ if ((NOT "@FORCE_RUN_ALL_TESTS@" STREQUAL "ON") AND (NOT "@USE_SANITIZER@" STREQ read-multiple read-legacy-files + write_frame_root + read_frame + write_sio read_sio read_and_write_sio write_timed_sio read_timed_sio check_benchmark_outputs_sio + write_frame_sio + read_frame_sio write_ascii diff --git a/tests/frame.cpp b/tests/frame.cpp new file mode 100644 index 000000000..38dd4e3c6 --- /dev/null +++ b/tests/frame.cpp @@ -0,0 +1,361 @@ +#include "podio/Frame.h" + +#include "catch2/catch_test_macros.hpp" + +#include "datamodel/ExampleClusterCollection.h" +#include "datamodel/ExampleHitCollection.h" + +#include +#include +#include + +TEST_CASE("Frame collections", "[frame][basics]") { + auto event = podio::Frame(); + auto clusters = ExampleClusterCollection(); + clusters.create(3.14f); + clusters.create(42.0f); + + event.put(std::move(clusters), "clusters"); + + auto& coll = event.get("clusters"); + REQUIRE(coll[0].energy() == 3.14f); + REQUIRE(coll[1].energy() == 42.0f); +} + +TEST_CASE("Frame parameters", "[frame][basics]") { + auto event = podio::Frame(); + + event.putParameter("aString", "from a string literal"); + REQUIRE(event.getParameter("aString") == "from a string literal"); + + event.putParameter("someInts", {42, 123}); + const auto& ints = event.getParameter>("someInts"); + REQUIRE(ints.size() == 2); + REQUIRE(ints[0] == 42); + REQUIRE(ints[1] == 123); + + event.putParameter("someStrings", {"one", "two", "three"}); + const auto& strings = event.getParameter>("someStrings"); + REQUIRE(strings.size() == 3); + REQUIRE(strings[0] == "one"); + REQUIRE(strings[1] == "two"); + REQUIRE(strings[2] == "three"); +} + +// NOTE: Due to the extremly small tasks that are done in these tests, they will +// most likely succeed with a very high probability and only running with a +// ThreadSanitizer will detect race conditions, so make sure to have that +// enabled (-DUSE_SANITIZER=Thread) when working on these tests + +TEST_CASE("Frame collections multithreaded insert", "[frame][basics][multithread]") { + constexpr int nThreads = 10; + std::vector threads; + threads.reserve(10); + + auto frame = podio::Frame(); + + // Fill collections from different threads + for (int i = 0; i < nThreads; ++i) { + threads.emplace_back([&frame, i]() { + auto clusters = ExampleClusterCollection(); + clusters.create(i * 3.14); + clusters.create(i * 3.14); + frame.put(std::move(clusters), "clusters_" + std::to_string(i)); + + auto hits = ExampleHitCollection(); + hits.create(i * 100ULL); + hits.create(i * 100ULL); + hits.create(i * 100ULL); + frame.put(std::move(hits), "hits_" + std::to_string(i)); + }); + } + + for (auto& t : threads) { + t.join(); + } + + // Check the frame contents after all threads have finished + for (int i = 0; i < nThreads; ++i) { + auto& hits = frame.get("hits_" + std::to_string(i)); + REQUIRE(hits.size() == 3); + for (const auto h : hits) { + REQUIRE(h.cellID() == i * 100ULL); + } + + auto& clusters = frame.get("clusters_" + std::to_string(i)); + REQUIRE(clusters.size() == 2); + for (const auto c : clusters) { + REQUIRE(c.energy() == i * 3.14); + } + } +} + +// Helper function to create a frame in the tests below +auto createFrame() { + auto frame = podio::Frame(); + + frame.put(ExampleClusterCollection(), "emptyClusters"); + + // Create a few hits inline (to avoid having to have two identifiers) + auto& hits = frame.put( + []() { + auto coll = ExampleHitCollection(); + auto hit = coll.create(0x42ULL, 0., 0., 0., 0.); + auto hit2 = coll.create(0x123ULL, 1., 1., 1., 1.); + return coll; + }(), + "hits"); + + auto clusters = ExampleClusterCollection(); + auto cluster = clusters.create(3.14f); + cluster.addHits(hits[0]); + auto cluster2 = clusters.create(42.0f); + cluster2.addHits(hits[1]); + cluster2.addClusters(cluster); + + // Create a few clustes inline and relate them to the hits from above + frame.put(std::move(clusters), "clusters"); + + frame.putParameter("anInt", 42); + frame.putParameter("someFloats", {1.23f, 2.34f, 3.45f}); + + return frame; +} + +// Helper function to get names easily below +std::string makeName(const std::string& prefix, int index) { + return prefix + "_" + std::to_string(index); +} + +// The Catch2 assertions are not threadsafe +// https://github.com/catchorg/Catch2/blob/devel/docs/limitations.md#thread-safe-assertions +// This is a poor-mans implementation where it is our responsibility to only +// pass in unshared counters +void CHECK_INCREASE(const bool condition, int& counter) { + if (condition) { + counter++; + } +} + +TEST_CASE("Frame collections multithreaded insert and read", "[frame][basics][multithread]") { + constexpr int nThreads = 10; + std::vector threads; + threads.reserve(10); + + // create a pre-populated frame + auto frame = createFrame(); + + // The Catch2 assertions are not threadsafe: + // https://github.com/catchorg/Catch2/blob/devel/docs/limitations.md#thread-safe-assertions + // Count the successes in this array here and check them outside + // Once the Catch2 deficiencies are resolved, this can be changed again + std::array successes{}; + + // Fill collections from different threads + for (int i = 0; i < nThreads; ++i) { + threads.emplace_back([&frame, i, &successes]() { + auto clusters = ExampleClusterCollection(); + clusters.create(i * 3.14); + clusters.create(i * 3.14); + frame.put(std::move(clusters), makeName("clusters", i)); + + // Retrieve a few collections in between and do iust a very basic testing + auto& existingClu = frame.get("clusters"); + CHECK_INCREASE(existingClu.size() == 2, successes[i]); + auto& existingHits = frame.get("hits"); + CHECK_INCREASE(existingHits.size() == 2, successes[i]); + + auto hits = ExampleHitCollection(); + hits.create(i * 100ULL); + hits.create(i * 100ULL); + hits.create(i * 100ULL); + frame.put(std::move(hits), makeName("hits", i)); + + // Fill in a lot of new collections to trigger a rehashing of the + // internal map, which invalidates iterators + constexpr int nColls = 100; + for (int k = 0; k < nColls; ++k) { + frame.put(ExampleHitCollection(), "h_" + std::to_string(i) + "_" + std::to_string(k)); + } + }); + } + + for (auto& t : threads) { + t.join(); + } + + // Check the frame contents after all threads have finished + for (int i = 0; i < nThreads; ++i) { + // Check whether the insertsions are as expected + REQUIRE(successes[i] == 2); + + auto& hits = frame.get(makeName("hits", i)); + REQUIRE(hits.size() == 3); + for (const auto h : hits) { + REQUIRE(h.cellID() == i * 100ULL); + } + + auto& clusters = frame.get(makeName("clusters", i)); + REQUIRE(clusters.size() == 2); + for (const auto c : clusters) { + REQUIRE(c.energy() == i * 3.14); + } + } +} + +// Helper function to keep the tests below a bit easier to read and not having +// to repeat this bit all the time. This checks that the contents are the ones +// that would be expected from the createFrame above +void checkFrame(const podio::Frame& frame) { + auto& hits = frame.get("hits"); + REQUIRE(hits.size() == 2); + REQUIRE(hits[0].energy() == 0); + REQUIRE(hits[0].cellID() == 0x42ULL); + REQUIRE(hits[1].energy() == 1); + REQUIRE(hits[1].cellID() == 0x123ULL); + + REQUIRE(frame.get("emptyClusters").size() == 0); + + auto& clusters = frame.get("clusters"); + REQUIRE(clusters.size() == 2); + REQUIRE(clusters[0].energy() == 3.14f); + REQUIRE(clusters[0].Hits().size() == 1); + REQUIRE(clusters[0].Hits()[0] == hits[0]); + REQUIRE(clusters[0].Clusters().empty()); + + REQUIRE(clusters[1].energy() == 42.f); + REQUIRE(clusters[1].Hits().size() == 1); + REQUIRE(clusters[1].Hits()[0] == hits[1]); + REQUIRE(clusters[1].Clusters()[0] == clusters[0]); + + REQUIRE(frame.getParameter("anInt") == 42); + auto& floats = frame.getParameter>("someFloats"); + REQUIRE(floats.size() == 3); + REQUIRE(floats[0] == 1.23f); + REQUIRE(floats[1] == 2.34f); + REQUIRE(floats[2] == 3.45f); +} + +TEST_CASE("Frame movability", "[frame][move-semantics]") { + auto frame = createFrame(); + checkFrame(frame); // just to ensure that the setup is as expected + + SECTION("Move constructor") { + auto otherFrame = std::move(frame); + checkFrame(otherFrame); + } + + SECTION("Move assignment operator") { + auto otherFrame = podio::Frame(); + otherFrame = std::move(frame); + checkFrame(otherFrame); + } + + SECTION("Use after move construction") { + auto otherFrame = std::move(frame); // NOLINT(clang-analyzer-cplusplus.Move) clang-tidy and the Catch2 sections + // setup do not go along here + otherFrame.putParameter("aString", "Can add strings after move-constructing"); + REQUIRE(otherFrame.getParameter("aString") == "Can add strings after move-constructing"); + + otherFrame.put( + []() { + auto coll = ExampleHitCollection(); + coll.create(); + coll.create(); + coll.create(); + return coll; + }(), + "moreHits"); + + auto& hits = otherFrame.get("moreHits"); + REQUIRE(hits.size() == 3); + checkFrame(otherFrame); + } +} + +TEST_CASE("Frame parameters multithread insert", "[frame][basics][multithread]") { + // Test that parameter access is thread safe + constexpr int nThreads = 10; + std::vector threads; + threads.reserve(nThreads); + + auto frame = podio::Frame(); + + for (int i = 0; i < nThreads; ++i) { + threads.emplace_back([&frame, i]() { + frame.putParameter(makeName("int_par", i), i); + + frame.putParameter(makeName("float_par", i), (float)i); + + frame.putParameter(makeName("string_par", i), std::to_string(i)); + }); + } + + for (auto& t : threads) { + t.join(); + } + + for (int i = 0; i < nThreads; ++i) { + REQUIRE(frame.getParameter(makeName("int_par", i)) == i); + REQUIRE(frame.getParameter(makeName("float_par", i)) == (float)i); + REQUIRE(frame.getParameter(makeName("string_par", i)) == std::to_string(i)); + } +} + +TEST_CASE("Frame parameters multithread insert and read", "[frame][basics][multithread]") { + constexpr int nThreads = 10; + std::vector threads; + threads.reserve(nThreads); + + auto frame = podio::Frame(); + frame.putParameter("int_par", 42); + frame.putParameter("string_par", "some string"); + frame.putParameter("float_pars", {1.23f, 4.56f, 7.89f}); + + // The Catch2 assertions are not threadsafe: + // https://github.com/catchorg/Catch2/blob/devel/docs/limitations.md#thread-safe-assertions + // Count the successes in this array here and check them outside + // Once the Catch2 deficiencies are resolved, this can be changed again + std::array successes{}; + + for (int i = 0; i < nThreads; ++i) { + threads.emplace_back([&frame, i, &successes]() { + frame.putParameter(makeName("int", i), i); + frame.putParameter(makeName("float", i), (float)i); + + CHECK_INCREASE(frame.getParameter("int_par") == 42, successes[i]); + CHECK_INCREASE(frame.getParameter(makeName("float", i)) == (float)i, successes[i]); + + frame.putParameter(makeName("string", i), std::to_string(i)); + CHECK_INCREASE(frame.getParameter("string_par") == "some string", successes[i]); + + const auto& floatPars = frame.getParameter>("float_pars"); + CHECK_INCREASE(floatPars.size() == 3, successes[i]); + CHECK_INCREASE(floatPars[0] == 1.23f, successes[i]); + CHECK_INCREASE(floatPars[1] == 4.56f, successes[i]); + CHECK_INCREASE(floatPars[2] == 7.89f, successes[i]); + + // Fill in a lot of new parameters to trigger rehashing of the internal + // map, which invalidates iterators + constexpr int nParams = 100; + for (int k = 0; k < nParams; ++k) { + frame.putParameter(makeName("intPar", i) + std::to_string(k), i * k); + frame.putParameter(makeName("floatPar", i) + std::to_string(k), (float)i * k); + frame.putParameter(makeName("stringPar", i) + std::to_string(k), std::to_string(i * k)); + } + }); + } + + for (auto& t : threads) { + t.join(); + } + + for (int i = 0; i < nThreads; ++i) { + // Check the insertion successes + REQUIRE(successes[i] == 7); + + REQUIRE(frame.getParameter(makeName("int", i)) == i); + REQUIRE(frame.getParameter(makeName("float", i)) == (float)i); + REQUIRE(frame.getParameter(makeName("string", i)) == std::to_string(i)); + } +} diff --git a/tests/read_frame.cpp b/tests/read_frame.cpp new file mode 100644 index 000000000..5a0929627 --- /dev/null +++ b/tests/read_frame.cpp @@ -0,0 +1,7 @@ +#include "podio/ROOTFrameReader.h" + +#include "read_frame.h" + +int main() { + return read_frames("example_frame.root"); +} diff --git a/tests/read_frame.h b/tests/read_frame.h new file mode 100644 index 000000000..e9951c3b8 --- /dev/null +++ b/tests/read_frame.h @@ -0,0 +1,58 @@ +#ifndef PODIO_TESTS_READ_FRAME_H // NOLINT(llvm-header-guard): folder structure not suitable +#define PODIO_TESTS_READ_FRAME_H // NOLINT(llvm-header-guard): folder structure not suitable + +#include "read_test.h" + +#include "podio/Frame.h" + +#include + +template +int read_frames(const std::string& filename) { + auto reader = ReaderT(); + reader.openFile(filename); + + if (reader.currentFileVersion() != podio::version::build_version) { + std::cerr << "The podio build version could not be read back correctly. " + << "(expected:" << podio::version::build_version << ", actual: " << reader.currentFileVersion() << ")" + << std::endl; + return 1; + } + + if (reader.getEntries("events") != 10) { + std::cerr << "Could not read back the number of events correctly. " + << "(expected:" << 10 << ", actual: " << reader.getEntries("events") << ")" << std::endl; + return 1; + } + + if (reader.getEntries("events") != reader.getEntries("other_events")) { + std::cerr << "Could not read back the number of events correctly. " + << "(expected:" << 10 << ", actual: " << reader.getEntries("other_events") << ")" << std::endl; + return 1; + } + + // Read the frames in a different order than when writing them here to make + // sure that the writing/reading order does not impose any usage requirements + for (size_t i = 0; i < reader.getEntries("events"); ++i) { + auto frame = podio::Frame(reader.readNextEntry("events")); + processEvent(frame, i, reader.currentFileVersion()); + + auto otherFrame = podio::Frame(reader.readNextEntry("other_events")); + processEvent(otherFrame, i + 100, reader.currentFileVersion()); + } + + if (reader.readNextEntry("events")) { + std::cerr << "Trying to read more frame data than is present should return a nullptr" << std::endl; + return 1; + } + + std::cout << "========================================================\n" << std::endl; + if (reader.readNextEntry("not_present")) { + std::cerr << "Trying to read non-existant frame data should return a nullptr" << std::endl; + return 1; + } + + return 0; +} + +#endif // PODIO_TESTS_READ_FRAME_H diff --git a/tests/read_frame_sio.cpp b/tests/read_frame_sio.cpp new file mode 100644 index 000000000..cdc0b8854 --- /dev/null +++ b/tests/read_frame_sio.cpp @@ -0,0 +1,7 @@ +#include "podio/SIOFrameReader.h" + +#include "read_frame.h" + +int main() { + return read_frames("example_frame.sio"); +} diff --git a/tests/read_test.h b/tests/read_test.h index 9b12cff6a..0e54f0adf 100644 --- a/tests/read_test.h +++ b/tests/read_test.h @@ -26,6 +26,7 @@ #include #include #include +#include #include template @@ -38,25 +39,47 @@ bool check_fixed_width_value(FixedWidthT actual, FixedWidthT expected, const std return true; } -void processEvent(podio::EventStore& store, int eventNum, podio::version::Version fileVersion) { +template +static constexpr bool isEventStore = std::is_same_v; - const auto& evtMD = store.getEventMetaData(); - auto evtWeight = evtMD.getValue("UserEventWeight"); +template +void processEvent(StoreT& store, int eventNum, podio::version::Version fileVersion) { + + float evtWeight = -1; + if constexpr (isEventStore) { + const auto& evtMD = store.getEventMetaData(); + evtWeight = evtMD.template getValue("UserEventWeight"); + } else { + evtWeight = store.template getParameter("UserEventWeight"); + } if (evtWeight != (float)100. * eventNum) { std::cout << " read UserEventWeight: " << evtWeight << " - expected : " << (float)100. * eventNum << std::endl; throw std::runtime_error("Couldn't read event meta data parameters 'UserEventWeight'"); } + std::stringstream ss; ss << " event_number_" << eventNum; - const auto& evtMD2 = store.getEventMetaData(); - const auto& evtName = evtMD2.getValue("UserEventName"); + std::string evtName = ""; + if constexpr (isEventStore) { + const auto& evtMD = store.getEventMetaData(); + evtName = evtMD.template getValue("UserEventName"); + } else { + evtName = store.template getParameter("UserEventName"); + } + if (evtName != ss.str()) { std::cout << " read UserEventName: " << evtName << " - expected : " << ss.str() << std::endl; throw std::runtime_error("Couldn't read event meta data parameters 'UserEventName'"); } if (fileVersion > podio::version::Version{0, 14, 1}) { - const auto& someVectorData = evtMD.getValue>("SomeVectorData"); + std::vector someVectorData{}; + if constexpr (isEventStore) { + const auto& evtMD = store.getEventMetaData(); + someVectorData = evtMD.template getValue>("SomeVectorData"); + } else { + someVectorData = store.template getParameter>("SomeVectorData"); + } if (someVectorData.size() != 4) { throw std::runtime_error("Couldn't read event meta data parameters: 'SomeVectorData'"); } @@ -70,7 +93,7 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio try { // not assigning to a variable, because it will remain unused, we just want // the exception here - store.get("notthere"); + store.template get("notthere"); } catch (const std::runtime_error& err) { if (std::string(err.what()) != "No collection \'notthere\' is present in the EventStore") { throw std::runtime_error("Trying to get non present collection \'notthere' should throw an exception"); @@ -78,17 +101,23 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio } // read collection meta data - auto& hits = store.get("hits"); - auto colMD = store.getCollectionMetaData(hits.getID()); - const auto& es = colMD.getValue("CellIDEncodingString"); - if (es != std::string("system:8,barrel:3,layer:6,slice:5,x:-16,y:-16")) { - std::cout << " meta data from collection 'hits' with id = " << hits.getID() << " read CellIDEncodingString: " << es - << " - expected : system:8,barrel:3,layer:6,slice:5,x:-16,y:-16" << std::endl; - throw std::runtime_error("Couldn't read event meta data parameters 'CellIDEncodingString'"); + auto& hits = store.template get("hits"); + if constexpr (isEventStore) { + const auto& colMD = store.getCollectionMetaData(hits.getID()); + const auto& es = colMD.template getValue("CellIDEncodingString"); + if (es != std::string("system:8,barrel:3,layer:6,slice:5,x:-16,y:-16")) { + std::cout << " meta data from collection 'hits' with id = " << hits.getID() + << " read CellIDEncodingString: " << es << " - expected : system:8,barrel:3,layer:6,slice:5,x:-16,y:-16" + << std::endl; + throw std::runtime_error("Couldn't read event meta data parameters 'CellIDEncodingString'"); + } + + } else { + // TODO: Integrate this into the frame workflow somehow } if (fileVersion > podio::version::Version{0, 14, 0}) { - auto& hitRefs = store.get("hitRefs"); + auto& hitRefs = store.template get("hitRefs"); if (hitRefs.size() != hits.size()) { throw std::runtime_error("hit and subset hit collection do not have the same size"); } @@ -97,7 +126,7 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio } } - auto& clusters = store.get("clusters"); + auto& clusters = store.template get("clusters"); if (clusters.isValid()) { auto cluster = clusters[0]; for (auto i = cluster.Hits_begin(), end = cluster.Hits_end(); i != end; ++i) { @@ -107,7 +136,7 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio throw std::runtime_error("Collection 'clusters' should be present"); } - auto& mcps = store.get("mcparticles"); + auto& mcps = store.template get("mcparticles"); if (!mcps.isValid()) { throw std::runtime_error("Collection 'mcparticles' should be present"); } @@ -175,7 +204,7 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio // Load the subset collection first to ensure that it pulls in objects taht // have not been read yet - auto& mcpRefs = store.get("mcParticleRefs"); + auto& mcpRefs = store.template get("mcParticleRefs"); if (!mcpRefs.isValid()) { throw std::runtime_error("Collection 'mcParticleRefs' should be present"); } @@ -187,7 +216,7 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio } } - auto& moreMCs = store.get("moreMCs"); + auto& moreMCs = store.template get("moreMCs"); // First check that the two mc collections that we store are the same if (mcps.size() != moreMCs.size()) { @@ -220,11 +249,11 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio } // std::cout << "Fetching collection 'refs'" << std::endl; - auto& refs = store.get("refs"); + auto& refs = store.template get("refs"); if (refs.isValid()) { auto ref = refs[0]; for (auto cluster : ref.Clusters()) { - for (auto hit : cluster.Hits()) { + for (auto hit [[maybe_unused]] : cluster.Hits()) { // std::cout << " Referenced object has an energy of " << hit.energy() << std::endl; } } @@ -232,7 +261,7 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio throw std::runtime_error("Collection 'refs' should be present"); } // std::cout << "Fetching collection 'OneRelation'" << std::endl; - auto& rels = store.get("OneRelation"); + auto& rels = store.template get("OneRelation"); if (rels.isValid()) { // std::cout << "Referenced object has an energy of " << (*rels)[0].cluster().energy() << std::endl; } else { @@ -240,7 +269,7 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio } // std::cout << "Fetching collection 'WithVectorMember'" << std::endl; - auto& vecs = store.get("WithVectorMember"); + auto& vecs = store.template get("WithVectorMember"); if (vecs.isValid()) { if (vecs.size() != 2) { throw std::runtime_error("Collection 'WithVectorMember' should have two elements'"); @@ -267,13 +296,13 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio throw std::runtime_error("Collection 'WithVectorMember' should be present"); } - auto& comps = store.get("Component"); + auto& comps = store.template get("Component"); if (comps.isValid()) { auto comp = comps[0]; int a [[maybe_unused]] = comp.component().data.x + comp.component().data.z; } - auto& arrays = store.get("arrays"); + auto& arrays = store.template get("arrays"); if (arrays.isValid() && arrays.size() != 0) { auto array = arrays[0]; if (array.myArray(1) != eventNum) { @@ -289,9 +318,10 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio throw std::runtime_error("Collection 'arrays' should be present"); } - auto& nmspaces = store.get("WithNamespaceRelation"); - auto& copies = store.get("WithNamespaceRelationCopy"); - auto& cpytest = store.create("TestConstCopy"); + auto& nmspaces = store.template get("WithNamespaceRelation"); + auto& copies = store.template get("WithNamespaceRelationCopy"); + + auto cpytest = ex42::ExampleWithARelationCollection{}; if (nmspaces.isValid() && copies.isValid()) { for (size_t j = 0; j < nmspaces.size(); j++) { auto nmsp = nmspaces[j]; @@ -328,7 +358,7 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio } if (fileVersion >= podio::version::Version{0, 13, 1}) { - const auto& fixedWidthInts = store.get("fixedWidthInts"); + const auto& fixedWidthInts = store.template get("fixedWidthInts"); if (not fixedWidthInts.isValid() or fixedWidthInts.size() != 3) { throw std::runtime_error("Collection \'fixedWidthInts\' should be present and have 3 elements"); } @@ -362,7 +392,12 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio } if (fileVersion >= podio::version::Version{0, 13, 2}) { - auto& usrInts = store.get>("userInts"); + auto& usrInts = store.template get>("userInts"); + + if (usrInts.size() != (unsigned)eventNum + 1) { + throw std::runtime_error("Could not read all userInts properly (expected: " + std::to_string(eventNum + 1) + + ", actual: " + std::to_string(usrInts.size()) + ")"); + } auto& uivec = usrInts.vec(); int myInt = 0; @@ -379,7 +414,12 @@ void processEvent(podio::EventStore& store, int eventNum, podio::version::Versio } } - auto& usrDbl = store.get>("userDoubles"); + auto& usrDbl = store.template get>("userDoubles"); + if (usrDbl.size() != 100) { + throw std::runtime_error( + "Could not read all userDoubles properly (expected: 100, actual: " + std::to_string(usrDbl.size()) + ")"); + } + for (double d : usrDbl) { if (d != 42.) { throw std::runtime_error("Couldn't read userDoubles properly"); diff --git a/tests/unittest.cpp b/tests/unittest.cpp index 325bb81be..443584b42 100644 --- a/tests/unittest.cpp +++ b/tests/unittest.cpp @@ -614,6 +614,17 @@ auto createCollections(const size_t nElements = 3u) { userDataColl.push_back(3.14f * i); } + vecMemColl.prepareForWrite(); + auto buffers = vecMemColl.getBuffers(); + auto vecBuffers = buffers.vectorMembers; + auto thisVec = (*vecBuffers)[0].second; + + // const auto floatVec = podio::CollectionWriteBuffers::asVector(thisVec); + const auto floatVec2 = podio::CollectionReadBuffers::asVector(thisVec); + + // std::cout << floatVec->size() << '\n'; + std::cout << "** " << floatVec2->size() << " vs " << vecMemColl.size() << '\n'; + return colls; } @@ -689,6 +700,9 @@ void checkCollections(/*const*/ ExampleHitCollection& hits, /*const*/ ExampleClu } } +template +struct TD; + TEST_CASE("Move-only collections", "[collections][move-semantics]") { // Setup a few collections that will be used throughout below auto [hitColl, clusterColl, vecMemColl, userDataColl] = createCollections(); @@ -731,6 +745,20 @@ TEST_CASE("Move-only collections", "[collections][move-semantics]") { auto newClusters = std::move(clusterColl); vecMemColl.prepareForWrite(); + auto buffers = vecMemColl.getBuffers(); + auto vecBuffers = buffers.vectorMembers; + auto thisVec = (*vecBuffers)[0].second; + + const auto floatVec = podio::CollectionWriteBuffers::asVector(thisVec); + const auto floatVec2 = podio::CollectionReadBuffers::asVector(thisVec); + + std::cout << floatVec->size() << '\n'; + std::cout << floatVec2->size() << '\n'; + + // auto vecBuffers = buffers.vectorMembers; + // const auto vecBuffer = podio::CollectionWriteBuffers::asVector((*vecBuffers)[0].second); + // TD td; + // REQUIRE(vecBuffer->size() == 2); auto newVecMems = std::move(vecMemColl); userDataColl.prepareForWrite(); @@ -739,6 +767,9 @@ TEST_CASE("Move-only collections", "[collections][move-semantics]") { checkCollections(newHits, newClusters, newVecMems, newUserData); } + SECTION("Moved collections can be prepared") { + } + SECTION("Prepared collections can be move assigned") { hitColl.prepareForWrite(); clusterColl.prepareForWrite(); diff --git a/tests/write_frame.h b/tests/write_frame.h new file mode 100644 index 000000000..f7ccdc267 --- /dev/null +++ b/tests/write_frame.h @@ -0,0 +1,390 @@ +#ifndef PODIO_TESTS_WRITE_FRAME_H // NOLINT(llvm-header-guard): folder structure not suitable +#define PODIO_TESTS_WRITE_FRAME_H // NOLINT(llvm-header-guard): folder structure not suitable + +#include "datamodel/EventInfoCollection.h" +#include "datamodel/ExampleClusterCollection.h" +#include "datamodel/ExampleHitCollection.h" +#include "datamodel/ExampleMCCollection.h" +#include "datamodel/ExampleReferencingTypeCollection.h" +#include "datamodel/ExampleWithARelationCollection.h" +#include "datamodel/ExampleWithArrayCollection.h" +#include "datamodel/ExampleWithFixedWidthIntegersCollection.h" +#include "datamodel/ExampleWithNamespaceCollection.h" +#include "datamodel/ExampleWithOneRelationCollection.h" +#include "datamodel/ExampleWithVectorMemberCollection.h" + +#include "podio/Frame.h" +#include "podio/UserDataCollection.h" + +#include +#include +#include + +static const std::vector collsToWrite = {"mcparticles", + "moreMCs", + "arrays", + "mcParticleRefs", + "hits", + "hitRefs", + "refs", + "refs2", + "clusters", + "OneRelation", + "info", + "WithVectorMember", + "fixedWidthInts", + "userInts", + "userDoubles", + "WithNamespaceMember", + "WithNamespaceRelation", + "WithNamespaceRelationCopy"}; + +auto createMCCollection() { + auto mcps = ExampleMCCollection(); + + // ---- add some MC particles ---- + auto mcp0 = mcps.create(); + auto mcp1 = mcps.create(); + auto mcp2 = mcps.create(); + auto mcp3 = mcps.create(); + auto mcp4 = mcps.create(); + auto mcp5 = mcps.create(); + auto mcp6 = mcps.create(); + auto mcp7 = mcps.create(); + auto mcp8 = mcps.create(); + auto mcp9 = mcps.create(); + + auto mcp = mcps[0]; + mcp.adddaughters(mcps[2]); + mcp.adddaughters(mcps[3]); + mcp.adddaughters(mcps[4]); + mcp.adddaughters(mcps[5]); + mcp = mcps[1]; + mcp.adddaughters(mcps[2]); + mcp.adddaughters(mcps[3]); + mcp.adddaughters(mcps[4]); + mcp.adddaughters(mcps[5]); + mcp = mcps[2]; + mcp.adddaughters(mcps[6]); + mcp.adddaughters(mcps[7]); + mcp.adddaughters(mcps[8]); + mcp.adddaughters(mcps[9]); + mcp = mcps[3]; + mcp.adddaughters(mcps[6]); + mcp.adddaughters(mcps[7]); + mcp.adddaughters(mcps[8]); + mcp.adddaughters(mcps[9]); + + //--- now fix the parent relations + // use a range-based for loop here to see if we get mutable objects from the + // begin/end iterators + for (auto mc : mcps) { + for (auto p : mc.daughters()) { + int dIndex = p.getObjectID().index; + auto d = mcps[dIndex]; + d.addparents(p); + } + } + + return mcps; +} + +auto createArrayCollection(int i) { + auto arrays = ExampleWithArrayCollection(); + + std::array arrayTest = {0, 0, 2, 3}; + std::array arrayTest2 = {4, 4, 2 * static_cast(i)}; + NotSoSimpleStruct a; + a.data.p = arrayTest2; + ex2::NamespaceStruct nstruct; + nstruct.x = static_cast(i); + std::array structArrayTest = {nstruct, nstruct, nstruct, nstruct}; + auto array = MutableExampleWithArray(a, arrayTest, arrayTest, arrayTest, arrayTest, structArrayTest); + array.myArray(1, i); + array.arrayStruct(a); + arrays.push_back(array); + + return arrays; +} + +auto createMCRefCollection(const ExampleMCCollection& mcps, const ExampleMCCollection& moreMCs) { + auto mcpsRefs = ExampleMCCollection(); + mcpsRefs.setSubsetCollection(); + // ----------------- add all "odd" mc particles into a subset collection + for (auto p : mcps) { + if (p.id() % 2) { + mcpsRefs.push_back(p); + } + } + // ----------------- add the "even" counterparts from a different collection + for (auto p : moreMCs) { + if (p.id() % 2 == 0) { + mcpsRefs.push_back(p); + } + } + + if (mcpsRefs.size() != mcps.size()) { + throw std::runtime_error( + "The mcParticleRefs collection should now contain as many elements as the mcparticles collection"); + } + + return mcpsRefs; +} + +auto createHitCollection(int i) { + ExampleHitCollection hits; + + auto hit1 = ExampleHit(0xbad, 0., 0., 0., 23. + i); + auto hit2 = ExampleHit(0xcaffee, 1., 0., 0., 12. + i); + + hits.push_back(hit1); + hits.push_back(hit2); + + return hits; +} + +auto createHitRefCollection(const ExampleHitCollection& hits) { + ExampleHitCollection hitRefs; + hitRefs.setSubsetCollection(); + + hitRefs.push_back(hits[1]); + hitRefs.push_back(hits[0]); + + return hitRefs; +} + +auto createClusterCollection(const ExampleHitCollection& hits) { + ExampleClusterCollection clusters; + + auto cluster = MutableExampleCluster(); + auto clu0 = MutableExampleCluster(); + auto clu1 = MutableExampleCluster(); + + auto hit1 = hits[0]; + auto hit2 = hits[1]; + + clu0.addHits(hit1); + clu0.energy(hit1.energy()); + clu1.addHits(hit2); + clu1.energy(hit2.energy()); + cluster.addHits(hit1); + cluster.addHits(hit2); + cluster.energy(hit1.energy() + hit2.energy()); + cluster.addClusters(clu0); + cluster.addClusters(clu1); + + clusters.push_back(clu0); + clusters.push_back(clu1); + clusters.push_back(cluster); + + return clusters; +} + +auto createReferencingCollections(const ExampleClusterCollection& clusters) { + auto retType = std::tuple(); + auto& [refs, refs2] = retType; + + auto ref = MutableExampleReferencingType(); + refs.push_back(ref); + + auto ref2 = ExampleReferencingType(); + refs2.push_back(ref2); + + ref.addClusters(clusters[2]); + ref.addRefs(ref2); + + auto cyclic = MutableExampleReferencingType(); + cyclic.addRefs(cyclic); + refs.push_back(cyclic); + + return retType; +} + +auto createOneRelCollection(const ExampleClusterCollection& clusters) { + ExampleWithOneRelationCollection oneRels; + + auto oneRel = MutableExampleWithOneRelation(); + oneRel.cluster(clusters[2]); + oneRels.push_back(oneRel); + + // write non-filled relation + auto oneRelEmpty = ExampleWithOneRelation(); + oneRels.push_back(oneRelEmpty); + + return oneRels; +} + +auto createVectorMemberCollection(int i) { + ExampleWithVectorMemberCollection vecs; + + auto vec = MutableExampleWithVectorMember(); + vec.addcount(i); + vec.addcount(i + 10); + vecs.push_back(vec); + auto vec1 = MutableExampleWithVectorMember(); + vec1.addcount(i + 1); + vec1.addcount(i + 11); + vecs.push_back(vec1); + + return vecs; +} + +auto createInfoCollection(int i) { + EventInfoCollection info; + + auto item1 = MutableEventInfo(); + item1.Number(i); + info.push_back(item1); + + return info; +} + +auto createFixedWidthCollection() { + auto fixedWidthInts = ExampleWithFixedWidthIntegersCollection(); + + auto maxValues = fixedWidthInts.create(); + maxValues.fixedI16(std::numeric_limits::max()); // 2^(16 - 1) - 1 == 32767 + maxValues.fixedU32(std::numeric_limits::max()); // 2^32 - 1 == 4294967295 + maxValues.fixedU64(std::numeric_limits::max()); // 2^64 - 1 == 18446744073709551615 + auto& maxComp = maxValues.fixedWidthStruct(); + maxComp.fixedUnsigned16 = std::numeric_limits::max(); // 2^16 - 1 == 65535 + maxComp.fixedInteger64 = std::numeric_limits::max(); // 2^(64 -1) - 1 == 9223372036854775807 + maxComp.fixedInteger32 = std::numeric_limits::max(); // 2^(32 - 1) - 1 == 2147483647 + + auto minValues = fixedWidthInts.create(); + minValues.fixedI16(std::numeric_limits::min()); // -2^(16 - 1) == -32768 + minValues.fixedU32(std::numeric_limits::min()); // 0 + minValues.fixedU64(std::numeric_limits::min()); // 0 + auto& minComp = minValues.fixedWidthStruct(); + minComp.fixedUnsigned16 = std::numeric_limits::min(); // 0 + minComp.fixedInteger64 = std::numeric_limits::min(); // -2^(64 - 1) == -9223372036854775808 + minComp.fixedInteger32 = std::numeric_limits::min(); // -2^(32 - 1) == -2147483648 + + auto arbValues = fixedWidthInts.create(); + arbValues.fixedI16(-12345); + arbValues.fixedU32(1234567890); + arbValues.fixedU64(1234567890123456789); + auto& arbComp = arbValues.fixedWidthStruct(); + arbComp.fixedUnsigned16 = 12345; + arbComp.fixedInteger32 = -1234567890; + arbComp.fixedInteger64 = -1234567890123456789ll; + + return fixedWidthInts; +} + +auto createUserDataCollections(int i) { + auto retType = std::tuple, podio::UserDataCollection>(); + auto& [usrInts, usrDoubles] = retType; + + // add some plain ints as user data + usrInts.resize(i + 1); + int myInt = 0; + for (auto& iu : usrInts) { + iu = myInt++; + } + + // and some user double values + unsigned nd = 100; + usrDoubles.resize(nd); + for (unsigned id = 0; id < nd; ++id) { + usrDoubles[id] = 42.; + } + + return retType; +} + +auto createNamespaceRelationCollection(int i) { + auto retVal = std::tuple{}; + auto& [namesps, namesprels, cpytest] = retVal; + + for (int j = 0; j < 5; j++) { + auto rel = ex42::MutableExampleWithARelation(); + rel.number(0.5 * j); + auto exWithNamesp = ex42::MutableExampleWithNamespace(); + exWithNamesp.component().x = i; + exWithNamesp.component().y = 1000 * i; + namesps.push_back(exWithNamesp); + if (j != 3) { // also check for empty relations + rel.ref(exWithNamesp); + for (int k = 0; k < 5; k++) { + auto namesp = ex42::MutableExampleWithNamespace(); + namesp.x(3 * k); + namesp.component().y = k; + namesps.push_back(namesp); + rel.addrefs(namesp); + } + } + namesprels.push_back(rel); + } + for (auto&& namesprel : namesprels) { + cpytest.push_back(namesprel.clone()); + } + + return retVal; +} + +podio::Frame makeFrame(int iFrame) { + podio::Frame frame{}; + + frame.put(createArrayCollection(iFrame), "arrays"); + frame.put(createVectorMemberCollection(iFrame), "WithVectorMember"); + frame.put(createInfoCollection(iFrame), "info"); + frame.put(createFixedWidthCollection(), "fixedWidthInts"); + + auto& mcps = frame.put(createMCCollection(), "mcparticles"); + + ExampleMCCollection moreMCs{}; + for (const auto&& mc : mcps) { + moreMCs.push_back(mc.clone()); + } + auto& otherMCs = frame.put(std::move(moreMCs), "moreMCs"); + frame.put(createMCRefCollection(mcps, otherMCs), "mcParticleRefs"); + + const auto& hits = frame.put(createHitCollection(iFrame), "hits"); + frame.put(createHitRefCollection(hits), "hitRefs"); + + const auto& clusters = frame.put(createClusterCollection(hits), "clusters"); + + auto [refs, refs2] = createReferencingCollections(clusters); + frame.put(std::move(refs), "refs"); + frame.put(std::move(refs2), "refs2"); + + frame.put(createOneRelCollection(clusters), "OneRelation"); + + auto [usrInts, usrDoubles] = createUserDataCollections(iFrame); + frame.put(std::move(usrInts), "userInts"); + frame.put(std::move(usrDoubles), "userDoubles"); + + auto [namesps, namespsrels, cpytest] = createNamespaceRelationCollection(iFrame); + frame.put(std::move(namesps), "WithNamespaceMember"); + frame.put(std::move(namespsrels), "WithNamespaceRelation"); + frame.put(std::move(cpytest), "WithNamespaceRelationCopy"); + + // Parameters + frame.putParameter("anInt", 42 + iFrame); + frame.putParameter("UserEventWeight", 100.f * iFrame); + frame.putParameter("UserEventName", " event_number_" + std::to_string(iFrame)); + frame.putParameter("SomeVectorData", {1, 2, 3, 4}); + + return frame; +} + +template +void write_frames(const std::string& filename) { + WriterT writer(filename); + + for (int i = 0; i < 10; ++i) { + auto frame = makeFrame(i); + writer.writeFrame(frame, "events", collsToWrite); + } + + for (int i = 100; i < 110; ++i) { + auto frame = makeFrame(i); + writer.writeFrame(frame, "other_events"); + } + + writer.finish(); +} + +#endif // PODIO_TESTS_WRITE_FRAME_H diff --git a/tests/write_frame_root.cpp b/tests/write_frame_root.cpp new file mode 100644 index 000000000..fd1d89beb --- /dev/null +++ b/tests/write_frame_root.cpp @@ -0,0 +1,8 @@ +#include "write_frame.h" + +#include "podio/ROOTFrameWriter.h" + +int main(int, char**) { + write_frames("example_frame.root"); + return 0; +} diff --git a/tests/write_frame_sio.cpp b/tests/write_frame_sio.cpp new file mode 100644 index 000000000..31df08171 --- /dev/null +++ b/tests/write_frame_sio.cpp @@ -0,0 +1,8 @@ +#include "write_frame.h" + +#include "podio/SIOFrameWriter.h" + +int main(int, char**) { + write_frames("example_frame.sio"); + return 0; +}