Along with an index, these chunks form a GDTable. After performing a groupby operation the data will be shuffled accordingly and new 'chunks' (Dagger partitions) containing only the data belonging to specific keys will be created. Provided table input will be partitioned according to either a chunksize argument or the existing partitioning (using the Tables.partitions interface). The diagram below presents a simple visual explanation of how the DTable and GDTable (grouped DTable) are built. To further support this goal, the set of operations that can be performed on a DTable is generic and only relies on interfaces offered by Tables.jl. Any Tables.jl compatible table type can be used for internal storage, which allows for greater composability with the ecosystem, and all of the advantages of the chosen table type. Any Tables.jl compatible source can be ingested by the DTable, and the DTable can also act as a sink in case you move the data somewhere else (such as to a CSV).Ī key feature is that the DTable doesn't use any dedicated structure for storing the table data in memory. It's built on top of Dagger.jl, which enables it to work in any worker and thread setup by letting Dagger take care of task scheduling and memory management. The DTable is a table structure providing distributed partitioning of data, and parallelization of operations performed on it, in any supported environment. Implementation details (for interested users).

Basic operations ( map, filter, reduce).Initial performance comparison (multithreaded).

Lighttable packaged julia