<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" href="assets/xml/rss.xsl" media="all"?><rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Blosc Home Page </title><link>https://blosc.org/</link><description>Blosc, an extremely fast, multi-threaded, meta-compressor library</description><atom:link href="https://blosc.org/rss.xml" rel="self" type="application/rss+xml"></atom:link><language>en</language><copyright>Contents © 2026 &lt;a href="mailto:blosc@blosc.org"&gt;The Blosc Developers&lt;/a&gt; </copyright><lastBuildDate>Thu, 19 Nov 2026 12:18:26 GMT</lastBuildDate><generator>Nikola (getnikola.com)</generator><docs>http://blogs.law.harvard.edu/tech/rss</docs><item><title>Sovereign Tech Agency Award for Blosc2</title><link>https://blosc.org/posts/sta-award-blosc2/</link><dc:creator>Francesc Alted, Scott Prater</dc:creator><description>&lt;div&gt;&lt;p&gt;We are thrilled to announce that the Blosc2 project has been awarded a contract of &lt;strong&gt;€286,000&lt;/strong&gt; from the &lt;a href="https://www.sovereign.tech/"&gt;Sovereign Tech Agency&lt;/a&gt; (STA) to modernize, expand, and harden our core C foundation (&lt;strong&gt;C-Blosc2&lt;/strong&gt;).&lt;/p&gt;
&lt;p&gt;Over the next 15 months, this funding will allow us to tackle long-standing architectural bottlenecks, decouple high-level capabilities from Python, formalize open binary specifications, and establish enterprise-grade security and compatibility testing across the entire Blosc ecosystem.&lt;/p&gt;
&lt;p&gt;&lt;img src="https://blosc.org/images/Blosc2-Project-STA.png" alt="Main Goals of the Blosc2 Project for the Sovereign Tech Agency" style="width: 100%; display: block; margin: 0 auto;"&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href="https://blosc.org/posts/sta-award-blosc2/"&gt;Read more…&lt;/a&gt; (6 min remaining to read)&lt;/p&gt;&lt;/div&gt;</description><category>award</category><category>Blosc2</category><category>C-Blosc2</category><category>Sovereign-Tech-Agency</category><guid>https://blosc.org/posts/sta-award-blosc2/</guid><pubDate>Tue, 17 Nov 2026 09:45:00 GMT</pubDate></item><item><title>When More Threads Make Things Slower</title><link>https://blosc.org/posts/when-more-threads-make-things-slower/</link><dc:creator>Johnny Kao</dc:creator><description>&lt;div&gt;&lt;p&gt;&lt;em&gt;Why matching the number of workers to the available compression blocks can make small requests much faster.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;More threads usually help compression when there is enough data to process in parallel. For a small request, however, the cost of preparing those threads can exceed the useful work. While working on &lt;a href="https://github.com/Blosc/c-blosc2/pull/805"&gt;C-Blosc2 PR #805&lt;/a&gt;, I found a case where a &lt;strong&gt;16 KiB compression request with sixteen configured threads&lt;/strong&gt; took about &lt;strong&gt;710 µs&lt;/strong&gt; with a fresh context. After the change, the same measured operation took about &lt;strong&gt;18 µs&lt;/strong&gt;, almost &lt;strong&gt;39× faster&lt;/strong&gt;. This is a result for that particular request and context lifecycle, not a 39× increase in the compression algorithm's throughput.&lt;/p&gt;
&lt;p&gt;The improvement came from two decisions that are easy to distinguish: &lt;strong&gt;skip parallel startup when a request contains only one block&lt;/strong&gt;, and &lt;strong&gt;avoid scheduling workers that have no blocks to process&lt;/strong&gt; when a request contains only a few blocks. Neither requires changing how the data is compressed. To understand the results, we first need to look at what each decision avoids, and then at the tests used to evaluate them.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://blosc.org/posts/when-more-threads-make-things-slower/"&gt;Read more…&lt;/a&gt; (5 min remaining to read)&lt;/p&gt;&lt;/div&gt;</description><category>Blosc2</category><category>compression</category><category>performance</category><category>threading</category><guid>https://blosc.org/posts/when-more-threads-make-things-slower/</guid><pubDate>Mon, 12 Oct 2026 13:00:00 GMT</pubDate></item><item><title>Common Interface for Remote and Local Datasets</title><link>https://blosc.org/posts/remote-local-objects/</link><dc:creator>Francesc Alted</dc:creator><description>&lt;div&gt;&lt;p&gt;I am happy to announce that the new &lt;strong&gt;Python-Blosc2 4.14&lt;/strong&gt; brings a common interface to remote Parquet tables, Zarr and HDF5 arrays, PyTables tables, and indeed, native Blosc2 containers.
Most interestingly, a local cache keeps fetched data available for the next request/query; moreover, if cache is persistent, you can reuse the downloaded data throughout different sessions.&lt;/p&gt;
&lt;h3&gt;TL;DR&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;One opener for everything&lt;/strong&gt;: &lt;code&gt;blosc2.open()&lt;/code&gt; provides a unified interface across remote Parquet, Zarr, HDF5, PyTables, and native Blosc2 datasets over HTTP, S3, or any &lt;code&gt;fsspec&lt;/code&gt; filesystem.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Zero full downloads&lt;/strong&gt;: Stream only the exact chunks, row groups, or physical columns your queries touch—no need to pull gigabytes of data to inspect a slice.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Smart persistent caching&lt;/strong&gt;: Passing &lt;code&gt;cache_dir=...&lt;/code&gt; automatically caches fetched data locally across sessions and processes, eliminating redundant network roundtrips.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Index-accelerated remote queries&lt;/strong&gt;: Remote queries automatically exploit persisted PyTables and Blosc2 indexes without fetching entire columns or rebuilding indexes locally.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ecosystem interoperability&lt;/strong&gt;: Combine remote arrays across different formats directly in lazy expressions (&lt;code&gt;b + z * h5&lt;/code&gt;), or stream Arrow batches into DuckDB and Polars.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;To whet your appetite, here is a practical tour using public datasets on Backblaze B2 (which is fully compatible with S3).
Here we are going to exercise the HTTP path, but all the protocols supported by &lt;a href="https://filesystem-spec.readthedocs.io/en/latest/index.html"&gt;fsspec&lt;/a&gt; (like S3, GDrive, GCS, SFTP and many others) should be supported too.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://blosc.org/posts/remote-local-objects/"&gt;Read more…&lt;/a&gt; (4 min remaining to read)&lt;/p&gt;&lt;/div&gt;</description><category>arrow</category><category>Blosc2</category><category>cache</category><category>ctable</category><category>hdf5</category><category>parquet</category><category>pytables</category><category>remote</category><category>zarr</category><guid>https://blosc.org/posts/remote-local-objects/</guid><pubDate>Wed, 30 Sep 2026 10:45:00 GMT</pubDate></item><item><title>Not an island: bringing compression to the tabular ecosystem</title><link>https://blosc.org/posts/not-an-island-tabular-ecosystem/</link><dc:creator>Francesc Alted</dc:creator><description>&lt;p&gt;A compression library can go one of two ways: try to be everything (its own dataframe, its own query engine, its own format that nothing else reads), or be a fast, compact layer that slots underneath the tools you already use. Python-Blosc2 has initially leaned toward the first path, but this is changing quite dramatically.  During the past year, we put a lot of effort in making Python-Blosc2 as much compatible as possible with the &lt;a class="reference external" href="https://data-apis.org/array-api/latest/"&gt;array API initiative&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;And today's 4.9.1 release pushes compatibility with the tabular library ecosystem further than ever: &lt;a class="reference external" href="https://blosc.org/python-blosc2/reference/ctable.html"&gt;CTable&lt;/a&gt;, our compressed table container, now plugs straight into Arrow, pandas, Polars and DuckDB.&lt;/p&gt;
&lt;section id="speaking-arrow"&gt;
&lt;h2&gt;Speaking Arrow&lt;/h2&gt;
&lt;p&gt;The star of this release is support for the &lt;a class="reference external" href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html"&gt;Arrow PyCapsule protocol&lt;/a&gt; — the lingua franca of modern tabular tools. In plain terms: any Arrow-aware tool can now read a &lt;code class="docutils literal"&gt;CTable&lt;/code&gt; directly, streaming it in small batches, without you converting anything first. Here is the same query — average fare per taxi company — asked five different ways, all against the same compressed file:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code python"&gt;&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-1" name="rest_code_958430c0047b4710a757bfc84ecf7230-1" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-1"&gt;&lt;/a&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;blosc2&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-2" name="rest_code_958430c0047b4710a757bfc84ecf7230-2" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-2"&gt;&lt;/a&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-3" name="rest_code_958430c0047b4710a757bfc84ecf7230-3" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-3"&gt;&lt;/a&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;blosc2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CTable&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"trips.b2z"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-4" name="rest_code_958430c0047b4710a757bfc84ecf7230-4" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-4"&gt;&lt;/a&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-5" name="rest_code_958430c0047b4710a757bfc84ecf7230-5" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-5"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# With Blosc2 itself...&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-6" name="rest_code_958430c0047b4710a757bfc84ecf7230-6" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-6"&gt;&lt;/a&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"company"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"fare"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-7" name="rest_code_958430c0047b4710a757bfc84ecf7230-7" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-7"&gt;&lt;/a&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-8" name="rest_code_958430c0047b4710a757bfc84ecf7230-8" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-8"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# ...or in SQL, with DuckDB reading the compressed table directly&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-9" name="rest_code_958430c0047b4710a757bfc84ecf7230-9" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-9"&gt;&lt;/a&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;duckdb&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-10" name="rest_code_958430c0047b4710a757bfc84ecf7230-10" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-10"&gt;&lt;/a&gt;&lt;span class="n"&gt;duckdb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"SELECT company, avg(fare) FROM t GROUP BY company"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;show&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-11" name="rest_code_958430c0047b4710a757bfc84ecf7230-11" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-11"&gt;&lt;/a&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-12" name="rest_code_958430c0047b4710a757bfc84ecf7230-12" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-12"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# ...or with Polars&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-13" name="rest_code_958430c0047b4710a757bfc84ecf7230-13" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-13"&gt;&lt;/a&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;polars&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;pl&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-14" name="rest_code_958430c0047b4710a757bfc84ecf7230-14" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-14"&gt;&lt;/a&gt;&lt;span class="n"&gt;pl&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DataFrame&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"company"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;agg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pl&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"fare"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-15" name="rest_code_958430c0047b4710a757bfc84ecf7230-15" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-15"&gt;&lt;/a&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-16" name="rest_code_958430c0047b4710a757bfc84ecf7230-16" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-16"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# ...or with pandas &amp;gt;= 3.0&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-17" name="rest_code_958430c0047b4710a757bfc84ecf7230-17" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-17"&gt;&lt;/a&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;pandas&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;pd&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-18" name="rest_code_958430c0047b4710a757bfc84ecf7230-18" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-18"&gt;&lt;/a&gt;&lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DataFrame&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_arrow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;groupby&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"company"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="s2"&gt;"fare"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-19" name="rest_code_958430c0047b4710a757bfc84ecf7230-19" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-19"&gt;&lt;/a&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-20" name="rest_code_958430c0047b4710a757bfc84ecf7230-20" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-20"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# ...or with plain pyarrow&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-21" name="rest_code_958430c0047b4710a757bfc84ecf7230-21" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-21"&gt;&lt;/a&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;pyarrow&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;pa&lt;/span&gt;
&lt;a id="rest_code_958430c0047b4710a757bfc84ecf7230-22" name="rest_code_958430c0047b4710a757bfc84ecf7230-22" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_958430c0047b4710a757bfc84ecf7230-22"&gt;&lt;/a&gt;&lt;span class="n"&gt;pa&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"company"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;([(&lt;/span&gt;&lt;span class="s2"&gt;"fare"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"mean"&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Pick whichever syntax you like best — the data stays compressed on disk either way.&lt;/p&gt;
&lt;p&gt;And it works the other way around too — hand &lt;code class="docutils literal"&gt;CTable.from_arrow()&lt;/code&gt; a Polars dataframe, a pyarrow table or a Parquet reader, and you get a compressed table back:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code python"&gt;&lt;a id="rest_code_95354e5df479411fa509ae59c79abbfb-1" name="rest_code_95354e5df479411fa509ae59c79abbfb-1" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_95354e5df479411fa509ae59c79abbfb-1"&gt;&lt;/a&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;polars&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;pl&lt;/span&gt;
&lt;a id="rest_code_95354e5df479411fa509ae59c79abbfb-2" name="rest_code_95354e5df479411fa509ae59c79abbfb-2" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_95354e5df479411fa509ae59c79abbfb-2"&gt;&lt;/a&gt;
&lt;a id="rest_code_95354e5df479411fa509ae59c79abbfb-3" name="rest_code_95354e5df479411fa509ae59c79abbfb-3" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_95354e5df479411fa509ae59c79abbfb-3"&gt;&lt;/a&gt;&lt;span class="n"&gt;pdf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pl&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DataFrame&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s2"&gt;"ticker"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"AAPL"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"GOOG"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="s2"&gt;"close"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mf"&gt;184.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;140.8&lt;/span&gt;&lt;span class="p"&gt;]})&lt;/span&gt;
&lt;a id="rest_code_95354e5df479411fa509ae59c79abbfb-4" name="rest_code_95354e5df479411fa509ae59c79abbfb-4" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_95354e5df479411fa509ae59c79abbfb-4"&gt;&lt;/a&gt;&lt;span class="n"&gt;t2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;blosc2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CTable&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_arrow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pdf&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Blosc2 isn't inventing anything here — it just implements a protocol the whole ecosystem already agreed on.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="strings-that-everyone-understands"&gt;
&lt;h2&gt;Strings that everyone understands&lt;/h2&gt;
&lt;p&gt;Text columns used to force an awkward choice in Blosc2: fixed-width strings (fast, but every value pays for the longest one) or variable-length strings (flexible, but slow to query). The new &lt;code class="docutils literal"&gt;blosc2.utf8()&lt;/code&gt; column type ends that dilemma: it stores text in the exact same layout Arrow uses, so every row costs only what it needs — up to 13x less memory than fixed-width on free-form text — and data flows to and from Arrow tools with no translation.&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code python"&gt;&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-1" name="rest_code_d11c16b0fc944d9581de498e3db592ca-1" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-1"&gt;&lt;/a&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;dataclasses&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-2" name="rest_code_d11c16b0fc944d9581de498e3db592ca-2" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-2"&gt;&lt;/a&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;blosc2&lt;/span&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-3" name="rest_code_d11c16b0fc944d9581de498e3db592ca-3" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-3"&gt;&lt;/a&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-4" name="rest_code_d11c16b0fc944d9581de498e3db592ca-4" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-4"&gt;&lt;/a&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-5" name="rest_code_d11c16b0fc944d9581de498e3db592ca-5" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-5"&gt;&lt;/a&gt;&lt;span class="k"&gt;class&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Place&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-6" name="rest_code_d11c16b0fc944d9581de498e3db592ca-6" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-6"&gt;&lt;/a&gt;    &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;blosc2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;blosc2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utf8&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-7" name="rest_code_d11c16b0fc944d9581de498e3db592ca-7" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-7"&gt;&lt;/a&gt;    &lt;span class="n"&gt;visitors&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;blosc2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;blosc2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;int64&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-8" name="rest_code_d11c16b0fc944d9581de498e3db592ca-8" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-8"&gt;&lt;/a&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-9" name="rest_code_d11c16b0fc944d9581de498e3db592ca-9" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-9"&gt;&lt;/a&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;blosc2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CTable&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Place&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;new_data&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-10" name="rest_code_d11c16b0fc944d9581de498e3db592ca-10" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-10"&gt;&lt;/a&gt;    &lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"café"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"O'Hare"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"日本語のテキスト"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"zürich"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-11" name="rest_code_d11c16b0fc944d9581de498e3db592ca-11" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-11"&gt;&lt;/a&gt;    &lt;span class="s2"&gt;"visitors"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;85_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;a id="rest_code_d11c16b0fc944d9581de498e3db592ca-12" name="rest_code_d11c16b0fc944d9581de498e3db592ca-12" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_d11c16b0fc944d9581de498e3db592ca-12"&gt;&lt;/a&gt;&lt;span class="p"&gt;})&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;And yes, it is fast: reading a full 10-million-row column requires just a fraction of a second; filtering is fast too, and &lt;code class="docutils literal"&gt;utf8()&lt;/code&gt; even beats fixed-width strings as a &lt;code class="docutils literal"&gt;group_by()&lt;/code&gt; key. Filters, sorting and grouping all work on it out of the box. However, sometimes you may still want to use &lt;code class="docutils literal"&gt;string()&lt;/code&gt; / &lt;code class="docutils literal"&gt;vlstring()&lt;/code&gt; for specific use cases; see the &lt;a class="reference external" href="https://blosc.org/python-blosc2/reference/ctable.html#choosing-a-string-column-type"&gt;"Choosing a string column type" guide&lt;/a&gt; if you want the details.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="feeling-at-home-for-pandas-users"&gt;
&lt;h2&gt;Feeling at home for pandas users&lt;/h2&gt;
&lt;p&gt;&lt;code class="docutils literal"&gt;CTable&lt;/code&gt; also picked up more of pandas' vocabulary this cycle, so familiar idioms now just work — chaining with &lt;code class="docutils literal"&gt;assign()&lt;/code&gt; and &lt;code class="docutils literal"&gt;col()&lt;/code&gt;, proper missing-data handling with &lt;code class="docutils literal"&gt;fillna()&lt;/code&gt; and &lt;code class="docutils literal"&gt;dropna()&lt;/code&gt;, and custom aggregation functions in &lt;code class="docutils literal"&gt;group_by()&lt;/code&gt;:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code python"&gt;&lt;a id="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-1" name="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-1" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_fd1302b580a94d28b9d9ff7d28decd8b-1"&gt;&lt;/a&gt;&lt;span class="kn"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;blosc2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;
&lt;a id="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-2" name="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-2" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_fd1302b580a94d28b9d9ff7d28decd8b-2"&gt;&lt;/a&gt;
&lt;a id="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-3" name="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-3" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_fd1302b580a94d28b9d9ff7d28decd8b-3"&gt;&lt;/a&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;assign&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;profit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"revenue"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"cost"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;a id="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-4" name="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-4" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_fd1302b580a94d28b9d9ff7d28decd8b-4"&gt;&lt;/a&gt;  &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"profit"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;a id="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-5" name="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-5" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_fd1302b580a94d28b9d9ff7d28decd8b-5"&gt;&lt;/a&gt;  &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"profit"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ascending&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;a id="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-6" name="rest_code_fd1302b580a94d28b9d9ff7d28decd8b-6" href="https://blosc.org/posts/not-an-island-tabular-ecosystem/#rest_code_fd1302b580a94d28b9d9ff7d28decd8b-6"&gt;&lt;/a&gt;  &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;head&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;And if you'd rather stay in pandas entirely, you can still bring Blosc2's compute engine with you: &lt;code class="docutils literal"&gt;df.apply(func, engine=blosc2.jit)&lt;/code&gt; now works correctly inside pandas 3, and &lt;code class="docutils literal"&gt;Series.map()&lt;/code&gt; supports it too. See the &lt;a class="reference external" href="https://blosc.org/python-blosc2/guides/pandas_engine.html"&gt;"Using Blosc2 as a pandas engine" guide&lt;/a&gt;.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="cooperation-not-completeness"&gt;
&lt;h2&gt;Cooperation, not completeness&lt;/h2&gt;
&lt;p&gt;Although &lt;code class="docutils literal"&gt;CTable&lt;/code&gt; object comes with powerful machinery, it doesn't need to replace DuckDB, Polars or pandas to be useful; it just needs to hand them data in the shape they already expect — while doing the compression and fast querying underneath. If interoperating with the tools you already love sounds like the right job for a compression library, this is the release that makes the case.&lt;/p&gt;
&lt;p&gt;Install it with:&lt;/p&gt;
&lt;pre class="literal-block"&gt;pip install blosc2 --upgrade&lt;/pre&gt;
&lt;p&gt;Full release notes: &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/releases"&gt;https://github.com/Blosc/python-blosc2/releases&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Enjoy data!&lt;/p&gt;
&lt;/section&gt;</description><category>ctable arrow pandas interop compression</category><guid>https://blosc.org/posts/not-an-island-tabular-ecosystem/</guid><pubDate>Fri, 17 Jul 2026 12:00:01 GMT</pubDate></item><item><title>CTable and .b2z: Querying Tabular Data, the Blosc Way</title><link>https://blosc.org/posts/ctable-b2z-queries/</link><dc:creator>Francesc Alted</dc:creator><description>&lt;p&gt;Here is a question we have been chasing, in one form or another, for more than fifteen years: &lt;em&gt;how much work can you avoid doing if your data is stored the right way?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;In this post we put that question to a concrete test: one selective query against 24.3 million Chicago taxi trips, stored on disk in two formats — Parquet and the new persistent &lt;code class="docutils literal"&gt;.b2z&lt;/code&gt; format for &lt;a class="reference external" href="https://blosc.org/posts/new-treestore-blosc2/"&gt;TreeStore&lt;/a&gt; and other Blosc2 objects — and answered by five different tools: DuckDB, PyArrow, pandas, polars, and Blosc2's own &lt;a class="reference external" href="https://blosc.org/python-blosc2/reference/ctable.html"&gt;CTable&lt;/a&gt;. But let me first tell you how we got here, because CTable did not appear out of thin air: it is the fourth floor of a building whose foundations were laid in 2009.&lt;/p&gt;
&lt;section id="from-a-turbo-charged-compressor"&gt;
&lt;h2&gt;From a turbo-charged compressor...&lt;/h2&gt;
&lt;p&gt;Blosc (acronym for BLOcked, Shuffled and Compressed) was born with a single, then-heretical idea: that compression could make data access &lt;em&gt;faster&lt;/em&gt;, not slower. CPUs were (and are) starving — they can crunch numbers far faster than memory can feed them — so if you split data into blocks that fit in CPU caches, shuffle the bytes so that similar ones sit together, and decompress with all your cores, the time spent decompressing can be smaller than the time saved moving fewer bytes. "Compress faster than &lt;code class="docutils literal"&gt;memcpy&lt;/code&gt;" was the provocative benchmark slogan of the time.&lt;/p&gt;
&lt;p&gt;That first Blosc was humble: a blocked, multithreaded meta-compressor for binary buffers. No containers, no files, no types. Just speed.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="to-containers-arrays-and-a-compute-engine"&gt;
&lt;h2&gt;...to containers, arrays, and a compute engine&lt;/h2&gt;
&lt;p&gt;The next decade taught us that a fast compressor alone is not enough; data needs a &lt;em&gt;home&lt;/em&gt;. C-Blosc2 (2.0 released in 2021) gave it one: 64-bit super-chunks, persistent frames, a richer filter pipeline, modern codecs, and a plugin system. On the Python side, this matured into &lt;a class="reference external" href="https://www.blosc.org/python-blosc2/"&gt;Python-Blosc2&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;2023 came with both &lt;a class="reference external" href="https://blosc.org/posts/blosc2-ndim-intro/"&gt;NDim&lt;/a&gt; and &lt;a class="reference external" href="https://blosc.org/python-blosc2/reference/ndarray.html"&gt;NDArray&lt;/a&gt;: a compressed, n-dimensional array, with a two-level partitioning scheme — chunks, sized to fit comfortably in higher-level CPU caches, divided into blocks — where the &lt;em&gt;block&lt;/em&gt; is the unit of decompression, sized to fit comfortably in lower-level CPU caches. Slicing an NDArray decompresses only the blocks that the slice touches, and several blocks can be decompressed in parallel too (as long as they belong to the same chunk). Keep that sentence in mind; it is important for what comes below.&lt;/p&gt;
&lt;p&gt;On top of that, Python-Blosc2 3.0 (early 2025) added a compute engine: &lt;a class="reference external" href="https://blosc.org/python-blosc2/reference/lazyarray.html#lazyexpr"&gt;lazy expressions&lt;/a&gt; like &lt;code class="docutils literal"&gt;a + b * 2&lt;/code&gt; that evaluate block by block, straight over compressed (possibly larger-than-RAM) operands, and return NumPy arrays. The engine never materializes whole arrays; it streams cache-sized blocks through the CPU. At this point we had fast compressed storage &lt;em&gt;and&lt;/em&gt; fast compute over it — what we were missing was a way to talk about &lt;em&gt;tables&lt;/em&gt;.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="ctable-a-columnar-table-on-blosc2-foundations"&gt;
&lt;h2&gt;CTable: a columnar table on Blosc2 foundations&lt;/h2&gt;
&lt;p&gt;&lt;a class="reference external" href="https://www.blosc.org/posts/ctable-blosc2-columnar-table/"&gt;CTable&lt;/a&gt; (introduced in 2026) is exactly that: a columnar table where each column is an NDArray (or a &lt;a class="reference external" href="https://blosc.org/python-blosc2/reference/list_array.html#blosc2.ListArray"&gt;ListArray&lt;/a&gt; for variable-length data), with typed schemas, nullable columns, and a &lt;code class="docutils literal"&gt;where()&lt;/code&gt; method (among others) that accepts plain Python expressions and is executed by the compute engine.&lt;/p&gt;
&lt;p&gt;Because columns are NDArrays, every column inherits the block structure — and this is where the design clicks together. CTable can build indexes for each column (like min/max statistics kept at &lt;em&gt;block&lt;/em&gt; granularity). When a query like &lt;code class="docutils literal"&gt;t.payment.tips &amp;gt; 100&lt;/code&gt; arrives, blocks whose maximum tip is below 100 are never read and never decompressed. The index granularity is exactly aligned with the unit of work it avoids.&lt;/p&gt;
&lt;p&gt;A CTable persists inside a &lt;code class="docutils literal"&gt;.b2z&lt;/code&gt; file: the single-file, zip-based flavor of a persistent &lt;a class="reference external" href="https://www.blosc.org/posts/new-treestore-blosc2/"&gt;TreeStore&lt;/a&gt; that holds all columns, indexes and metadata in one compact, openable-anywhere container. Like Parquet, the data stays compressed on disk; unlike Parquet, you can open it and immediately get NumPy-addressable columns, no engine in between.&lt;/p&gt;
&lt;p&gt;Now, where does the fourth floor hold the weight? Time to experiment.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="the-benchmark-setup-one-selective-query-five-tools"&gt;
&lt;h2&gt;The benchmark setup: one selective query, five tools&lt;/h2&gt;
&lt;p&gt;The dataset is the classic &lt;a class="reference external" href="https://data.cityofchicago.org/Transportation/Taxi-Trips/wrvz-psew"&gt;Chicago Taxi trips&lt;/a&gt; table, with a selection of 24.3 million rows, 14 columns (floats, timestamps, dictionary-encoded strings, and even a variable-length GPS path per trip). The query is a typical needle-in-a-haystack filter with projection and sort:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code python"&gt;&lt;a id="rest_code_104f4f6cb62243d984e4ba0ba652cd3e-1" name="rest_code_104f4f6cb62243d984e4ba0ba652cd3e-1" href="https://blosc.org/posts/ctable-b2z-queries/#rest_code_104f4f6cb62243d984e4ba0ba652cd3e-1"&gt;&lt;/a&gt;&lt;span class="n"&gt;columns&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"payment.tips"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"payment.total"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"trip.sec"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"trip.km"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"company"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;a id="rest_code_104f4f6cb62243d984e4ba0ba652cd3e-2" name="rest_code_104f4f6cb62243d984e4ba0ba652cd3e-2" href="https://blosc.org/posts/ctable-b2z-queries/#rest_code_104f4f6cb62243d984e4ba0ba652cd3e-2"&gt;&lt;/a&gt;&lt;span class="n"&gt;condition&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;payment&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tips&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trip&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;km&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trip&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;begin&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lon&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;a id="rest_code_104f4f6cb62243d984e4ba0ba652cd3e-3" name="rest_code_104f4f6cb62243d984e4ba0ba652cd3e-3" href="https://blosc.org/posts/ctable-b2z-queries/#rest_code_104f4f6cb62243d984e4ba0ba652cd3e-3"&gt;&lt;/a&gt;&lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;condition&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"trip.sec"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;or, in SQL:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code sql"&gt;&lt;a id="rest_code_90ad9694d3094c1385c654fa141aff39-1" name="rest_code_90ad9694d3094c1385c654fa141aff39-1" href="https://blosc.org/posts/ctable-b2z-queries/#rest_code_90ad9694d3094c1385c654fa141aff39-1"&gt;&lt;/a&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;payment&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tips&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;payment&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;trip&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sec&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;trip&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;km&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;company&lt;/span&gt;
&lt;a id="rest_code_90ad9694d3094c1385c654fa141aff39-2" name="rest_code_90ad9694d3094c1385c654fa141aff39-2" href="https://blosc.org/posts/ctable-b2z-queries/#rest_code_90ad9694d3094c1385c654fa141aff39-2"&gt;&lt;/a&gt;&lt;span class="k"&gt;WHERE&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;payment&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tips&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AND&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;trip&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;km&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AND&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;trip&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;begin&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lon&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;
&lt;a id="rest_code_90ad9694d3094c1385c654fa141aff39-3" name="rest_code_90ad9694d3094c1385c654fa141aff39-3" href="https://blosc.org/posts/ctable-b2z-queries/#rest_code_90ad9694d3094c1385c654fa141aff39-3"&gt;&lt;/a&gt;&lt;span class="k"&gt;ORDER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;BY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;trip&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sec&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Only &lt;strong&gt;67 of 24.3 million rows&lt;/strong&gt; match — a highly selective query, which is precisely the regime where indexing can shine.&lt;/p&gt;
&lt;p&gt;The contenders: &lt;strong&gt;DuckDB&lt;/strong&gt;, &lt;strong&gt;PyArrow&lt;/strong&gt;, &lt;strong&gt;pandas&lt;/strong&gt; and &lt;strong&gt;polars&lt;/strong&gt; querying the Parquet file, and &lt;strong&gt;Blosc2's CTable&lt;/strong&gt; querying the &lt;code class="docutils literal"&gt;.b2z&lt;/code&gt;. Every tool reads from disk on demand; nothing is preloaded. Each engine runs in a fresh subprocess, and it reports the &lt;em&gt;query time&lt;/em&gt; each script measures internally, excluding interpreter and import overhead. Cold-cache runs happen right after flushing the OS file cache (&lt;code class="docutils literal"&gt;sudo purge&lt;/code&gt;); warm runs are best-of-7. The machine is a Mac mini (Apple M4 Pro) with 24 GB RAM. The full, reproducible notebook is &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/blob/main/bench/chicago-taxi/compare-query-methods.ipynb"&gt;in the python-blosc2 repository&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;First, the storage footprint:&lt;/p&gt;
&lt;img alt="File sizes: parquet 654.0 MB vs b2z 670.3 MB" class="align-center" src="https://blosc.org/images/ctable-b2z-queries/compare-size.png" style="width: 60%;"&gt;
&lt;p&gt;The &lt;code class="docutils literal"&gt;.b2z&lt;/code&gt; lands at 670 MB versus Parquet's 654 MB — a 2% premium. Those extra bytes are mostly the block-level indexes. It is important to note that these indexes are computed automatically when importing a Parquet file into a CTable .b2z file.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="cold-cache-reading-less-wins"&gt;
&lt;h2&gt;Cold cache: reading less wins&lt;/h2&gt;
&lt;p&gt;The cold run is the scenario we care most about: you have a large file on disk, it is &lt;em&gt;not&lt;/em&gt; in the OS cache, and you want one answer, fast.&lt;/p&gt;
&lt;img alt="Cold query times: blosc2 0.057s, duckdb 0.106s, arrow 0.142s, polars 0.302s, pandas 0.539s" class="align-center" src="https://blosc.org/images/ctable-b2z-queries/compare-query-time-cold.png" style="width: 75%;"&gt;
&lt;p&gt;CTable answers in &lt;strong&gt;0.057 s&lt;/strong&gt; — about &lt;strong&gt;1.9x faster than DuckDB&lt;/strong&gt; (0.106 s), 2.5x faster than PyArrow, 5.3x faster than polars and 9.4x faster than pandas.  It is important to note that all methods try to use streaming and filter pushdown where possible, although pandas is not designed for that, and it shows; interestingly polars is more advanced in that regard.&lt;/p&gt;
&lt;p&gt;The reason why CTable can go faster is fine-grained indexing (min/max on blocks). On a cold cache, the dominant cost is bytes coming off the disk. The indexes let CTable prune roughly &lt;strong&gt;90% of the blocks&lt;/strong&gt; for this query: those blocks are neither read nor decompressed. Pruning pays twice — less I/O &lt;em&gt;and&lt;/em&gt; less CPU — and on a first-touch query the I/O half is the whole ballgame.  Parquet also has min/max statistics, but at a coarser row-group granularity, which for this query can prune nothing (see below).&lt;/p&gt;
&lt;/section&gt;
&lt;section id="warm-cache-a-dead-heat-with-a-real-database"&gt;
&lt;h2&gt;Warm cache: a dead heat with a real database&lt;/h2&gt;
&lt;p&gt;Once the file is fully cached in RAM, I/O is nearly free and raw engine throughput takes over. This is DuckDB's home turf — a vectorized, multithreaded analytical SQL engine with filter pushdown and late materialization.&lt;/p&gt;
&lt;img alt="Warm query times: blosc2 0.032s and duckdb 0.034s in a dead heat, ahead of arrow, polars, pandas" class="align-center" src="https://blosc.org/images/ctable-b2z-queries/compare-query-time-warm.png" style="width: 75%;"&gt;
&lt;p&gt;CTable finishes in 0.032 s, DuckDB in 0.034 s — a &lt;strong&gt;dead heat&lt;/strong&gt; (the two trade places within run-to-run noise), with both about 2.8x ahead of PyArrow, ~7x ahead of polars, and ~16x ahead of pandas. We find this result remarkable because Blosc2 brings no SQL engine: a storage container + compute + indexing engine holding the tie with a purpose-built database tells us the layout is doing the heavy lifting.  Again, this layout is determined automatically, so users get the benefit without tuning.&lt;/p&gt;
&lt;p&gt;Memory tells a similar story:&lt;/p&gt;
&lt;img alt="Peak memory: duckdb ~61 MB, blosc2 ~83 MB, arrow ~232 MB, polars ~415 MB, pandas ~1.6 GB" class="align-center" src="https://blosc.org/images/ctable-b2z-queries/compare-query-mem-warm.png" style="width: 75%;"&gt;
&lt;p&gt;DuckDB (~61 MB) and CTable (~83 MB) are the two leanest by a wide margin — more than an order of magnitude below pandas (~1.6 GB), which materializes full columns before filtering. polars can make use of better streaming, while PyArrow is still better, but both are still much heavier than the two leaders.  This is important for large datasets that may not fit comfortably in RAM, because that means that larger-than-RAM queries are more likely to succeed without hitting swap when using CTable or DuckDB.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="why-pruning-wins-granularity"&gt;
&lt;h2&gt;Why pruning wins: granularity&lt;/h2&gt;
&lt;p&gt;Parquet also carries min/max statistics — at &lt;strong&gt;row-group&lt;/strong&gt; granularity, here ~970,000 rows per group. CTable keeps them at &lt;strong&gt;block&lt;/strong&gt; granularity, ~27,000 rows per block: roughly 36x finer. A Blosc2 block is &lt;em&gt;the unit of decompression&lt;/em&gt; — the same cache-sized block the compute engine streams. An index at a coarser granularity than the I/O unit can only skip work in big, lucky lumps. Choosing an appropriate block size is important, and this is automatically handled by Blosc2 when importing from Parquet.&lt;/p&gt;
&lt;p&gt;Note that the advantage of indexes rides on &lt;strong&gt;selectivity&lt;/strong&gt;, not on any general superiority. &lt;code class="docutils literal"&gt;tips &amp;gt; 100&lt;/code&gt; is rare enough that most 27K-row blocks contain no match.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="conclusions"&gt;
&lt;h2&gt;Conclusions&lt;/h2&gt;
&lt;p&gt;Several interesting takeaways emerge from this experiment:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;For selective cold queries on large tabular files, CTable/.b2z is genuinely fast&lt;/strong&gt; — the fastest of the five tools here, on a query and dataset it was not specially tuned for. If your workload looks like "open a big file, fetch a small subset, move on", the block-level indexing earns its 2% of disk many times over.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Warm, it ties.&lt;/strong&gt; DuckDB remains an excellent engine, and on cached data it matches CTable while speaking full SQL with joins and aggregations that CTable cannot cover (at least not yet). If your problems are relational, use a relational engine.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;The result is arrays, not a result set.&lt;/strong&gt; &lt;code class="docutils literal"&gt;&lt;span class="pre"&gt;t.where(...)&lt;/span&gt;&lt;/code&gt; hands back NumPy-addressable columns with their original dtypes — no &lt;code class="docutils literal"&gt;.to_numpy()&lt;/code&gt; hop, no DataFrame conversion tax. For NumPy-centric pipelines, that removes a whole impedance layer. And since columns are NDArrays, a CTable column can even be n-dimensional, or hold variable-length data (this dataset stores a GPS trace per row).&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Parquet is not going anywhere.&lt;/strong&gt; It remains the lingua franca of the data ecosystem, readable by everything. &lt;code class="docutils literal"&gt;.b2z&lt;/code&gt; is young and its natural habitat is the Python/NumPy world. What this experiment shows is that the trade is real and the price is modest: a couple percent of disk for first-touch queries that run in a fraction of the time.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Sixteen years after asking whether compression could be faster than &lt;code class="docutils literal"&gt;memcpy&lt;/code&gt;, the question has scaled up but taken on a slightly different shape: the fastest byte is the one you never touch. Blocks sized for caches made decompression cheap; the compute engine made math over blocks cheap; and CTable's block-level indexes now make &lt;em&gt;not touching&lt;/em&gt; most of a table cheap, too. The fourth floor stands on the three below it.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="reproduce-it-yourself"&gt;
&lt;h2&gt;Reproduce it yourself&lt;/h2&gt;
&lt;p&gt;Everything in this post lives in &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/tree/main/bench/chicago-taxi"&gt;bench/chicago-taxi&lt;/a&gt;. The &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/blob/main/bench/chicago-taxi/compare-query-methods.ipynb"&gt;notebook&lt;/a&gt;, the driver, the five per-engine query scripts, and a README with the details. The notebook downloads the dataset on first run and builds the &lt;code class="docutils literal"&gt;.b2z&lt;/code&gt; from it, so the whole thing is two commands away:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code console"&gt;&lt;a id="rest_code_7a5904530bfa428ab59ae1f2d2afa03c-1" name="rest_code_7a5904530bfa428ab59ae1f2d2afa03c-1" href="https://blosc.org/posts/ctable-b2z-queries/#rest_code_7a5904530bfa428ab59ae1f2d2afa03c-1"&gt;&lt;/a&gt;&lt;span class="go"&gt;pip install "blosc2&amp;gt;=4.4.3" pyarrow duckdb polars pandas matplotlib jupyter&lt;/span&gt;
&lt;a id="rest_code_7a5904530bfa428ab59ae1f2d2afa03c-2" name="rest_code_7a5904530bfa428ab59ae1f2d2afa03c-2" href="https://blosc.org/posts/ctable-b2z-queries/#rest_code_7a5904530bfa428ab59ae1f2d2afa03c-2"&gt;&lt;/a&gt;&lt;span class="go"&gt;jupyter-lab compare-query-methods.ipynb   # then: Run All&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;/section&gt;
&lt;section id="more-info"&gt;
&lt;h2&gt;More info&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;p&gt;&lt;a class="reference external" href="https://www.blosc.org/posts/ctable-blosc2-columnar-table/"&gt;Introducing CTable&lt;/a&gt; — the design and feature tour&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a class="reference external" href="https://blosc.org/python-blosc2/getting_started/tutorials/13.ctable-basics.html"&gt;Getting started with CTable&lt;/a&gt; and &lt;a class="reference external" href="https://blosc.org/python-blosc2/getting_started/tutorials/15.indexing-ctables.html"&gt;Indexing CTables&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/tree/main/bench/chicago-taxi"&gt;The benchmark directory&lt;/a&gt; — notebook, driver, per-engine scripts and README&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a class="reference external" href="https://blosc.org/python-blosc2/reference/ctable.html"&gt;CTable API reference&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/section&gt;
&lt;section id="acknowledgments"&gt;
&lt;h2&gt;Acknowledgments&lt;/h2&gt;
&lt;p&gt;The benchmark uses the &lt;a class="reference external" href="https://data.cityofchicago.org/Transportation/Taxi-Trips/wrvz-psew"&gt;Taxi Trips dataset&lt;/a&gt; published by the City of Chicago under its open data program. Thanks also to the NumFOCUS foundation and our sponsors for making sustained work on Blosc possible.&lt;/p&gt;
&lt;p&gt;Enjoy data!&lt;/p&gt;
&lt;/section&gt;</description><category>ctable b2z parquet queries tabular indexing compression</category><guid>https://blosc.org/posts/ctable-b2z-queries/</guid><pubDate>Thu, 11 Jun 2026 10:00:00 GMT</pubDate></item><item><title>Introducing CTable: a Blosc2-based columnar table</title><link>https://blosc.org/posts/ctable-blosc2-columnar-table/</link><dc:creator>Jorge Albiol, Francesc Alted</dc:creator><description>&lt;p&gt;Working with large structured datasets in Python often means choosing between speed and simplicity. The new &lt;a class="reference external" href="https://blosc.org/python-blosc2/reference/ctable.html#"&gt;CTable&lt;/a&gt; object has born out of the need for a columnar store that compresses data on the fly, stays close to NumPy, and does not require an external database engine. It is the logical extension of current compressed data storage and computation in Python-Blosc2, brought to tabular datasets.&lt;/p&gt;
&lt;p&gt;As compression is paramount in Blosc2 ecosystem, we have chosen a columnar approach because, by placing together data that is similar (values in the same column), it allows for best compression ratios.  Column storage also allows for better data management for some cases, like adding, deleting, accessing or replacing entire columns; admittedly, it it also has its own drawbacks, like more costly access along the row axis. Nevertheless, columnar storage is quite common in modern libraries.&lt;/p&gt;
&lt;p&gt;Another important piece for CTable is to leverage the extremely efficient compute engine that can operate on compressed data without dropping too much performance (and in some cases, even improving it).  That puts the basis for allowing great analytics machinery on top of the CTable object, without the need to decompress entire columns (just small excerpts of them, fitting in CPU caches, is enough).&lt;/p&gt;
&lt;img alt="CTable: compressed columns and selective queries" class="align-center" src="https://blosc.org/images/new-ctable-intro/ctable_compressed_columns_selective_queries.png" style="width: 50%;"&gt;
&lt;p&gt;Last but not least, the CTable object inherits the independence of media storage of underlying structures (NDArray, ObjectArray, ListArray...) so that data can be stored and used straight from memory, disk or the network (coming soon).  That means that you can open a data file containing a big CTable and immediately start doing analytics with it without the need to load/parse everything in-memory.  Of course, for maximum speed, you may also load everything in-memory too; but as the format is the same, loading/saving is just a matter of copying data from one media to another, without the need for parsing or conversion.&lt;/p&gt;
&lt;p&gt;Keep reading to learn more about CTable, its features and how to use it in your projects.&lt;/p&gt;
&lt;section id="how-it-works"&gt;
&lt;h2&gt;How it works&lt;/h2&gt;
&lt;p&gt;CTable stores each column as an independent &lt;code class="docutils literal"&gt;blosc2.NDArray&lt;/code&gt;, compressed in chunks. Column types are defined through a &lt;a class="reference external" href="https://blosc.org/python-blosc2/reference/ctable.html#schema-specs"&gt;schema&lt;/a&gt; — a plain Python dataclass where each field is annotated with a Blosc2 type spec such as &lt;code class="docutils literal"&gt;b2.int64()&lt;/code&gt;, &lt;code class="docutils literal"&gt;b2.float32(ge=0)&lt;/code&gt;, or &lt;code class="docutils literal"&gt;b2.string()&lt;/code&gt;. Specs can carry constraints (e.g. &lt;code class="docutils literal"&gt;ge=0&lt;/code&gt; for non-negative values) and are compiled into a schema that validates every row on insert, either one at a time via Pydantic or in bulk via vectorized NumPy checks.&lt;/p&gt;
&lt;p&gt;Rows are tracked with a boolean tombstone mask: deleting a row simply flips its entry in the mask to &lt;code class="docutils literal"&gt;False&lt;/code&gt;, with no data movement at all. The actual space is reclaimed lazily when you call &lt;code class="docutils literal"&gt;compact()&lt;/code&gt;. Appending is also efficient because the underlying arrays are pre-allocated up front — they only grow when the pre-allocated capacity is exhausted, so there is no resize on every single insert.&lt;/p&gt;
&lt;p&gt;Because the data lives in Blosc2 chunks, many queries can skip full chunks entirely. When a chunk's stored metadata (min/max) rules out any match, it is never decompressed. This is where a lot of the query speed comes from, and it is also why explicit indexes (described in Features below) can push performance even further.&lt;/p&gt;
&lt;p&gt;Since &lt;code class="docutils literal"&gt;blosc2.NDArray&lt;/code&gt; stores fixed-width binary data, &lt;code class="docutils literal"&gt;null&lt;/code&gt; has no natural representation for integers, floats, or booleans. CTable solves this by letting you declare a column as nullable, and a sentinel value as the null marker is chosen automatically, although you can always use your own. For example, if you are storing ages and sometimes the value is unknown, you can set &lt;code class="docutils literal"&gt;&lt;span class="pre"&gt;-1&lt;/span&gt;&lt;/code&gt; as the null value since ages are never negative. Aggregates such as &lt;code class="docutils literal"&gt;.mean()&lt;/code&gt; or &lt;code class="docutils literal"&gt;.std()&lt;/code&gt; skip those rows automatically, and helper methods like &lt;code class="docutils literal"&gt;.is_null()&lt;/code&gt; and &lt;code class="docutils literal"&gt;.null_count()&lt;/code&gt; make it easy to work with them.&lt;/p&gt;
&lt;p&gt;Not all data fits neatly into fixed-width columns though. Think of a column storing the tags of an article, the purchase history of a user, or the list of measurements taken at a sensor in a given day — each row may have a different number of items. For these cases CTable supports list columns, declared as &lt;code class="docutils literal"&gt;blosc2.list(item_spec)&lt;/code&gt; (e.g. &lt;code class="docutils literal"&gt;&lt;span class="pre"&gt;blosc2.list(blosc2.float32())&lt;/span&gt;&lt;/code&gt;), structured objects via &lt;code class="docutils literal"&gt;blosc2.struct(item_spec)&lt;/code&gt; or completely general objects via &lt;code class="docutils literal"&gt;b2.object()&lt;/code&gt;. These columns are backed by a different storage class internally, one that keeps a compressed stream of items alongside an offsets array to know where each row starts and ends. From the user's perspective they behave like any other column, but each cell holds a Python list instead of a scalar, and individual lists can also be &lt;code class="docutils literal"&gt;None&lt;/code&gt;.  Internally, the underlying C-Blosc2 has been improved (and &lt;a class="reference external" href="https://github.com/Blosc/c-blosc2/releases"&gt;released as 3.0.0&lt;/a&gt;) to allow variable-length data in super-chunks in a very efficient (and backward-compatible) way.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="main-features"&gt;
&lt;h2&gt;Main features&lt;/h2&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Creation&lt;/strong&gt;: A CTable can be created in several ways. The most direct is declaring a typed schema as a dataclass and passing it to the constructor. You can also build a CTable from existing data — &lt;code class="docutils literal"&gt;from_arrow()&lt;/code&gt; and &lt;code class="docutils literal"&gt;from_csv()&lt;/code&gt; import Arrow tables and CSV files respectively, inferring or  mapping types automatically. Finally, &lt;code class="docutils literal"&gt;copy()&lt;/code&gt; produces a new independent CTable from an existing one, already compacted.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Modification&lt;/strong&gt;: Appending a single row uses &lt;code class="docutils literal"&gt;append()&lt;/code&gt;, while bulk insertion uses &lt;code class="docutils literal"&gt;extend()&lt;/code&gt;. Deleting rows sets their mask entry to &lt;code class="docutils literal"&gt;False&lt;/code&gt; and is essentially free. Columns can be added with a default value or dropped and renamed at any time. Beyond stored columns, CTable also supports two kinds
of virtual columns: &lt;em&gt;computed columns&lt;/em&gt; are evaluated on-the-fly from an expression over other columns and never touch storage; &lt;em&gt;materialized columns&lt;/em&gt; look like stored columns but are auto-filled automatically during every &lt;code class="docutils literal"&gt;extend()&lt;/code&gt;.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Querying&lt;/strong&gt;: &lt;code class="docutils literal"&gt;where(expr)&lt;/code&gt; filters rows and returns a &lt;em&gt;view&lt;/em&gt; — a new CTable object that shares the same column arrays as the parent but carries its own mask. No data is copied; only the mask is computed. Views block structural changes (adding/dropping columns, deleting rows) but do allow writing values to existing cells. &lt;code class="docutils literal"&gt;select(cols)&lt;/code&gt; gives a column-projection view in the same  spirit. Both can be made into a fully independent mutable table with &lt;code class="docutils literal"&gt;copy()&lt;/code&gt;. Aggregates (&lt;code class="docutils literal"&gt;sum()&lt;/code&gt;, &lt;code class="docutils literal"&gt;mean()&lt;/code&gt;, &lt;code class="docutils literal"&gt;std()&lt;/code&gt;, &lt;code class="docutils literal"&gt;min()&lt;/code&gt;, &lt;code class="docutils literal"&gt;max()&lt;/code&gt;, ...) and &lt;code class="docutils literal"&gt;sort_by()&lt;/code&gt; also work on views and respect the mask.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Indexing&lt;/strong&gt;: For workloads that repeatedly query the same column, CTable supports three index flavors: &lt;code class="docutils literal"&gt;FULL&lt;/code&gt; (sorted positions array, best for range and comparison queries), &lt;code class="docutils literal"&gt;BUCKET&lt;/code&gt; (hash-based, best for equality lookups), and &lt;code class="docutils literal"&gt;PARTIAL&lt;/code&gt; (a lighter-weight sorted structure). Once an index is created, &lt;code class="docutils literal"&gt;where()&lt;/code&gt; uses it automatically when the query can benefit from it. Indexes are persisted alongside the table and survive &lt;code class="docutils literal"&gt;.b2z&lt;/code&gt; round-trips.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Persistence&lt;/strong&gt;: Tables can live fully in memory or be backed by files on disk. &lt;code class="docutils literal"&gt;save()&lt;/code&gt; writes an in-memory table to a directory or &lt;code class="docutils literal"&gt;.b2z&lt;/code&gt; archive. &lt;code class="docutils literal"&gt;CTable.open()&lt;/code&gt; attaches directly to an on-disk table for reading or writing without loading everything into RAM. &lt;code class="docutils literal"&gt;CTable.load()&lt;/code&gt; copies the on-disk table fully into memory for faster subsequent access. Both &lt;code class="docutils literal"&gt;.b2d&lt;/code&gt; directories and &lt;code class="docutils literal"&gt;.b2z&lt;/code&gt; zip archives are supported transparently.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/section&gt;
&lt;section id="mini-benchmarks"&gt;
&lt;h2&gt;Mini benchmarks&lt;/h2&gt;
&lt;p&gt;All numbers below are from a single machine, 1 million rows, using the benchmark scripts in the repository. They are meant to give a feel for the performance characteristics, not as absolute guarantees.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Bulk loading speed&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;How you feed data into a CTable matters a lot. Loading 1M rows from a Python list of dicts takes around 0.66 s. Switching to a NumPy structured array brings that down to 0.03 s — a &lt;strong&gt;22x speedup&lt;/strong&gt;. Loading from an existing CTable is even faster at &lt;strong&gt;28x&lt;/strong&gt;. The takeaway is simple: if you have NumPy data, hand it directly to &lt;code class="docutils literal"&gt;extend()&lt;/code&gt; and it will be ingested at close to raw array speed.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Filtering vs pandas&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Filtering 1M rows with a range query (&lt;code class="docutils literal"&gt;id&lt;/code&gt; between 250k and 750k, so 50% of the table) takes around 13 ms in CTable vs 31 ms in pandas — &lt;strong&gt;2.4x faster&lt;/strong&gt;. On top of that, the CTable occupies 20 MB compressed versus 31 MB for the equivalent pandas DataFrame, a &lt;strong&gt;1.6x reduction in memory&lt;/strong&gt; essentially for free thanks to Blosc2's compression pipeline.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;extend() vs append() — always batch if you can&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;CTable has two ways to insert data: &lt;code class="docutils literal"&gt;append()&lt;/code&gt; adds one row at a time and goes through a full Pydantic validation cycle per row; &lt;code class="docutils literal"&gt;extend()&lt;/code&gt; takes a batch and validates it in one vectorized NumPy pass. At 100k rows the difference is &lt;strong&gt;2000x in favour of ``extend()``&lt;/strong&gt;. Even at 10k rows it is already 700x. The message is simple: if you have more than a handful of rows to insert, always batch them into a single &lt;code class="docutils literal"&gt;extend()&lt;/code&gt; call.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;where() is nearly free regardless of selectivity&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;One property of the lazy mask approach is that &lt;code class="docutils literal"&gt;where()&lt;/code&gt; costs roughly the same whether the result contains 10 rows or 999,990 rows out of 1M. In practice the time stays between 12 ms and 18 ms across all selectivity levels. You are not paying to materialise the matching rows — you are only computing a mask. The data is only read when you actually access it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Combining filters is 4x faster than chaining them&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;It is tempting to filter a CTable step by step — first narrow by one condition, then filter the result by another. But each &lt;code class="docutils literal"&gt;where()&lt;/code&gt; call creates a new view with its own mask computation. A single &lt;code class="docutils literal"&gt;where()&lt;/code&gt; with all conditions joined by &lt;code class="docutils literal"&gt;&amp;amp;&lt;/code&gt; does the same work in one pass and is &lt;strong&gt;4.4x faster&lt;/strong&gt; than five chained calls returning the same final result.&lt;/p&gt;
&lt;img alt="Single combined where() vs three chained where() calls across selectivity levels" class="align-center" src="https://blosc.org/images/new-ctable-intro/where_single_vs_chained.png"&gt;
&lt;p&gt;&lt;strong&gt;Memory footprint depends on your data&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;CTable compresses each column independently with Blosc2, so how much memory you save depends on how much structure your data has. With highly repetitive data — sequential integer IDs, a handful of distinct float values, constant booleans — a 100-million-row table fits in under 4 MB, versus over 1.6 GB for the equivalent pandas DataFrame. With fully random data the gain is more modest (around 1.6×), since high-entropy values leave little for the compressor to exploit. Real-world datasets typically land somewhere in between, but CTable is consistently more memory-efficient than pandas regardless of data entropy.&lt;/p&gt;
&lt;img alt="Memory footprint comparison between pandas and CTable across data entropy levels" class="align-center" src="https://blosc.org/images/new-ctable-intro/memory_footprint.png"&gt;
&lt;p&gt;&lt;strong&gt;Efficient Indexing&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Indexes can speed up queries by orders of magnitude.  The embedded indexing engine stores data in compressed form, allowing for large savings in memory and disk. For example, a FULL index on a 1 &lt;em&gt;billion&lt;/em&gt; rows with random values takes around 5.8 GB on disk, while an equivalent DuckDB index takes around 41 GB. Typically, queries that would take seconds can be reduced to milliseconds.&lt;/p&gt;
&lt;img alt="Indexing performance comparison between CTable and DuckDB" class="align-center" src="https://blosc.org/images/new-ctable-intro/indexing_vs_duckdb_linux_amd_7800x3d_build_storage.png"&gt;
&lt;img alt="Lookup performance comparison between CTable and DuckDB" class="align-center" src="https://blosc.org/images/new-ctable-intro/indexing_vs_duckdb_linux_amd_7800x3d_lookup.png"&gt;
&lt;img alt="Memory usage comparison between CTable and DuckDB" class="align-center" src="https://blosc.org/images/new-ctable-intro/indexing_vs_duckdb_memory.png"&gt;
&lt;p&gt;&lt;strong&gt;Schema validation has near-zero cost at scale&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Every CTable has a typed schema with optional constraints (ranges, string lengths, etc.). When inserting data with &lt;code class="docutils literal"&gt;extend()&lt;/code&gt;, these constraints are checked via a vectorized NumPy path rather than row by row. At 1M rows with a NumPy structured array the validation overhead is essentially &lt;strong&gt;1.00x —indistinguishable from skipping validation entirely&lt;/strong&gt;. Even with Python list input it only adds 1.31x. You get correctness guarantees without paying for them at scale.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="ai-role"&gt;
&lt;h2&gt;AI role&lt;/h2&gt;
&lt;p&gt;During the development of CTable, we have been using AI tools to help us in the design and implementation of the API, as well as in the documentation.  Tools like Perplexity, and agents like Pi, Codex or Claude have been instrumental in helping us throughout the process, which allowed us to be much more ambitious in the features we wanted to implement. &lt;em&gt;Note from Francesc&lt;/em&gt;: I specially liked the combination of the Pi agent and GPT 5.5 model (essentially GPT &amp;gt;= 5.3); that worked really well!&lt;/p&gt;
&lt;p&gt;Of course, we already borrowed some ideas from other libraries, like Apache Arrow, Pandas, Polars, DuckDB or PyTables, but we also wanted to bring some unique features to CTable, like the ability to operate on compressed data without decompressing it, or the rich schema specs for expressing complex data types; AI has been instrumental in allowing us doing this.&lt;/p&gt;
&lt;p&gt;Being a powerful tool, AI always need supervision and guidance to be used effectively, and we have spent lots of time bringing our cumulated decades-long experience for review code, designing tests and benchmarks, and fine-tuning the internal knobs for allowing best performance and user experience.   We must say that we are very happy with the results: combining our experience with the power of AI has allowed us to create a powerful and flexible tabular data container that is well tested and documented.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="more-info"&gt;
&lt;h2&gt;More info&lt;/h2&gt;
&lt;p&gt;We have setup a couple of tutorials and a complete API reference to get you started with CTable:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;p&gt;&lt;a class="reference external" href="https://blosc.org/python-blosc2/getting_started/tutorials/13.ctable-basics.html"&gt;Getting started with CTable&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a class="reference external" href="https://blosc.org/python-blosc2/getting_started/tutorials/15.indexing-ctables.html"&gt;Indexing CTables&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/tree/main/examples/ctable"&gt;More CTable examples&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a class="reference external" href="https://blosc.org/python-blosc2/reference/ctable.html#blosc2.CTable"&gt;CTable API reference&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/section&gt;
&lt;section id="conclusions"&gt;
&lt;h2&gt;Conclusions&lt;/h2&gt;
&lt;p&gt;CTable brings together compression, schema validation, and query acceleration in a self-contained Python package. It is still young, but the architecture is solid and the feature set already covers most common analytical workflows, and we hope it will be useful for many users in the Python ecosystem. We are looking forward to seeing how the community will use and contribute to Python-Blosc2 in general, an CTable in particular, and to continue improving it based on feedback and contributions from users.&lt;/p&gt;
&lt;p&gt;Enjoy data!&lt;/p&gt;
&lt;/section&gt;</description><category>ctable columnar table compression</category><guid>https://blosc.org/posts/ctable-blosc2-columnar-table/</guid><pubDate>Wed, 13 May 2026 09:37:04 GMT</pubDate></item><item><title> Cumulative reductions in Blosc2</title><link>https://blosc.org/posts/cumsum/</link><dc:creator>Luke Shaw</dc:creator><description>&lt;p&gt;As mentioned in previous blog posts (see &lt;a class="reference external" href="https://ironarray.io/blog/array-api"&gt;this blog&lt;/a&gt;) the maintainers of &lt;code class="docutils literal"&gt;&lt;span class="pre"&gt;python-blosc2&lt;/span&gt;&lt;/code&gt; are going all-in on Array API integration. This means adding new functions to bring the library up to the standard. Of course, integrating a given function may be more or less difficult for a given library which aspires to compatibility, depending on legacy code, design principles, and the overarching philosophy of the package. Since &lt;code class="docutils literal"&gt;&lt;span class="pre"&gt;python-blosc2&lt;/span&gt;&lt;/code&gt; uses chunked arrays, handling reductions and mapping between local chunk- and global array-indexing can be tricky. We had some help from Yang Kang Chua at UConn with this functionality - many thanks to him!&lt;/p&gt;
&lt;section id="cumulative-reductions"&gt;
&lt;h2&gt;Cumulative reductions&lt;/h2&gt;
&lt;p&gt;Consider an array &lt;code class="docutils literal"&gt;a&lt;/code&gt; of shape &lt;code class="docutils literal"&gt;(1000, 2000, 3000)&lt;/code&gt; and data type &lt;code class="docutils literal"&gt;float64&lt;/code&gt; (more on numerical precision later). The result of &lt;code class="docutils literal"&gt;sum(a, axis=0)&lt;/code&gt; would be &lt;code class="docutils literal"&gt;(20, 30)&lt;/code&gt; and &lt;code class="docutils literal"&gt;sum(a, axis=1)&lt;/code&gt; would be &lt;code class="docutils literal"&gt;(1000, 3000)&lt;/code&gt;. In general we can say that reductions &lt;em&gt;reduce&lt;/em&gt; the sizes of arrays. On the other hand, cumulative reductions store the intermediate reduction results along the reduction axis, so that the shape of the result is always the same as that of the input array: &lt;code class="docutils literal"&gt;cumulative_sum(a, axis=ax)&lt;/code&gt; is always &lt;code class="docutils literal"&gt;(1000, 2000, 3000)&lt;/code&gt; for any (valid) value of &lt;code class="docutils literal"&gt;ax&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;This has a couple of consequences. One is that memory consumption may be rather important: the array &lt;code class="docutils literal"&gt;a&lt;/code&gt; will occupy &lt;code class="docutils literal"&gt;&lt;span class="pre"&gt;math.prod((1000,&lt;/span&gt; 2000, &lt;span class="pre"&gt;3000))*8/(1024**3)&lt;/span&gt; = 44.7GB&lt;/code&gt;, but its sum along the first axis only &lt;code class="docutils literal"&gt;.0447GB&lt;/code&gt;. Thus we can easily store the final result in memory. Not so for the result of &lt;code class="docutils literal"&gt;cumulative_sum&lt;/code&gt; which also occupies &lt;code class="docutils literal"&gt;44.7GB&lt;/code&gt;!&lt;/p&gt;
&lt;p&gt;The second consequence, for chunked array libraries, is that the order in which one loads chunks and calculates the result matters. Consider the following diagram, where we have a 1D array of three elements. To calculate the final sum, we may load the chunks in any order and do not require access to any previous value except the running total - loading the first, third and finally second chunks, we obtain the correct sum of 4. However, for the cumulative sum, each element of the result depends on the previous element (and from there the sum of all prior elements of the array). Consequently, we must ensure we load the chunks according to their order in memory - if not, we will end up with an incorrect final result. A minimal criterion is that the final element of the cumulative sum should be the same as the sum, which is not the case here!&lt;/p&gt;
&lt;img alt="/images/cumulative_sumprod/ordermatters.png" class="align-center" src="https://blosc.org/images/cumulative_sumprod/ordermatters.png" style="width: 50%;"&gt;
&lt;/section&gt;
&lt;section id="consequences-for-numerical-precision"&gt;
&lt;h2&gt;Consequences for numerical precision&lt;/h2&gt;
&lt;p&gt;When calculating reductions, numerical precision is a common hiccup. For products, one can quickly overflow the data type - the product of &lt;code class="docutils literal"&gt;arange(1, 14)&lt;/code&gt; already overflows the maximum value of &lt;code class="docutils literal"&gt;int32&lt;/code&gt;. For sums, rounding errors incurred due to adding elements of a small size to the running total of a large size can quickly become significant. For this reason, Numpy will try to use pairwise summation to calculate &lt;code class="docutils literal"&gt;sum(a)&lt;/code&gt; - this involves breaking the array into small parts, calculating the sum on each small part (i.e. simply successively adding elements to a running total), and then recursively summing pairs of sums until the final result is reached. Each recursive sum operation thus involves the sum of two numbers of similar size, thus reducing the rounding errors incurred when summing disparate numbers. This algorithm also only has a minimal additional overhead compared to the naive approach and is eminently parallelisable. And it has a natural recursive implementation, something which computer scientists always find appealing even if only for aesthetic reasons!&lt;/p&gt;
&lt;img alt="/images/cumulative_sumprod/pairwise_sum.png" class="align-center" src="https://blosc.org/images/cumulative_sumprod/pairwise_sum.png" style="width: 50%;"&gt;
&lt;p&gt;Unfortunately, such an approach is not possible for cumulative sums since, as discussed above, order matters! One possibility is to use Kahan summation (the &lt;a class="reference external" href="https://en.wikipedia.org/wiki/Kahan_summation_algorithm"&gt;Wikipedia article is excellent&lt;/a&gt;), which does have additional costs (both in terms of FLOPS and memory consumption) although these are not prohibitive. One essentially keeps track of the rounding errors incurred with an auxiliary running total and uses this to correct the sum:&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code python"&gt;&lt;a id="rest_code_1dbc57ea73524d069d6e94bde73fd3be-1" name="rest_code_1dbc57ea73524d069d6e94bde73fd3be-1" href="https://blosc.org/posts/cumsum/#rest_code_1dbc57ea73524d069d6e94bde73fd3be-1"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# Kahan summation algorithm&lt;/span&gt;
&lt;a id="rest_code_1dbc57ea73524d069d6e94bde73fd3be-2" name="rest_code_1dbc57ea73524d069d6e94bde73fd3be-2" href="https://blosc.org/posts/cumsum/#rest_code_1dbc57ea73524d069d6e94bde73fd3be-2"&gt;&lt;/a&gt;&lt;span class="n"&gt;tot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;a id="rest_code_1dbc57ea73524d069d6e94bde73fd3be-3" name="rest_code_1dbc57ea73524d069d6e94bde73fd3be-3" href="https://blosc.org/posts/cumsum/#rest_code_1dbc57ea73524d069d6e94bde73fd3be-3"&gt;&lt;/a&gt;&lt;span class="n"&gt;tracker&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;a id="rest_code_1dbc57ea73524d069d6e94bde73fd3be-4" name="rest_code_1dbc57ea73524d069d6e94bde73fd3be-4" href="https://blosc.org/posts/cumsum/#rest_code_1dbc57ea73524d069d6e94bde73fd3be-4"&gt;&lt;/a&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;el&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;a id="rest_code_1dbc57ea73524d069d6e94bde73fd3be-5" name="rest_code_1dbc57ea73524d069d6e94bde73fd3be-5" href="https://blosc.org/posts/cumsum/#rest_code_1dbc57ea73524d069d6e94bde73fd3be-5"&gt;&lt;/a&gt;    &lt;span class="n"&gt;corrected_el&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;el&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="c1"&gt;# nudge el with accumulated lost digits&lt;/span&gt;
&lt;a id="rest_code_1dbc57ea73524d069d6e94bde73fd3be-6" name="rest_code_1dbc57ea73524d069d6e94bde73fd3be-6" href="https://blosc.org/posts/cumsum/#rest_code_1dbc57ea73524d069d6e94bde73fd3be-6"&gt;&lt;/a&gt;    &lt;span class="n"&gt;temp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tot&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;corrected_el&lt;/span&gt; &lt;span class="c1"&gt;# lose last few digits of el&lt;/span&gt;
&lt;a id="rest_code_1dbc57ea73524d069d6e94bde73fd3be-7" name="rest_code_1dbc57ea73524d069d6e94bde73fd3be-7" href="https://blosc.org/posts/cumsum/#rest_code_1dbc57ea73524d069d6e94bde73fd3be-7"&gt;&lt;/a&gt;    &lt;span class="n"&gt;tracker&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;temp&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;tot&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;corrected_el&lt;/span&gt;  &lt;span class="c1"&gt;# store the lost digits of el&lt;/span&gt;
&lt;a id="rest_code_1dbc57ea73524d069d6e94bde73fd3be-8" name="rest_code_1dbc57ea73524d069d6e94bde73fd3be-8" href="https://blosc.org/posts/cumsum/#rest_code_1dbc57ea73524d069d6e94bde73fd3be-8"&gt;&lt;/a&gt;    &lt;span class="n"&gt;tot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;temp&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;In implementation, we calculate the cumulative sum on a decompressed chunk in order and then carry forward the last element of the cumulative sum (i.e. the sum of the whole chunk) to the next chunk, incrementing the result of the cumulative sum by this carried-over value to give the &lt;em&gt;global&lt;/em&gt; cumulative sum. Thus, we can use Kahan summation between the small(er) values of the local chunk cumulative sum and the large(r) carried-forward running total to try and conserve precision.&lt;/p&gt;
&lt;p&gt;Unfortunately, we still observe discrepancies with respect to the Numpy implementation (which sums element-by-element essentially) of cumulative sum - but this also differs from the results of &lt;code class="docutils literal"&gt;np.sum&lt;/code&gt; due to the latter's use of pairwise summation! Finite arithmetic imposes an insuperable barrier: three different algorithms cannot guarantee agreement in every possible case. Since the Kahan sum approach has a slight overhead, we decided to junk it, as it did not improve precision sufficiently to justify its use.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="experiments"&gt;
&lt;h2&gt;Experiments&lt;/h2&gt;
&lt;p&gt;We performed some experiments comparing the new &lt;code class="docutils literal"&gt;blosc2.cumulative_sum&lt;/code&gt; function to Numpy's version for some large arrays of (of size &lt;code class="docutils literal"&gt;(N, N, N)&lt;/code&gt; for various values of &lt;code class="docutils literal"&gt;N&lt;/code&gt;). Since the working set is double the size of the input array (input + output), we expect to see significant benefits from Blosc2 compression and exploitation of caching. Indeed, once the working set size starts to approach the available RAM (32 GB), NumPy begins to slow down rapidly and when the working set exceeds memory and swap must be used NumPy becomes vastly slower.&lt;/p&gt;
&lt;img alt="/images/cumulative_sumprod/cumsumbench.png" class="align-center" src="https://blosc.org/images/cumulative_sumprod/cumsumbench.png" style="width: 50%;"&gt;
&lt;p&gt;The plot shows the average computation time for &lt;code class="docutils literal"&gt;cumulative_sum&lt;/code&gt; over the three different axes of the input array. The benchmark code may be found &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/blob/main/bench/ndarray/cumsum_bench.py"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="conclusions"&gt;
&lt;h2&gt;Conclusions&lt;/h2&gt;
&lt;p&gt;Blosc2 achieves superior compression and enables computation on larger datasets by tightly integrating compression and computation and interleaving I/O and computation. The returns on such an approach are clear in an era of &lt;a class="reference external" href="https://arstechnica.com/gadgets/2025/11/spiking-memory-prices-mean-that-it-is-once-again-a-horrible-time-to-build-a-pc/"&gt;increasingly expensive RAM&lt;/a&gt; and thus increasingly desirable memory efficiency. As an array library catering in a unique way to this growing need, bringing Blosc2 into greater alignment with the interlibrary array API standard is of utmost importance to ease its integration into users' workflows and applications. We are thus especially pleased that the performance of the freshly-implemented cumulative reduction operations mandated by the Array API standard only underline the validity of chunkwise operations.&lt;/p&gt;
&lt;p&gt;The Blosc team isn't resting on our laurels either, as we continue to optimise the existing framework to accelerate computations further. The recent introduction of the &lt;code class="docutils literal"&gt;miniexpr&lt;/code&gt; library into the backend is the capstone to these efforts, and has made the compression/computation integration truly seamless, &lt;a class="reference external" href="https://ironarray.io/blog/miniexpr-powered-blosc2"&gt;bringing incredible speedups for memory-bound computations&lt;/a&gt;, justifying Blosc2's compression-first, cache-aware philosophy. This all allows Blosc2 to handle significantly larger working sets than other solutions, delivering high performance for both in-memory and on-disk datasets, even exceeding available RAM.&lt;/p&gt;
&lt;p&gt;If you find our work useful and valuable, we would be grateful if you could support us by &lt;a class="reference external" href="https://www.blosc.org/pages/donate/"&gt;making a donation&lt;/a&gt;. Your contribution will help us continue to develop and improve Blosc packages, making them more accessible and useful for everyone.  Our team is committed to creating high-quality and efficient software, and your support will help us to achieve this goal.&lt;/p&gt;
&lt;/section&gt;</description><category>blosc array-api reductions computation</category><guid>https://blosc.org/posts/cumsum/</guid><pubDate>Mon, 16 Feb 2026 10:32:20 GMT</pubDate></item><item><title> OpenZL Plugin for Blosc2</title><link>https://blosc.org/posts/openzl-plugin/</link><dc:creator>Luke Shaw</dc:creator><description>&lt;p&gt;Blosc's philosophy of meta-compression is incredibly powerful - one is able to compose pipelines to optimally compress data (for speed or compression ratio), store information about the pipeline alognside the data in metadata, and then rely on a generic decompressor to read this and reverse the pipeline. The OpenZL team share our belief in the validity of this approach and have designed &lt;a class="reference external" href="https://openzl.org/"&gt;a graph-based formalisation with extensive support for all kinds of compression pipelines&lt;/a&gt; for all kinds of data.&lt;/p&gt;
&lt;p&gt;However, Blosc2 is now much more than just a compression library - it offers comprehensive indexing support (including fancy indexing via the python-blosc2 interface) as well as an increasingly rapid compute engine (see &lt;a class="reference external" href="https://ironarray.io/blog/miniexpr-powered-blosc2"&gt;this blog!&lt;/a&gt;). What if we could marry the incredibly comprehensive compression coverage of OpenZL with Blosc2's extended array manipulation functionality?&lt;/p&gt;
&lt;p&gt;Foreseeing precisely this sort of challenge, prior Blosc2 developers implemented a dynamic plugin register functionality (loading the plugin in C-Blosc2, which can be called via Python-Blosc2). This means that with some unintrusive, relatively concise interface code, one can link Blosc2 and OpenZL at runtime (without substantially modifying either) and offer Blosc2 arrays compressed and decompressed with OpenZL.&lt;/p&gt;
&lt;section id="the-openzl-plugin"&gt;
&lt;h2&gt;The OpenZL plugin&lt;/h2&gt;
&lt;p&gt;The source code for the plugin can be found &lt;a class="reference external" href="https://github.com/Blosc/blosc2-openzl"&gt;here&lt;/a&gt;. The minimal skeleton for the plugin layout follows&lt;/p&gt;
&lt;pre class="literal-block"&gt;├── CMakeLists.txt
├── blosc2_openzl
│   └── __init__.py
├── pyproject.toml
├── requirements-build.txt
└── src
    ├── CMakeLists.txt
    ├── blosc2_openzl.c
    └── blosc2_openzl.h&lt;/pre&gt;
&lt;p&gt;The &lt;code class="docutils literal"&gt;blosc2_openzl.c&lt;/code&gt; must implement an encoder and decoder which are exported via an &lt;code class="docutils literal"&gt;info&lt;/code&gt; struct:&lt;/p&gt;
&lt;pre class="literal-block"&gt;#include "blosc2_openzl.h"

BLOSC2_OPENZL_EXPORT codec_info info = {
    .encoder=(char *)"blosc2_openzl_encoder",
    .decoder=(char *)"blosc2_openzl_decoder"
};

int blosc2_openzl_encoder(const uint8_t* src, uint8_t* dest,
                                  int32_t size, uint8_t meta,
                                  blosc2_cparams *cparams, uint8_t id) {
  // code
}


int blosc2_openzl_decoder(const uint8_t *input, int32_t input_len, uint8_t *output,
                            int32_t output_len, uint8_t meta, blosc2_dparams *dparams,
                            const void *chunk) {
  // code
}&lt;/pre&gt;
&lt;p&gt;The header &lt;code class="docutils literal"&gt;blosc2_openzl.h&lt;/code&gt; then makes the &lt;code class="docutils literal"&gt;info&lt;/code&gt; and &lt;code class="docutils literal"&gt;encoder/decoder&lt;/code&gt; functions available to Blosc2:&lt;/p&gt;
&lt;pre class="literal-block"&gt;#include "blosc2.h"
#include "blosc2/codecs-registry.h"
#include "openzl/openzl.h"

BLOSC2_OPENZL_EXPORT int blosc2_openzl_encoder(...);

BLOSC2_OPENZL_EXPORT int blosc2_openzl_decoder(...);

// Declare the info struct as extern
extern BLOSC2_OPENZL_EXPORT codec_info info;&lt;/pre&gt;
&lt;/section&gt;
&lt;section id="pep-427-and-wheel-structure"&gt;
&lt;h2&gt;PEP 427 and wheel structure&lt;/h2&gt;
&lt;p&gt;In order for the plugin to dynamically link to Blosc2, it has to be able to find the Blosc2 library at runtime. This has historically been quite finicky since different platforms and package managers may store Python packages (and the associated &lt;code class="docutils literal"&gt;&lt;span class="pre"&gt;.so/.dylib/.dll&lt;/span&gt;&lt;/code&gt; library objects differently). Consequently, PEP 427 recommends distributing the Python wheels for packages which depend on compiled objects such as Python-Blosc2 in the following way&lt;/p&gt;
&lt;pre class="literal-block"&gt;blosc2
  ├── __init__.py
  ├── lib
  │   ├── libblosc2.so
  │   ├── cmake
  │   └── pkgconfig
  └── include
      └── blosc2.h&lt;/pre&gt;
&lt;p&gt;Finding the necessary &lt;code class="docutils literal"&gt;libblosc2.so&lt;/code&gt; object from the top-level &lt;code class="docutils literal"&gt;CMakeLists.txt&lt;/code&gt; file for the plugin is then as easy as:&lt;/p&gt;
&lt;pre class="literal-block"&gt;# Find blosc2 package location using Python
execute_process(
    COMMAND "${Python_EXECUTABLE}" -c "import blosc2, pathlib; print(pathlib.Path(blosc2.__file__).parent)"
    OUTPUT_VARIABLE BLOSC2_PACKAGE_DIR
)
set(BLOSC2_INCLUDE_DIR "${BLOSC2_PACKAGE_DIR}/include")
set(BLOSC2_LIB_DIR "${BLOSC2_PACKAGE_DIR}/lib")&lt;/pre&gt;
&lt;p&gt;After building the plugin backend in &lt;code class="docutils literal"&gt;src/CMakelists.txt&lt;/code&gt; one simply links the plugin to the backend (in this case &lt;code class="docutils literal"&gt;openzl&lt;/code&gt;) and installs like so:&lt;/p&gt;
&lt;pre class="literal-block"&gt;add_library(blosc2_openzl SHARED blosc2_openzl.c)
target_include_directories(blosc2_openzl PUBLIC ${BLOSC2_INCLUDE_DIR})
target_link_libraries(blosc2_openzl ${OPENZL_TARGET})
# Install
install(TARGETS blosc2_openzl
    RUNTIME DESTINATION blosc2_openzl
    LIBRARY DESTINATION blosc2_openzl
)&lt;/pre&gt;
&lt;p&gt;Note that it is not necessary to link &lt;code class="docutils literal"&gt;blosc2_openzl&lt;/code&gt; and &lt;code class="docutils literal"&gt;blosc2&lt;/code&gt; in &lt;code class="docutils literal"&gt;target_link_libraries&lt;/code&gt; as the former depends only on macros and structs defined in header files - and not functions. This makes the &lt;code class="docutils literal"&gt;libblosc2_openzl.so&lt;/code&gt; object especially light and robust, as blosc2 is not registered as an explicit dependency. In fact on Linux, even if the &lt;code class="docutils literal"&gt;blosc2_openzl.c&lt;/code&gt; were to include blosc2 functions, it is still not necessary to perform such linking!&lt;/p&gt;
&lt;p&gt;Following PEP 427 allows one to add an additional safeguard to check if the plugin fails to find blosc2 by adding the RUNTIME_PATH property to the installed object&lt;/p&gt;
&lt;pre class="literal-block"&gt;set_target_properties(blosc2_openzl PROPERTIES
    INSTALL_RPATH "$ORIGIN/../blosc2/lib"
)&lt;/pre&gt;
&lt;p&gt;It also allows one to easily find the plugin &lt;code class="docutils literal"&gt;.so&lt;/code&gt; object when calling via python - in the &lt;code class="docutils literal"&gt;blosc2_openzl/__init__.py&lt;/code&gt; file one can find the library path as easily as &lt;code class="docutils literal"&gt;&lt;span class="pre"&gt;os.path.abspath(Path(__file__).parent&lt;/span&gt; / libname)&lt;/code&gt; where &lt;code class="docutils literal"&gt;libname&lt;/code&gt; is the desired &lt;code class="docutils literal"&gt;&lt;span class="pre"&gt;.so/.dylib/.dll&lt;/span&gt;&lt;/code&gt; object (depending on platform). All these benefits have led us to update the wheel structure for &lt;code class="docutils literal"&gt;&lt;span class="pre"&gt;python-blosc2&lt;/span&gt;&lt;/code&gt; in the latest 4.0 release.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="using-openzl-from-python"&gt;
&lt;h2&gt;Using OpenZL from Python&lt;/h2&gt;
&lt;p&gt;Installing is then as simple as:&lt;/p&gt;
&lt;pre class="literal-block"&gt;pip install blosc2_openzl&lt;/pre&gt;
&lt;p&gt;One can also download the project and use the &lt;code class="docutils literal"&gt;cmake&lt;/code&gt; and &lt;code class="docutils literal"&gt;cmake &lt;span class="pre"&gt;--build&lt;/span&gt;&lt;/code&gt; commands to compile C-level tests or examples. But let's get compressing with &lt;code class="docutils literal"&gt;python&lt;/code&gt; straight away:&lt;/p&gt;
&lt;pre class="literal-block"&gt;import blosc2
import numpy as np
import blosc2_openzl
from blosc2_openzl import OpenZLProfile as OZLP
prof = OZLP.OZLPROF_SH_BD_LZ4
# Define the compression parameters for Blosc2
cparams = {'codec': blosc2.Codec.OPENZL, 'codec_meta': prof.value}

# Create (uncompressed) array
np_array = np.arange(1000).reshape((10,100))

# Compression with the OpenZL codec
bl_array = blosc2.asarray(np_array, cparams=cparams)
print(bl_array.cratio) # print compression ratio
&amp;gt;&amp;gt; 25.078369905956112&lt;/pre&gt;
&lt;dl class="simple"&gt;
&lt;dt&gt;The &lt;code class="docutils literal"&gt;OpenZLProfile&lt;/code&gt; enum contains the available profile pipelines that have been implemented in the plugin, which use the &lt;code class="docutils literal"&gt;codec_meta&lt;/code&gt; field (an 8-bit integer) to specify the desired transformation via codecs, filters and other nodes for the compression graph. Starting from the Least-Significant-Bit (LSB), setting the bits tells OpenZL how to build the graph:&lt;/dt&gt;
&lt;dd&gt;&lt;p&gt;CODEC | SHUFFLE | DELTA | SPLIT | CRC | x | x | x |&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;p&gt;CODEC - If set, use LZ4. Else ZSTD.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;SHUFFLE - If set, use shuffle (outputs a stream for every byte of input data typesize)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;DELTA - If set, apply a bytedelta (to all streams if necessary)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;SPLIT - If set, do not recombine the byte streams&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;CRC - If set, store a checksum during compression and check it during decompression&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/dd&gt;
&lt;/dl&gt;
&lt;p&gt;The remaining bits may be used in the future.&lt;/p&gt;
&lt;p&gt;In the future it would be great to further expand the OpenZL functionalities that we can offer via the plugin, such as bespoke transformers trained via machine learning techniques - see the OpenZL page for a flavour of what can be done with the (still evolving) library.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="conclusions"&gt;
&lt;h2&gt;Conclusions&lt;/h2&gt;
&lt;p&gt;C-Blosc2's ability to support dynamically loaded plugins allows the library to grow in features without increasing the size and complexity of the library itself. For more information about user-defined plugins, refer to this &lt;a class="reference external" href="https://www.blosc.org/posts/registering-plugins/"&gt;blog entry&lt;/a&gt;. We have put this to work to offer linkage with the rather complex OpenZL library with a relatively rapid turnaround from design to prototype to full release in around a month. This is thanks to prior hard work by open source contributors from Blosc but naturally also OpenZL - many thanks to all!&lt;/p&gt;
&lt;p&gt;If you find our work useful and valuable, we would be grateful if you could support us by &lt;a class="reference external" href="https://www.blosc.org/pages/donate/"&gt;making a donation&lt;/a&gt;. Your contribution will help us continue to develop and improve Blosc packages, making them more accessible and useful for everyone.  Our team is committed to creating high-quality and efficient software, and your support will help us to achieve this goal.&lt;/p&gt;
&lt;/section&gt;</description><category>blosc plugins codecs openzl</category><guid>https://blosc.org/posts/openzl-plugin/</guid><pubDate>Fri, 30 Jan 2026 10:32:20 GMT</pubDate></item><item><title>The Surprising Speed of Compressed Data: A Roofline Story</title><link>https://blosc.org/posts/roofline-analysis-blosc2/</link><dc:creator>Francesc Alted</dc:creator><description>&lt;p&gt;Can a library designed for computing with compressed data ever hope to outperform highly optimized numerical engines like NumPy and Numexpr? The answer is complex, and it hinges on the "memory wall" — a phenomenon which occurs when system memory limitations start to drag on CPU. This post uses Roofline analysis to explore this very question, dissecting the performance of Blosc2 and revealing the surprising scenarios where it can gain a competitive edge.&lt;/p&gt;
&lt;aside class="admonition note"&gt;
&lt;p class="admonition-title"&gt;Note&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Update on 2026-02-06:&lt;/strong&gt; We have published a follow-up post, &lt;a class="reference external" href="https://ironarray.io/blog/miniexpr-powered-blosc2"&gt;Python-Blosc2 4.0: Unleashing Compute Speed with miniexpr&lt;/a&gt;, which revisits this topic. This new post explains how the integration of miniexpr into Blosc2's compute engine has significantly improved performance—especially for in-memory operations—updating the conclusions drawn in this original analysis. We highly recommend reading the new post for the latest insights.&lt;/p&gt;
&lt;/aside&gt;
&lt;section id="tl-dr"&gt;
&lt;h2&gt;TL;DR&lt;/h2&gt;
&lt;p&gt;Before we dive in, here's what we discovered:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;p&gt;For in-memory tasks, Blosc2's overhead can make it slower than Numexpr, especially on x86 CPUs.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;This changes on Apple Silicon, where Blosc2's performance is much more competitive.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;For on-disk tasks, Blosc2 consistently outperforms NumPy/Numexpr on both platforms.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The "memory wall" is real, and disk I/O is an even bigger one, which is where compression shines.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/section&gt;
&lt;section id="a-trip-down-memory-lane"&gt;
&lt;h2&gt;A Trip Down Memory Lane&lt;/h2&gt;
&lt;p&gt;Let's rewind to 2008. NumPy 1.0 was just a toddler, and the computing world was buzzing with the arrival of multi-core CPUs and their shiny new SIMD instructions. On the &lt;a class="reference external" href="https://mail.python.org/archives/list/numpy-discussion@python.org/thread/YPX5PGM5WZXQAMQ5AZLLEU67D5RZBOVH/#YFX3G2RYHTIYMFDPCHKHED5F7CT4OTVK"&gt;NumPy mailing list&lt;/a&gt;, a group of us were brainstorming how to harness this new power to make Python's number-crunching faster.&lt;/p&gt;
&lt;p&gt;The idea seemed simple: trust newer compilers to use SIMD (and, possibly, data alignment) to perform operations on multiple data points at once. To test this, a &lt;a class="reference external" href="https://mail.python.org/archives/list/numpy-discussion@python.org/message/S2IEJV7U7TXHQLEMORGME6KIGRZTG33L/"&gt;simple benchmark&lt;/a&gt; was shared: multiply two large vectors element-wise. Developers from around the community ran the code and shared their results. What came back was a revelation.&lt;/p&gt;
&lt;p&gt;For small arrays that fit snugly into the CPU's high-speed cache, SIMD was quite good at accelerating computations. But as soon as the arrays grew larger, the performance boost vanished. Some of us were already suspicious about the new "memory wall" that had been growing lately, seemingly due to the widening gap between CPU speeds and memory bandwidth.  However, a conclusive answer (and solution) was still lacking.&lt;/p&gt;
&lt;p&gt;But amidst the confusion, a curious anomaly emerged. One machine, belonging to NumPy legend Charles Harris, was consistently outperforming the rest—even those with faster processors. It made no sense. We checked our code, our compilers, everything. Yet, his machine remained inexplicably faster. The answer, when it finally came, wasn't in the software at all. Charles, a hardware wizard, had &lt;a class="reference external" href="https://mail.python.org/archives/list/numpy-discussion@python.org/message/YFX3G2RYHTIYMFDPCHKHED5F7CT4OTVK/"&gt;tinkered with his BIOS to overclock his RAM&lt;/a&gt; from 667 MHz to a whopping 800 MHz.&lt;/p&gt;
&lt;p&gt;That was my lightbulb moment: for data-intensive tasks, raw CPU clock speed was not the limiting factor; memory bandwidth was what truly mattered.&lt;/p&gt;
&lt;p&gt;This led me to a wild idea: what if we could make memory &lt;em&gt;effectively&lt;/em&gt; faster? What if we could compress data in memory and decompress it on-the-fly, just in time for the CPU? This would &lt;a class="reference external" href="https://www.blosc.org/docs/StarvingCPUs-CISE-2010.pdf"&gt;slash the amount of data being moved&lt;/a&gt;, boosting our effective memory bandwidth. That idea became the seed for &lt;a class="reference external" href="https://www.blosc.org"&gt;Blosc&lt;/a&gt;, a project I started in 2010 that has been &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2"&gt;my passion ever since&lt;/a&gt;. Now, 15 years later, it is time to revisit that idea and see how well it holds up in today's computing landscape.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="roofline-model-understanding-the-memory-wall"&gt;
&lt;h2&gt;Roofline Model: Understanding the Memory Wall&lt;/h2&gt;
&lt;p&gt;Not all computations are equally affected by the memory wall - in general performance can be either CPU-bound or memory-bound. To diagnose which resource is the limiting factor, the &lt;a class="reference external" href="https://en.wikipedia.org/wiki/Roofline_model"&gt;Roofline model&lt;/a&gt; provides an insightful analytical framework. This model &lt;a class="reference external" href="https://docs.nersc.gov/tools/performance/roofline/"&gt;plots computational performance against arithmetic intensity&lt;/a&gt; (i.e. floating-point operations per second versus memory accesses per second) to visually determine whether a task is constrained by CPU speed or memory bandwidth.&lt;/p&gt;
&lt;img alt="/images/roofline-surprising-story/roofline-intro.avif" src="https://blosc.org/images/roofline-surprising-story/roofline-intro.avif"&gt;
&lt;p&gt;We will use Roofline plots to analyze Blosc2's performance, compared to that of NumPy and Numexpr. NumPy, with its highly optimized linear algebra backends, and Numexpr, with its efficient evaluation of element-wise expressions, together form a strong performance baseline for the full range of arithmetic intensities tested.&lt;/p&gt;
&lt;p&gt;To highlight the role of memory bandwidth, we will conduct our benchmarks on an AMD Ryzen 7800X3D CPU at two different memory speeds: the standard 4800 MTS and an overclocked 6000 MTS. This allows us to directly observe how memory frequency impacts computational performance.&lt;/p&gt;
&lt;p&gt;To cover a range of computational scenarios, our benchmarks include five operations with varying arithmetic intensities:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Very Low&lt;/strong&gt;: A simple element-wise addition (a + b + c).&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Low&lt;/strong&gt;: A moderately complex element-wise expression (sqrt(a + 2 * b + (c / 2)) ^ 1.2).&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Medium&lt;/strong&gt;: A highly complex element-wise calculation involving trigonometric and exponential functions.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;High&lt;/strong&gt;: Matrix multiplication on small matrices (labeled matmul0).&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Very High&lt;/strong&gt;: Matrix multiplication on large matrices (labeled matmul1 and matmul2).&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;img alt="/images/roofline-surprising-story/roofline-mem-speed-AMD-7800X3D.png" src="https://blosc.org/images/roofline-surprising-story/roofline-mem-speed-AMD-7800X3D.png"&gt;
&lt;p&gt;The Roofline plot confirms that increasing memory speed only benefits memory-bound operations (low arithmetic intensity), while CPU-bound tasks (high arithmetic intensity) are unaffected, as expected. Although this might suggest the "memory wall" is not a major obstacle, low-intensity operations like element-wise calculations, reductions, and selections are extremely common and often create performance bottlenecks. Therefore, optimizing for memory performance remains crucial.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="the-in-memory-surprise-why-wasn-t-compression-faster"&gt;
&lt;h2&gt;The In-Memory Surprise: Why Wasn't Compression Faster?&lt;/h2&gt;
&lt;p&gt;We benchmarked Blosc2 (both compressed and uncompressed) against NumPy and Numexpr. For this test, Blosc2 was configured with the LZ4 codec and shuffle filter, a setup known for its balance of speed and compression ratio.  The benchmarks were executed on an AMD Ryzen 7800X3D CPU with memory speed set to 6000 MTS, ensuring optimal memory bandwidth for the tests.&lt;/p&gt;
&lt;img alt="/images/roofline-surprising-story/roofline-7800X3D-mem-def.png" src="https://blosc.org/images/roofline-surprising-story/roofline-7800X3D-mem-def.png"&gt;
&lt;p&gt;The analysis reveals a surprising outcome: for memory-bound operations, Blosc2 is up to five times slower than Numexpr. Although operating on compressed data provides a marginal improvement over uncompressed Blosc2, it is not enough to overcome this performance gap. This result is unexpected because Blosc2 leverages Numexpr internally, and the reduced memory bandwidth from compression should theoretically lead to better performance in these scenarios.&lt;/p&gt;
&lt;p&gt;To understand this counter-intuitive result, we must examine Blosc2's core architecture. The key lies in its double partitioning scheme, which, while powerful, introduces an overhead that can negate the benefits of compression in memory-bound contexts.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="unpacking-the-overhead-a-look-inside-blosc2-s-architecture"&gt;
&lt;h2&gt;Unpacking the Overhead: A Look Inside Blosc2's Architecture&lt;/h2&gt;
&lt;p&gt;The performance characteristics of Blosc2 are rooted in its double partitioning architecture, which organizes data into chunks and blocks.&lt;/p&gt;
&lt;img alt="/images/roofline-surprising-story/double-partition-b2nd.avif" src="https://blosc.org/images/roofline-surprising-story/double-partition-b2nd.avif"&gt;
&lt;p&gt;This design is crucial for both aligning with the CPU's memory hierarchy and enabling efficient multidimensional array representation (important for things like e.g. n-dimensional slicing). However, this structure introduces an inherent overhead from additional indexing logic. In memory-bound scenarios, this latency counteracts the performance gains from reduced memory traffic, explaining why Blosc2 does not surpass Numexpr.&lt;/p&gt;
&lt;p&gt;Conversely, as arithmetic intensity increases, the computational demands begin to dominate the total execution time. In these CPU-bound regimes, the partitioning overhead is effectively amortized, allowing Blosc2 to close the performance gap and eventually match NumPy's performance in tasks like large matrix multiplications.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="modern-arm-architectures"&gt;
&lt;h2&gt;Modern ARM Architectures&lt;/h2&gt;
&lt;p&gt;CPU architecture is a rapidly evolving field. To investigate how these changes impact performance, we extended our analysis to the Apple Silicon M4 Pro, a modern ARM-based processor.&lt;/p&gt;
&lt;img alt="/images/roofline-surprising-story/roofline-m4pro-mem-def.png" src="https://blosc.org/images/roofline-surprising-story/roofline-m4pro-mem-def.png"&gt;
&lt;p&gt;The results show that Blosc2 performs significantly better on this platform, narrowing the performance gap with NumPy/NumExpr, especially for operations on compressed data. While compute engines optimized for uncompressed data still hold an edge, these findings suggest that compression will play an increasingly important role in improving computational performance in the future.&lt;/p&gt;
&lt;p&gt;However, while the in-memory results are revealing, they don't tell the whole story. Blosc2 was designed not just to fight the memory wall, but to conquer an even greater bottleneck: disk I/O. Although compression has the benefit of fitting more data into RAM when used in-memory (which is per se extremely interesting in these times, where &lt;a class="reference external" href="https://arstechnica.com/gadgets/2025/11/spiking-memory-prices-mean-that-it-is-once-again-a-horrible-time-to-build-a-pc/"&gt;RAM prices skyrocketed&lt;/a&gt;), its true power is unleashed when computations move off-motherboard. Now, let's shift the battlefield to the disk and see how Blosc2 performs in its native territory.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="a-different-battlefield-blosc2-shines-with-on-disk-data"&gt;
&lt;h2&gt;A Different Battlefield: Blosc2 Shines with On-Disk Data&lt;/h2&gt;
&lt;p&gt;Blosc2's architecture extends its computational engine to operate seamlessly on data stored on disk, a significant advantage for large-scale analysis.  This is particularly relevant in scenarios where datasets exceed available memory, necessitating out-of-core processing, as commonly encountered in data science, machine learning workflows or &lt;a class="reference external" href="https://ironarray.io/cat2cloud"&gt;cloud computing environments&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Our on-disk benchmarks were designed to use datasets larger than the system's available memory to prevent filesystem caching from influencing the results. To establish a baseline, we implemented an out-of-core solution for NumPy/NumExpr, leveraging memory-mapped files. Here Blosc2 has a performance edge, particularly for memory-bound operations on compressed data, being able to send and receive data faster to disk than the memory-mapped NumPy arrays.&lt;/p&gt;
&lt;p&gt;In this case, we've used high-performance NVMe SSDs (NVMe 4.0) to minimize the impact of disk speed on the results.  We also switched to the ZSTD codec for Blosc2, as its superior compression ratio over LZ4 further minimizes data transfer to and from the disk.&lt;/p&gt;
&lt;p&gt;First, let's see the results for the AMD Ryzen 7800X3D system:&lt;/p&gt;
&lt;img alt="/images/roofline-surprising-story/roofline-7800X3D-disk-def.png" src="https://blosc.org/images/roofline-surprising-story/roofline-7800X3D-disk-def.png"&gt;
&lt;p&gt;The plots above show that Blosc2 outperforms both NumPy and Numexpr for all low-to-medium intensity operations. This is because the high latency of disk I/O amortizes the overhead of Blosc2's double partitioning scheme. Furthermore, the reduced bandwidth required for compressed data gives Blosc2 an additional performance advantage in this scenario.&lt;/p&gt;
&lt;p&gt;Now, let's see the results for the Apple Silicon M4 Pro system:&lt;/p&gt;
&lt;img alt="/images/roofline-surprising-story/roofline-m4pro-disk-def.png" src="https://blosc.org/images/roofline-surprising-story/roofline-m4pro-disk-def.png"&gt;
&lt;p&gt;On the Apple Silicon M4 Pro system, Blosc2 again outperforms both NumPy and Numexpr for all on-disk operations, mirroring the results from the AMD system. However, the performance advantage is even more significant here, especially for memory-bound tasks. This is mainly because memory-mapped arrays are less efficient on Apple Silicon than on x86_64 systems, increasing the overhead for the NumPy/Numexpr baseline.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="roofline-plot-in-memory-vs-on-disk"&gt;
&lt;h2&gt;Roofline Plot: In-Memory vs On-Disk&lt;/h2&gt;
&lt;p&gt;To better understand the trade-offs between in-memory and on-disk processing with Blosc2, the following plot contrasts their performance characteristics for compressed data:&lt;/p&gt;
&lt;img alt="/images/roofline-surprising-story/roofline-mem-disk-def.png" src="https://blosc.org/images/roofline-surprising-story/roofline-mem-disk-def.png"&gt;
&lt;p&gt;A notable finding for the AMD system is that Blosc2's on-disk operations are noticeably faster than its in-memory operations, especially for memory-bound tasks (low arithmetic intensity). This is likely due to two factors: first, the larger datasets used for on-disk tests allow Blosc2 to use more efficient internal partitions (chunks and blocks), and second, parallel data reads from disk further reduce bandwidth requirements.&lt;/p&gt;
&lt;p&gt;In contrast, for CPU-bound tasks (high arithmetic intensity), on-disk performance is comparable to, albeit slightly slower than, in-memory performance. The analysis also reveals a specific weakness: small matrix multiplications (matmul0) are significantly slower on-disk, identifying a clear target for future optimization.&lt;/p&gt;
&lt;p&gt;In contrast to the AMD system, the Apple Silicon M4 Pro shows that Blosc2's on-disk operations are slower than in-memory, a difference that is most significant for memory-bound tasks. This performance disparity suggests that current on-disk optimizations may favor x86_64 architectures over ARM.&lt;/p&gt;
&lt;p&gt;As with the AMD platform, CPU-bound operations exhibit similar performance for both on-disk and in-memory contexts. The notable exception remains the small matrix multiplication (matmul0), which performs significantly worse on-disk. This recurring pattern pinpoints a clear opportunity for future optimization efforts.&lt;/p&gt;
&lt;p&gt;Finally, and in addition to its on-disk performance, Blosc2 offers a significant cost advantage. With the &lt;a class="reference external" href="https://arstechnica.com/gadgets/2025/11/spiking-memory-prices-mean-that-it-is-once-again-a-horrible-time-to-build-a-pc/"&gt;recent rise in SSD prices&lt;/a&gt;, compressing data on disk becomes an economically attractive strategy, allowing you to store more data in less space and thereby reduce hardware expenses.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="reproducibility"&gt;
&lt;h2&gt;Reproducibility&lt;/h2&gt;
&lt;p&gt;All the &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/blob/main/bench/ndarray/roofline-analysis.py"&gt;benchmarks&lt;/a&gt; and &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/blob/main/bench/ndarray/roofline-plot.py"&gt;plots&lt;/a&gt; presented in this blog post can be reproduced. You are invited to run the scripts on your own hardware to explore the performance characteristics of Blosc2 in different environments. In case you get interesting results, please consider sharing them with the community!&lt;/p&gt;
&lt;/section&gt;
&lt;section id="conclusions"&gt;
&lt;h2&gt;Conclusions&lt;/h2&gt;
&lt;p&gt;In this blog post, we explored the Roofline model to analyze the performance of Blosc2, NumPy, and Numexpr. We've confirmed that memory-bound operations are significantly affected by the "memory wall", making data compression of interest when maximizing performance. However, for in-memory operations, the overhead of Blosc2's double partitioning scheme can be a limiting factor, especially on x86_64 architectures. Encouragingly, this performance gap narrows considerably on modern ARM platforms like Apple Silicon, suggesting a promising future.&lt;/p&gt;
&lt;p&gt;The situation changes dramatically for on-disk operations. Here, Blosc2 consistently outperforms NumPy and Numexpr, as the high latency of disk I/O (even if we used SSDs here) amortizes its internal overhead. This makes Blosc2 a compelling choice for out-of-core computations, one of its primary use cases.&lt;/p&gt;
&lt;p&gt;Overall, this analysis has provided valuable insights, highlighting the importance of the memory hierarchy. It has also exposed specific areas for improvement, such as the performance of small matrix multiplications. As Blosc2 continues to evolve, I am confident we can address these points and further enhance its performance, making it an even more powerful tool for numerical computations in Python.&lt;/p&gt;
&lt;hr class="docutils"&gt;
&lt;p&gt;Read more about &lt;a class="reference external" href="https://ironarray.io"&gt;ironArray SLU&lt;/a&gt; — the company behind Blosc2, Caterva2, Numexpr and other high-performance data processing libraries.&lt;/p&gt;
&lt;p&gt;Compress Better, Compute Bigger!&lt;/p&gt;
&lt;/section&gt;</description><category>Blosc2</category><category>memory wall</category><category>numexpr</category><category>numpy</category><category>performance</category><category>roofline</category><guid>https://blosc.org/posts/roofline-analysis-blosc2/</guid><pubDate>Thu, 27 Nov 2025 08:05:21 GMT</pubDate></item><item><title>Blosc2: A Universal Lazy Engine for Array Operations</title><link>https://blosc.org/posts/tensordot-pure-persistent/</link><dc:creator>Francesc Alted, Luke Shaw</dc:creator><description>&lt;p&gt;While compression is often seen merely as a way to save storage, the Blosc development team has long viewed it as a foundational element for high-performance computing. This philosophy is at the heart of Blosc2, which is not just a compression library but a powerful framework for handling large datasets. This post will highlight one of Python-Blosc2's most exciting capabilities: its lazy evaluation engine for array operations.&lt;/p&gt;
&lt;p&gt;Libraries optimised for computation on large datasets that don't fit in memory - such as Dask or Spark - often use lazy evaluation of computation expressions. This typically speeds up evaluation since one can build the full chain of computations and only execute them when the final result is needed. Consequently, Python-Blosc2's compute engine also uses the lazy imperative paradigm, which proves to be both &lt;a class="reference external" href="https://ironarray.io/blog/compute-bigger"&gt;powerful and efficient&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;An additional benefit of the engine is its ability to act as a universal backend. Python-Blosc2 has a native &lt;code class="docutils literal"&gt;blosc2.NDArray&lt;/code&gt; format, but it can also easily execute lazy operations on arrays from other popular libraries like NumPy, HDF5, Zarr, Xarray or TileDB - basically any array object which complies with a minimal protocol.&lt;/p&gt;
&lt;p&gt;In the recent &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/releases"&gt;Python-Blosc2 3.10.x series&lt;/a&gt;, we added support for lazy evaluation of eager functions, expanding the capabilities of the compute engine, and making interaction with other formats easier. Let's explore how this works using an out-of-core &lt;a class="reference external" href="https://www.blosc.org/python-blosc2/reference/linalg.html#blosc2.linalg.tensordot"&gt;tensordot&lt;/a&gt; operation as an example.&lt;/p&gt;
&lt;section id="from-eager-to-lazy-with-blosc2-lazyexpr"&gt;
&lt;h2&gt;From Eager to Lazy with &lt;code class="docutils literal"&gt;blosc2.lazyexpr&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;Functions which return a result with a different shape to the input operands - such as reductions or linear algebra operations - must be evaluated eagerly (computed and the result returned immediately). For example, &lt;code class="docutils literal"&gt;blosc2.tensordot()&lt;/code&gt; executes eagerly.&lt;/p&gt;
&lt;p&gt;Nevertheless, we can defer this computation, by wrapping the call in a string and passing it to &lt;code class="docutils literal"&gt;blosc2.lazyexpr&lt;/code&gt;. This creates a &lt;code class="docutils literal"&gt;LazyExpr&lt;/code&gt; object that represents the operation without executing it.&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code python"&gt;&lt;a id="rest_code_58bd32c9eed448869a28246e438a7c6f-1" name="rest_code_58bd32c9eed448869a28246e438a7c6f-1" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_58bd32c9eed448869a28246e438a7c6f-1"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# Assume a and b are large, on-disk blosc2 arrays&lt;/span&gt;
&lt;a id="rest_code_58bd32c9eed448869a28246e438a7c6f-2" name="rest_code_58bd32c9eed448869a28246e438a7c6f-2" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_58bd32c9eed448869a28246e438a7c6f-2"&gt;&lt;/a&gt;&lt;span class="n"&gt;axis&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;a id="rest_code_58bd32c9eed448869a28246e438a7c6f-3" name="rest_code_58bd32c9eed448869a28246e438a7c6f-3" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_58bd32c9eed448869a28246e438a7c6f-3"&gt;&lt;/a&gt;
&lt;a id="rest_code_58bd32c9eed448869a28246e438a7c6f-4" name="rest_code_58bd32c9eed448869a28246e438a7c6f-4" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_58bd32c9eed448869a28246e438a7c6f-4"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# Create a lazy expression object&lt;/span&gt;
&lt;a id="rest_code_58bd32c9eed448869a28246e438a7c6f-5" name="rest_code_58bd32c9eed448869a28246e438a7c6f-5" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_58bd32c9eed448869a28246e438a7c6f-5"&gt;&lt;/a&gt;&lt;span class="n"&gt;lexpr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;blosc2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lazyexpr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"tensordot(a, b, axes=(axis, axis))"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;a id="rest_code_58bd32c9eed448869a28246e438a7c6f-6" name="rest_code_58bd32c9eed448869a28246e438a7c6f-6" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_58bd32c9eed448869a28246e438a7c6f-6"&gt;&lt;/a&gt;
&lt;a id="rest_code_58bd32c9eed448869a28246e438a7c6f-7" name="rest_code_58bd32c9eed448869a28246e438a7c6f-7" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_58bd32c9eed448869a28246e438a7c6f-7"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# The computation has not run yet.&lt;/span&gt;
&lt;a id="rest_code_58bd32c9eed448869a28246e438a7c6f-8" name="rest_code_58bd32c9eed448869a28246e438a7c6f-8" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_58bd32c9eed448869a28246e438a7c6f-8"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# To execute it and save the result to a new persistent array:&lt;/span&gt;
&lt;a id="rest_code_58bd32c9eed448869a28246e438a7c6f-9" name="rest_code_58bd32c9eed448869a28246e438a7c6f-9" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_58bd32c9eed448869a28246e438a7c6f-9"&gt;&lt;/a&gt;&lt;span class="n"&gt;out_blosc2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lexpr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;compute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;urlpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"out.b2nd"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;This is useful, and highly efficient both in terms of computation time and memory usage, as we'll see later. But the real magic happens when we use this computation engine with other array formats.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="one-engine-many-backends"&gt;
&lt;h2&gt;One Engine, Many Backends&lt;/h2&gt;
&lt;p&gt;The &lt;code class="docutils literal"&gt;blosc2.evaluate()&lt;/code&gt; function takes the same string expression but can operate on any array-like objects that follow the &lt;code class="docutils literal"&gt;blosc2.Array&lt;/code&gt; protocol. This protocol simply requires the object to have &lt;code class="docutils literal"&gt;shape&lt;/code&gt;, &lt;code class="docutils literal"&gt;dtype&lt;/code&gt;, &lt;code class="docutils literal"&gt;__getitem__&lt;/code&gt;, and &lt;code class="docutils literal"&gt;__setitem__&lt;/code&gt; attributes, which are standard in &lt;code class="docutils literal"&gt;h5py&lt;/code&gt;, &lt;code class="docutils literal"&gt;zarr&lt;/code&gt;, &lt;code class="docutils literal"&gt;tiledb&lt;/code&gt;, &lt;code class="docutils literal"&gt;xarray&lt;/code&gt; and &lt;code class="docutils literal"&gt;numpy&lt;/code&gt; arrays.&lt;/p&gt;
&lt;p&gt;This means you can use Blosc2's efficient evaluation engine to perform out-of-core computations directly on your existing (HDF5, Zarr, etc.) datasets.&lt;/p&gt;
&lt;section id="example-with-hdf5"&gt;
&lt;h3&gt;Example with HDF5&lt;/h3&gt;
&lt;p&gt;Here, we instruct &lt;code class="docutils literal"&gt;blosc2.evaluate&lt;/code&gt; to run the &lt;code class="docutils literal"&gt;tensordot&lt;/code&gt; operation on two &lt;code class="docutils literal"&gt;h5py&lt;/code&gt; datasets and store the result in a third one.&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code python"&gt;&lt;a id="rest_code_f62c781a474446aeaeac2e90f5dee7bb-1" name="rest_code_f62c781a474446aeaeac2e90f5dee7bb-1" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_f62c781a474446aeaeac2e90f5dee7bb-1"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# Open HDF5 datasets&lt;/span&gt;
&lt;a id="rest_code_f62c781a474446aeaeac2e90f5dee7bb-2" name="rest_code_f62c781a474446aeaeac2e90f5dee7bb-2" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_f62c781a474446aeaeac2e90f5dee7bb-2"&gt;&lt;/a&gt;&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;h5py&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;File&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"a_b_out.h5"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;a id="rest_code_f62c781a474446aeaeac2e90f5dee7bb-3" name="rest_code_f62c781a474446aeaeac2e90f5dee7bb-3" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_f62c781a474446aeaeac2e90f5dee7bb-3"&gt;&lt;/a&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;a id="rest_code_f62c781a474446aeaeac2e90f5dee7bb-4" name="rest_code_f62c781a474446aeaeac2e90f5dee7bb-4" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_f62c781a474446aeaeac2e90f5dee7bb-4"&gt;&lt;/a&gt;&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"b"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;a id="rest_code_f62c781a474446aeaeac2e90f5dee7bb-5" name="rest_code_f62c781a474446aeaeac2e90f5dee7bb-5" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_f62c781a474446aeaeac2e90f5dee7bb-5"&gt;&lt;/a&gt;&lt;span class="n"&gt;out_hdf5&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"out"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;a id="rest_code_f62c781a474446aeaeac2e90f5dee7bb-6" name="rest_code_f62c781a474446aeaeac2e90f5dee7bb-6" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_f62c781a474446aeaeac2e90f5dee7bb-6"&gt;&lt;/a&gt;
&lt;a id="rest_code_f62c781a474446aeaeac2e90f5dee7bb-7" name="rest_code_f62c781a474446aeaeac2e90f5dee7bb-7" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_f62c781a474446aeaeac2e90f5dee7bb-7"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# Use blosc2.evaluate() with HDF5 arrays&lt;/span&gt;
&lt;a id="rest_code_f62c781a474446aeaeac2e90f5dee7bb-8" name="rest_code_f62c781a474446aeaeac2e90f5dee7bb-8" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_f62c781a474446aeaeac2e90f5dee7bb-8"&gt;&lt;/a&gt;&lt;span class="n"&gt;blosc2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"tensordot(a, b, axes=(axis, axis))"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;out_hdf5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Notice that the expression string is identical to the one we used before. &lt;code class="docutils literal"&gt;blosc2&lt;/code&gt; inspects the objects in the expression's namespace and computes with them, regardless of their underlying format.&lt;/p&gt;
&lt;/section&gt;
&lt;section id="example-with-zarr"&gt;
&lt;h3&gt;Example with Zarr&lt;/h3&gt;
&lt;p&gt;The same principle applies to Zarr arrays.&lt;/p&gt;
&lt;div class="code"&gt;&lt;pre class="code python"&gt;&lt;a id="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-1" name="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-1" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_ba3ab40d8ad8413b9db2f1496243dd85-1"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# Open Zarr arrays&lt;/span&gt;
&lt;a id="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-2" name="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-2" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_ba3ab40d8ad8413b9db2f1496243dd85-2"&gt;&lt;/a&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;zarr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"a.zarr"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"r"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;a id="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-3" name="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-3" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_ba3ab40d8ad8413b9db2f1496243dd85-3"&gt;&lt;/a&gt;&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;zarr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"b.zarr"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"r"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;a id="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-4" name="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-4" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_ba3ab40d8ad8413b9db2f1496243dd85-4"&gt;&lt;/a&gt;&lt;span class="n"&gt;zout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;zarr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;open_array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"out.zarr"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;a id="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-5" name="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-5" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_ba3ab40d8ad8413b9db2f1496243dd85-5"&gt;&lt;/a&gt;
&lt;a id="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-6" name="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-6" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_ba3ab40d8ad8413b9db2f1496243dd85-6"&gt;&lt;/a&gt;&lt;span class="c1"&gt;# Use blosc2.evaluate() with Zarr arrays&lt;/span&gt;
&lt;a id="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-7" name="rest_code_ba3ab40d8ad8413b9db2f1496243dd85-7" href="https://blosc.org/posts/tensordot-pure-persistent/#rest_code_ba3ab40d8ad8413b9db2f1496243dd85-7"&gt;&lt;/a&gt;&lt;span class="n"&gt;blosc2&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"tensordot(a, b, axes=(axis, axis))"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;zout&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;This makes &lt;code class="docutils literal"&gt;blosc2.evaluate&lt;/code&gt; a powerful, backend-agnostic tool for out-of-core array computations.&lt;/p&gt;
&lt;/section&gt;
&lt;/section&gt;
&lt;section id="performance-comparison"&gt;
&lt;h2&gt;Performance Comparison&lt;/h2&gt;
&lt;p&gt;As well as offering smooth integration, &lt;code class="docutils literal"&gt;blosc2.evaluate&lt;/code&gt; is highly performant. Python-Blosc2 uses a lazy evaluation engine that integrates tightly with the Blosc2 format. This means that the computation is performed on-the-fly, without any intermediate copies. This is a huge advantage for large datasets, as it allows us to perform computations on arrays that don't fit in memory.  In addition, it actively tries to leverage the hierarchical memory layout in modern CPUs, so that it can use both private and shared caches in the best way possible.&lt;/p&gt;
&lt;p&gt;We ran a &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/blob/main/bench/ndarray/tensordot_pure_persistent.ipynb"&gt;benchmark&lt;/a&gt; performing a &lt;code class="docutils literal"&gt;tensordot&lt;/code&gt; operation (run over three different axis combinations) on two 3D arrays stored on disk; we then write the output to disk as well.
We consider four approaches:&lt;/p&gt;
&lt;ol class="arabic simple"&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Blosc2 Native&lt;/strong&gt;: Using &lt;code class="docutils literal"&gt;blosc2.lazyexpr&lt;/code&gt; with &lt;code class="docutils literal"&gt;blosc2.NDArray&lt;/code&gt; containers.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Blosc2+HDF5&lt;/strong&gt;: Using &lt;code class="docutils literal"&gt;blosc2.evaluate&lt;/code&gt; with HDF5 for storage.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Blosc2+Zarr&lt;/strong&gt;: Using &lt;code class="docutils literal"&gt;blosc2.evaluate&lt;/code&gt; with Zarr for storage.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Dask+HDF5&lt;/strong&gt;: The combination of Dask for computation and HDF5 for storage.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Dask+Zarr&lt;/strong&gt;: The combination of Dask for computation and Zarr for storage.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;For each approach we plot the memory consumption vs. time for arrays of increasing size.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Results on two (600, 600, 600) float64 arrays (3 GB working set):&lt;/strong&gt;&lt;/p&gt;
&lt;img alt="/images/tensordot_pure_persistent/tensordot-600c-amd.png" src="https://blosc.org/images/tensordot_pure_persistent/tensordot-600c-amd.png" style="width: 100%;"&gt;
&lt;p&gt;&lt;strong&gt;Results on two (1200, 1200, 1200) float64 arrays (26 GB working set):&lt;/strong&gt;&lt;/p&gt;
&lt;img alt="/images/tensordot_pure_persistent/tensordot-1200c-amd.png" src="https://blosc.org/images/tensordot_pure_persistent/tensordot-1200c-amd.png" style="width: 100%;"&gt;
&lt;p&gt;&lt;strong&gt;Results on two (1500, 1500, 1500) float64 arrays (50 GB working set):&lt;/strong&gt;&lt;/p&gt;
&lt;img alt="/images/tensordot_pure_persistent/tensordot-1500c-amd.png" src="https://blosc.org/images/tensordot_pure_persistent/tensordot-1500c-amd.png" style="width: 100%;"&gt;
&lt;p&gt;As can be seen, the amount of memory required by the different approaches is very different, although none requires more than a small fraction of the total working set (which is 3, 26 and 50 GB, respectively). This is because all approaches are out-of-core, and only load small chunks of data into memory at any given time.&lt;/p&gt;
&lt;p&gt;The benchmarks were executed on an AMD Ryzen 9800X3D CPU, with 16 logical cores and 64GB of RAM, using Ubuntu Linux 25.04. We have used the following versions of the libraries: python-blosc2 3.10.1, h5py 3.14.0, zarr 3.1.3, 2025.9.1, and numpy 2.3.3.  All backends are using Blosc or Blosc2 as the compression backend, with same codecs and filters, and using the same number of threads for compression and decompression.&lt;/p&gt;
&lt;section id="analysis"&gt;
&lt;h3&gt;Analysis&lt;/h3&gt;
&lt;p&gt;The results are revealing:&lt;/p&gt;
&lt;ul class="simple"&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Blosc2 native is fastest&lt;/strong&gt;: The tight integration between the Blosc2 compute engine and its native array format yields the best performance, making it the fastest solution by a significant margin.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Rapid computation time&lt;/strong&gt;: &lt;code class="docutils literal"&gt;blosc2.evaluate&lt;/code&gt; delivers impressive speed when operating directly on HDF5 and Zarr files, outperforming the more complex Dask+HDF5 and Dask+Zarr stack. This is great news for anyone with existing HDF5/Zarr datasets.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Low memory usage&lt;/strong&gt;: While the memory consumption for the Blosc2+HDF5 combination is a bit high (we are still analyzing why), the memory usage for the Blosc2 native approach is pretty low, making it suitable for systems with limited RAM and/or operands not fitting in memory.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This is not to say that Dask (or Spark) is an inferior choice for out-of-core computations. It's a great tool for large-scale data processing, especially when using clusters, is very flexible, and offers a wide range of functions; it's certainly a first-class citizen in the PyData ecosystem. However, if your needs are more modest and you want a simple, efficient way to run computations on existing datasets, using a core of common functions, and leveraging the full capabilities of modern multi-core systems, all without the overhead of a full Dask setup, &lt;code class="docutils literal"&gt;blosc2.evaluate()&lt;/code&gt; is a fantastic alternative.&lt;/p&gt;
&lt;/section&gt;
&lt;/section&gt;
&lt;section id="conclusion"&gt;
&lt;h2&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Python-Blosc2 is more than just a compression library for storing data in &lt;code class="docutils literal"&gt;blosc2.NDArray&lt;/code&gt; objects; it's a high-performance computing tool as well. Its lazy evaluation engine provides a simple yet powerful way to handle out-of-core operations. The computation engine is completely decoupled from the compression backend, and thus can easily work with many different array formats; however, the compute engine meshes most tightly with the Blosc2 native array format, achieving maximal performance (in terms of both computation time and memory usage).&lt;/p&gt;
&lt;p&gt;By adhering to the &lt;a class="reference external" href="https://data-apis.org/array-api/"&gt;Array API standard&lt;/a&gt;, it acts as a universal engine that can work with different storage backends; we already implement &lt;a class="reference external" href="https://ironarray.io/blog/array-api"&gt;more than 100 functions that are required by that standard&lt;/a&gt;, and the number will only grow in the future. If you have existing datasets in HDF5 or Zarr or TileDB (and we are always looking forward to support even more formats), and need a lightweight, efficient way to run computations on them, &lt;code class="docutils literal"&gt;blosc2.evaluate()&lt;/code&gt; is a fantastic tool to have in your arsenal. Of course, for maximum performance, the native Blosc2 format is a clear winner.&lt;/p&gt;
&lt;p&gt;Our work continues. We are committed to enhancing Python-Blosc2 by expanding its supported operations, improving performance across backends, and adding new ones. Stay tuned for more updates! If you found this post useful, please share it. For questions or comments, reach out to us on &lt;a class="reference external" href="https://github.com/Blosc/python-blosc2/discussions"&gt;GitHub&lt;/a&gt;.&lt;/p&gt;
&lt;/section&gt;</description><category>blosc2 hdf5 zarr tiledb dask numpy</category><guid>https://blosc.org/posts/tensordot-pure-persistent/</guid><pubDate>Wed, 15 Oct 2025 10:32:20 GMT</pubDate></item></channel></rss>