EuclidRust – Rust FITS reader for Euclid, 10x faster than astropy

The euclidrust package, available on GitHub, provides a high-performance toolkit for astronomers working with Flexible Image Transport System (FITS) files. It is designed to speed up data processing for the European Space Agency's Euclid mission, which aims to map the dark matter and dark energy in the universe.

The toolkit is written in Rust and exposes its functionality to Python via PyO3, allowing it to be installed and used like any standard Python library. This approach combines the speed of compiled code with the ease of use of Python, a popular language in the scientific community.

The project was created as a testbed for data-processing tools for the Euclid mission. It is built to be useful for any FITS-based workflow, not just those specific to the mission.

High-performance Python extension

The core of euclidrust is a compiled Rust library that is installed into a Python environment. This library is then imported as the euclidrust module, making its functions available for direct use in Python scripts.

The package includes functions for calculating statistics on images, extracting cutouts around specific sky coordinates, and exporting those cutouts to the Parquet file format. These operations are performed entirely within the Rust code, which is significantly faster than pure Python implementations.

The project was developed with the help of Claude Code, an AI coding agent from Anthropic. The author, MaxIKweeger, directed the agent, deciding what to build, in what order, and how to measure performance.

Support for Euclid data formats

The toolkit is designed to work with the specific data formats used by the Euclid mission. It supports 2-D FITS images, whether they are raw or tile-compressed using Rice or GZIP algorithms.

FITS files are the standard format for astronomy. They consist of a sequence of blocks called HDUs (Header/Data Units), each containing a text header and a payload of image data or a table. Euclid's MER mosaics store their images in the primary HDU, while calibrated and raw frames have an empty primary HDU and one image extension per detector.

Users can specify which HDU to work with using the hdu argument, which accepts either an index or a name like 'DET11.SCI'. The package also includes a function to list all HDUs in a file, making it easy to identify the correct one.

Installation and usage

Installing euclidrust requires a Rust toolchain, version 1.88 or newer, and Python 3.8 or newer. The package is built using maturin, which is invoked by pip during installation.

The first installation can take a few minutes as the Rust code is compiled with full link-time optimization. Subsequent rebuilds are faster. The package has been tested on Debian 13 with Python 3.13 and Rust 1.100 on an Intel Core i7 processor.

Once installed, users can import the module and use its functions. For example, to get global statistics on an image, one can run euclidrust.fits_stats('tile.fits'). To extract a cutout around a specific sky position, the function euclidrust.extract_cutout() is used with the image path, RA and Dec coordinates, and width and height.

Performance and future plans

The primary goal of euclidrust is to provide a fast and reliable tool for astronomers. The author notes that the project should be treated as experimental and encourages users to run benchmarks themselves to judge its performance.

The project's roadmap includes plans to support more file types and improve its functionality. The current version has limitations, such as not honoring the BLANK null value for integer images and not supporting null pixels.

The author has validated the compressed-image support on copies of real Euclid data that were compressed with astropy or cfitsio. This ensures that the toolkit can handle the data it is intended to work with.