ctddump

ctddump is a small, fast command-line tool for converting oceanographic CTD (Conductivity, Temperature, Depth) data from NetCDF into analysis-ready formats:

• Parquet: the observation data, as a uniform flat table.
• YAML: the file metadata (dimensions, variables, global attributes).

It is written in Rust, streams data in bounded memory, and processes whole directory trees in parallel.

Data sources

Two families of source files are supported:

What you can do

Quick example

# One file to Parquet
ctddump convert nrt_ar input.nc output.parquet

# A whole Arctic directory, then merge into a single Parquet file
ctddump batch convert nrt_ar --output ./parquet ./netcdf
ctddump concat convert ./parquet arctic.parquet

New here? Start with Installation, skim the Commands, then follow an end-to-end regional workflow.

Installation

There are three ways to install ctddump. Pick the first one that applies:

Prebuilt binary

Each release ships an archive per platform. It contains the ctddump executable with HDF5 and netCDF built in, so no Rust toolchain and no system libraries are required:

Download, extract, and put the binary somewhere on your PATH:

VERSION=v0.27.0
TARGET=x86_64-unknown-linux-gnu

curl -LO "https://github.com/AIQC-Hub/ctddump/releases/download/$VERSION/ctddump-$VERSION-$TARGET.tar.gz"
tar -xzf "ctddump-$VERSION-$TARGET.tar.gz"
cd "ctddump-$VERSION-$TARGET"

./ctddump --version

To install it for your user, move it onto your PATH:

mkdir -p ~/.local/bin
mv ctddump ~/.local/bin/

If ctddump is still not found afterwards, ~/.local/bin is not on your PATH; add export PATH="$HOME/.local/bin:$PATH" to your shell profile.

Every archive is listed in SHA256SUMS on the same release page. To verify a download before trusting it:

curl -LO "https://github.com/AIQC-Hub/ctddump/releases/download/$VERSION/SHA256SUMS"
sha256sum -c SHA256SUMS --ignore-missing

What else is in the archive

Alongside the binary are the helper scripts, plus README.md, LICENSE, and CHANGELOG.md. The four pipeline scripts (convert_data.sh, clean_data.sh, dedup_data.sh, summary_data.sh) need only ctddump on your PATH, so they work as soon as the step above is done. Three others call external tools that are not bundled and must be installed separately: download_data.sh needs copernicusmarine, summary_site.sh needs mdbook, and fetch_test_data.sh needs gh and unzip.

The scripts are bash, so on Windows they need WSL or Git Bash.

System dependencies

The two routes below compile ctddump themselves and link the HDF5 C library, so the development headers must be installed first. (The prebuilt binaries above need none of this.)

# Ubuntu / Debian
sudo apt-get install libhdf5-dev libnetcdf-dev

# macOS (Homebrew)
brew install hdf5

Install from crates.io

With the system dependencies in place, install the published crate:

cargo install ctddump

This builds the binary and puts it in ~/.cargo/bin, which is normally already on your PATH. Check it works:

ctddump --help

If the build stops with “A system version of libnetcdf could not be found”, the development headers are missing or are somewhere the build script does not look. Install them as shown above, or point at them explicitly:

NETCDF_DIR=/path/to/netcdf HDF5_DIR=/path/to/hdf5 cargo install ctddump

Failing that, the static-netcdf feature builds HDF5 and netCDF from source instead of looking for system ones. It is much slower but needs no headers:

cargo install ctddump --features static-netcdf

Build from source

To work on ctddump itself, or to run a version that is not published yet:

git clone https://github.com/AIQC-Hub/ctddump.git
cd ctddump
cargo build --release

The binary is placed at target/release/ctddump. Copy it somewhere on your PATH (or run it directly) and check it works:

ctddump --help

Every command and subcommand supports -h / --help, so you can always discover the available options interactively:

ctddump convert --help
ctddump batch convert nrt_ar --help

Note: On systems with HDF5 ≤ 1.10 you may see harmless HDF5-DIAG messages in the output. The data is read correctly and results are unaffected.

convert

Convert a single NetCDF file into a Parquet file.

ctddump convert <subcommand> [OPTIONS] <src_file> <target_file>

The subcommand selects the source format and its defaults:

Examples

ctddump convert nrt_ar      input.nc output.parquet
ctddump convert nrt_bo      input.nc output.parquet
ctddump convert cora        input.nc output.parquet
ctddump convert cora_legacy input.nc output.parquet

Use a saved TOML preset, or override individual fields with flags:

# Apply a config preset
ctddump convert nrt_ar --config my_preset.toml input.nc output.parquet

# Override a single field
ctddump convert nrt_bo --no-deph-source input.nc output.parquet

See Configuration for the available flags and the TOML format, and Output schema for the columns produced.

batch

Process a whole directory tree in parallel. batch recursively finds every matching .nc file and converts each one, using all CPU cores by default.

ctddump batch convert <subcommand> [OPTIONS] <src_dir>
ctddump batch header  <subcommand> [OPTIONS] <src_dir>

• batch convert writes one Parquet file per source file.
• batch header writes one YAML metadata file per source file.

Each output keeps its source’s filename stem with a new extension. If --output is omitted, each result is written alongside its source; with --output <dir> all results land flat in that directory.

Examples

# Convert all NRT Arctic files to Parquet, flat into ./output
ctddump batch convert nrt_ar --output ./output ./data/arctic

# Limit to 4 threads
ctddump batch convert nrt_ar --threads 4 --output ./output ./data/arctic

# Override the filename pattern
ctddump batch convert nrt_ar --pattern "AR_PR_CT_ITP-*.nc" --output ./output ./data

# Extract YAML metadata for all NRT files
ctddump batch header nrt --output ./output ./data/arctic

Default filename patterns

--pattern matches the filename only (not the path) and supports *, ? and […]. When omitted, each subcommand uses a sensible default:

Threads. --threads N caps the worker count; omit it to use all logical cores. Memory scales with the thread count, so lower it if you are processing very large files on a memory-constrained machine.

header

Extract the metadata of a single NetCDF file to a YAML file, dimensions, variables, and global attributes. No observation data is read.

ctddump header <subcommand> <src_file> <target_file>

Examples

ctddump header nrt  input.nc output.yaml
ctddump header cora input.nc output.yaml

To extract metadata for a whole directory at once, use batch header; to combine many YAML files into one, use concat header.

concat

Merge many files from a directory tree into a single file.

ctddump concat convert [OPTIONS] <src_dir> <output_file>
ctddump concat header  [OPTIONS] <src_dir> <output_file>

• concat convert merges Parquet data files.
• concat header merges YAML metadata files.

concat convert

By default the merge re-assigns profile_no and observation_no so that profile numbers are unique and sequential within each platform. Its behaviour is controlled by a few flags:

Renumbering processes platform ranges in parallel via temporary files in the output folder. Higher thread counts are faster but use more memory, the merged result is identical either way.

Examples

# Merge all Parquet files, with profile renumbering (the default)
ctddump concat convert ./parquet merged.parquet

# Merge without renumbering
ctddump concat convert --no-renumber ./parquet merged.parquet

# Keep each profile's observations in their original order
ctddump concat convert --no-pres-sort ./parquet merged.parquet

# Keep rows with missing pres
ctddump concat convert --keep-na-pres ./parquet merged.parquet

# Sequential, lowest-memory merge
ctddump concat convert --threads 1 ./parquet merged.parquet

# Merge only a subset
ctddump concat convert --pattern "AR_PR_CT_*.parquet" ./parquet merged.parquet

concat header

Merges YAML header files, each file contributes its top-level keys to the combined output. An error is raised if any two files share the same key.

ctddump concat header ./yaml merged.yaml

report

Summarise a produced Parquet data file or YAML header file as a text report, TSV (default), plain text, or JSON, or assemble a multi-section summary page (report summary) from the per-stage TSV reports.

ctddump report parquet [--level global|platform|profile] [--format tsv|text|json] <src.parquet> [dest]
ctddump report yaml    [--format tsv|text|json] <src.yaml> [dest]
ctddump report summary [--report-dir report] [--out-dir output] [--format md|html]
                       [--title TEXT] [--note TEXT]... [-o dest] <stem>

The report is written to dest, or to stdout when dest is omitted (so it pipes cleanly into other tools).

report parquet

Aggregates a data file at one of three levels (--level, default platform):

Every row also reports:

• n_obs: number of observations.
• time_qc_good / position_qc_good: number of profiles whose flag is "1" (CMEMS “good”). These flags are per-profile; at --level profile the single flag value is shown instead.
• na_temp / na_psal / na_pres: count of missing (null/NaN) values.
• {temp,psal,pres}_{min,max,mean}: statistics over the valid (non-NaN) values. (Median is intentionally not computed.)
• {longitude,latitude}_{min,max}: geographic bounding box over the valid positions (global and platform levels only; at profile level the single longitude/latitude is already shown).

# Per-platform summary
ctddump report parquet --level platform merged.parquet report.tsv

# Whole-file summary, human-readable, to stdout
ctddump report parquet --level global --format text merged.parquet

report yaml

Summarises a merged header YAML (as produced by concat header): one row per source file.

extra_params is detected automatically: any Float variable dimensioned (TIME, DEPTH) that is not a _QC flag and not a core physical (TEMP/PSAL/PRES/DEPH). This captures biogeochemical/biological parameters (DOXY, FLU2, TUR3, CPHL, NTRA, …) as well as other non-core measurements (CNDC, SVEL, …) without a hard-coded list.

# YAML header summary as JSON
ctddump report yaml --format json merged.yaml report.json

report summary

Assembles a single Markdown (default) or HTML (--format html) page for one file stem (e.g. nrt_ar_ar) by reading the TSV reports the pipeline already produced. It does not scan any Parquet itself, it aggregates the small per-stage report TSVs, so it is instant.

Given a stem, the source files are auto-located at their standard pipeline paths under --report-dir (default report) and --out-dir (default output):

Any section whose file is absent is skipped, so a partially-run pipeline still produces a valid page (a parent heading appears only when it has at least one present subsection). The Mark-duplicates section shows its within/across tables only when the .dups.tsv is also present.

Each stage table reports platforms / profiles / observations. The % of original and Deleted columns compare against the Conversion stage (the baseline “original”); if Conversion is absent, the earliest present stage is used.

The two Deduplication stages ran on the cleaned data, not on the original, so their tables carry three further columns, Cleaned, % of cleaned and Deleted (cleaned), comparing them against the last cleaning stage present (Filter, else Drop all-NA, else Drop bad QC). When no cleaning stage ran there is nothing to compare against and the columns are omitted.

The Filter by region section carries a second table, Bounding box, with the minimum and maximum longitude and latitude of the profiles that survived the filter, in decimal degrees to three places. The extremes are aggregated from the per-platform longitude_min / longitude_max / latitude_min / latitude_max columns of the stage TSV, so they span every platform in the file. When the Conversion report is present a second column, Original, gives the same extremes for the converted data before any cleaning ran, which shows how far the filter tightened the box. Profiles with a missing position are ignored, and if no profile has a valid position the table is omitted entirely.

The Mark-duplicates section additionally lists the duplicate profiles that dedup would remove, split into:

• within a platform: a dup_group confined to a single platform_code;
• across platforms: a group spanning two or more.

(A dup_group is a set of profiles sharing the duplicate key, date + rounded position, so an across-platform group is the same cast reported by more than one platform.)

A third table, Duplicate group sizes, shows how many profiles the groups hold: one row per distinct size up to 10, then a single 11+ row for the tail.

Title and notes

Each section carries a short, generic explanation of what the stage did, the same prose for every region and dataset. Anything region- or product-specific goes in:

• --title TEXT: replaces the default Summary: <stem> heading.
• --note TEXT: a note rendered under the title; repeat for several notes.

Both are plain text and are escaped in HTML output. In the pipeline these are supplied per stem by summary_data.sh, which is the place to edit what a page says about a given region or product.

# Markdown page for the Arctic AR stem, to stdout
ctddump report summary nrt_ar_ar

# HTML page, with a title, notes, and non-default roots, to a file
ctddump report summary nrt_bo_bo --report-dir report --out-dir output \
  --format html --title "Baltic Sea: Near Real Time, regional product (BO)" \
  --note "Each source file holds a single platform." \
  -o summary/nrt_bo_bo.html

filter

Filter a produced Parquet data file by a geographic bounding box, keeping or dropping whole profiles, and write the result to a new Parquet file.

ctddump filter [--mode include|exclude] \
  --min-lon <W> --max-lon <E> --min-lat <S> --max-lat <N> \
  <src.parquet> <dest.parquet>

The box is inclusive on all four edges, and --min-* must not exceed the matching --max-* (an inverted box is rejected; antimeridian wrap is not supported).

Because a profile’s longitude/latitude are constant across its observations, the box acts on whole profiles, every observation of a matching profile is kept or dropped together.

--mode (default include):

A profile whose position is NaN (unknown) is treated as outside the box: dropped by include, kept by exclude.

The file is streamed one row group at a time, so peak memory stays bounded regardless of file size (tune with CTDDUMP_CHUNK_ROWS, as for convert).

# Keep only profiles inside a Mediterranean sub-box
ctddump filter \
  --min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 \
  merged.parquet med_box.parquet

# Remove profiles inside that box, keep the rest
ctddump filter --mode exclude \
  --min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 \
  merged.parquet outside_box.parquet

dropna

Filter a produced Parquet data file, dropping whole profiles that carry no usable data in one of the core measurement parameters, and write the survivors to a new Parquet file.

ctddump dropna <src.parquet> <dest.parquet>

A profile is kept only if each of temp, psal, and pres has at least one non-NA observation. It is dropped if any one of those parameters is entirely NA (null or NaN) across the profile. Partial NAs within a parameter are fine, a kept profile retains all of its observations, including the NA ones.

The file is processed in two streaming passes (build the keep-set, then re-emit the kept rows), so peak memory stays bounded regardless of file size and the result is independent of how the file is chunked (tune with CTDDUMP_CHUNK_ROWS, as for convert).

# Drop profiles with an all-NA temp, psal, or pres
ctddump dropna merged.parquet cleaned.parquet

dropqc

Filter a produced Parquet data file, dropping whole profiles whose profile-level quality-control flags mark them as bad, and write the survivors to a new Parquet file.

ctddump dropqc <src.parquet> <dest.parquet>

A profile is kept only if both time_qc and position_qc are either "1" (OK) or missing. A missing flag is one that is absent from the source NetCDF, or stored as the NA byte -128, both render as the empty string "" in the Parquet output. Any other flag ("0", "2"…"9", or a non-numeric char code) drops the whole profile.

Missing QC is treated as acceptable on purpose: several source files ship no profile-level position_qc (or time_qc) at all, and those profiles must be kept rather than discarded. Note that the Argo “missing value” flag 9 is a present value, not the NA byte, so a "9" flag is dropped.

Because time_qc/position_qc are constant within a profile, this is a plain per-row predicate that keeps or drops whole profiles. The file is streamed one row group at a time (set_parallel(false)), so peak memory stays bounded regardless of file size (tune with CTDDUMP_CHUNK_ROWS, as for convert).

# Drop profiles flagged bad in time_qc or position_qc
ctddump dropqc merged.parquet cleaned.parquet

markdup

Mark duplicate profiles in a produced Parquet data file with an is_dup column, and write a TSV listing the duplicated profiles.

ctddump markdup [OPTIONS] <src.parquet> <dest.parquet> <dups.tsv>

Two profiles are duplicates when they share the same key, built from:

• profile_timestamp: formatted with a strftime string; date only (%Y-%m-%d) by default.
• longitude and latitude: rounded to 3 decimals by default.

platform_code is deliberately not part of the key, so duplicates are detected across platforms. A profile whose position is NaN or whose timestamp is null has no key and is never marked.

The is_dup column (Boolean) is true for every profile whose key is shared by at least one other profile. Two streaming passes bound memory regardless of file size.

Options

Outputs

1. dest.parquet: the input plus the is_dup column.
2. dups.tsv: one row per duplicated profile, grouped by dup_group: dup_group, platform_code, profile_no, profile_time, profile_timestamp, longitude, latitude, n_obs.

# Default key: same date + coordinates rounded to 3 decimals
ctddump markdup merged.parquet marked.parquet duplicates.tsv

# Match on the full hour and 2-decimal coordinates
ctddump markdup --time-format "%Y-%m-%d %H" --decimals 2 merged.parquet marked.parquet dups.tsv

Use dedup to then drop the duplicates, and report parquet to summarise the duplicate counts.

dedup

Remove duplicate profiles from a produced Parquet data file, keeping one profile per duplicate group.

ctddump dedup [OPTIONS] <src.parquet> <dest.parquet>

dedup re-derives the same duplicate key as markdup, from profile_timestamp (date only by default) and longitude/latitude (3-decimal rounding by default), so it can run standalone with the same options. Within each group of profiles sharing a key, it keeps the profile with the most observation rows; ties are broken by first appearance. Profiles with no key (NaN position or null timestamp) are always kept.

If the input has an is_dup column (from markdup), it is reset to false: the survivors are unique, so the schema stays stable across the markdup → dedup pipeline and a follow-up report shows zero duplicates.

Two streaming passes bound memory regardless of file size.

Options

The key options match markdup and should be given the same values used there:

# Drop duplicates using the default key (same date + 3-dp coordinates)
ctddump dedup marked.parquet deduped.parquet

compare

Compare two produced Parquet data files and report how far each one covers the other’s platforms and profiles.

ctddump compare [OPTIONS] <a.parquet> <b.parquet> [dest]

Without dest the report goes to stdout.

What it answers

Two questions, for a pair of files:

• How much do they have in common? Platforms present in both, and profiles present in both.
• Do the matched profiles agree? For every profile found in both files, whether it has the same number of observation rows in each.

Coverage is reported in both directions, because it is not symmetric. A small file fully contained in a large one is 100% covered while covering only a fraction of the large one. The first output row uses the second file as the reference, the second row uses the first.

How profiles are matched

A profile’s key is built from its platform code, its time reduced to a date, and its longitude and latitude rounded to 3 decimals. This is the same key markdup uses for duplicates, with one difference: markdup deliberately leaves the platform code out, while compare includes it by default. Two files being compared are usually different extracts of the same platforms, so the platform code is normally a wanted part of the identity. Pass --no-platform-key to match on time and position alone, which finds the same cast recorded under two different platform codes.

Profiles whose time or position is missing get no key and can never match. They are counted separately as ref_unkeyed_profiles so a low coverage figure can be told apart from a file full of unusable positions.

The time column may be either a datetime (profile_timestamp) or a float of days since 1950 (profile_time); the type is detected and handled automatically.

Output columns

One row per direction:

profile_cov_pct is taken over all reference profiles, including unkeyed ones, since a profile that cannot be matched is still a profile the other file does not demonstrably have. A percentage with no denominator (comparing against an empty file, or nobs_agree_pct when nothing matched) is left empty rather than shown as zero.

When several profiles share one key, the observation counts agree if any of the profiles carrying that key in the other file has the same count.

Options

Examples

# Default key, report to stdout as an aligned table
ctddump compare --format text old.parquet new.parquet

# Save the summary
ctddump compare old.parquet new.parquet compare.tsv

# Ignore platform codes, so the same cast filed under two codes still matches
ctddump compare --no-platform-key old.parquet new.parquet

# Match to the hour and to 4 decimals instead of the date and 3
ctddump compare --time-format '%Y-%m-%dT%H' --decimals 4 old.parquet new.parquet

Reading the result: if the smaller file shows profile_cov_pct near 100 while the larger shows much less, the smaller is close to a subset. If both are low, the files overlap only partly. A high profile_cov_pct with a low nobs_agree_pct means the same profiles are present in both but carry different numbers of observations, which usually points at different QC or cleaning having been applied.

Memory is bounded: each file is streamed and reduced to one record per profile, so peak use follows the profile count rather than the file size.

Configuration

All convert and batch convert subcommands accept a --config TOML file plus individual flag overrides. The priority order is:

built-in default  <  --config file  <  individual CLI flags

--config may set any subset of fields; a per-field flag then overrides the config for that field only.

NRT flags

DEPH is auto-detected. NRT always performs the bidirectional PRES↔DEPH conversion when the file contains a DEPH variable, regardless of has_deph_source. The flag only forces DEPH handling on for files where the variable might otherwise be skipped.

CORA flags

TOML config format

# NRT
has_deph_source    = true
has_profile_coords = false
pattern            = "AR_PR_CT_*.nc"  # optional

# CORA
time_var        = "TIME"
qc_type         = "int"   # "int" or "char"
has_time_qc     = true
has_deph_source = true
pattern         = "*.nc"  # optional

Environment variables

ctddump honours a few environment variables for operational tuning. They do not change the contents of the output, only memory use, speed, and the on-disk row-group layout, which is why they live in the environment rather than as per-command flags.

CTDDUMP_CHUNK_ROWS is the only one most users would ever change, see Technical notes for why streaming in chunks matters. The helper scripts expose it directly as --chunk-rows N, which exports it for every ctddump process they launch.

Output schema

Every converter produces the same observation-level flat table, so files from different sources can be merged directly.

Pressure and depth are converted to each other with the TEOS-10 (gsw) routines when only one is present, and the *_conv columns flag which values were derived rather than measured.

Technical notes

This page collects the non-obvious technical problems we ran into while building ctddump and how each was solved. It is written for a general audience, you do not need to know Rust to follow it. The recurring themes are memory use, multi-threading, how data is read and written in chunks, and a handful of quirks in the Polars data-frame library we depend on.

Each note is laid out the same way: what the symptom looked like, why it happened, and what we did about it.

1. Converting a file used far too much memory

Symptom. Converting a single large NetCDF file (a few hundred MB on disk) pushed memory use to ~8–9 GB, enough to fail on modest machines.

Why. A NetCDF CTD file is stored as a dense rectangular grid: every profile (a cast of the instrument) times every depth level. Most of that grid is empty: a shallow cast still reserves a slot for every deep level other casts reach. If you load the whole grid into memory at once and only then throw away the empty cells, you have already paid for the full rectangle.

Fix: stream the file in chunks. Instead of loading everything, the converter walks the file in slices along the profile dimension (about one million data rows at a time). Each slice is turned into a small table and written straight to disk as one “row group” of the output Parquet file, then discarded before the next slice is read. The file is only ever partially in memory. On a 230 MB test file this cut peak memory from ~8.8 GB to ~0.7 GB, and lower still with smaller chunks.

The chunk size defaults to one million rows and can be tuned with the CTDDUMP_CHUNK_ROWS environment variable, smaller means less memory but more row groups. The helper scripts expose it as --chunk-rows N, and it is listed alongside the other tuning variables under Configuration. Importantly, the chunking only changes how the output is laid out on disk; the actual data and its order are identical no matter what chunk size you pick.

2. Empty rows were dropped the slow, memory-leaky way

Symptom. Running the batch converter over thousands of small files caused memory to climb steadily and never come back down, roughly 0.2 MB lost per file, with no upper limit. Converting 7,905 tiny files eventually consumed 88 GB.

Why. This is a bug in the version of the Polars library we use (0.43.1). Several of Polars’ parallel operations, filtering rows, gathering rows, and the default parallel way of writing Parquet columns, leak a small amount of memory on every call that is never released. On one file it is invisible; across thousands of files in one long-running process it adds up without bound. The leak is independent of the memory allocator and cannot be reclaimed manually.

Fix: avoid the leaky parallel paths. Two changes, both inside the converters:

1. When dropping the empty (all-missing) rows, we do it on the plain raw arrays before handing the data to Polars, so Polars’ leaky parallel filter is never called and the empty rows never enter Polars at all.
2. Every Parquet write in the converters (and in concat) turns off Polars’ parallel column encoding.

Together these keep batch memory flat regardless of how many files are processed (3,000 small files went from ~0.9 GB down to ~40 MB), and the output is verified identical to the old parallel path. If a future Polars release fixes the leak, these workarounds can be revisited.

3. Reading a big Parquet file in slices returned the wrong rows

Symptom. Commands that re-read a Parquet file in slices (filter, dropqc, dropna, markdup, dedup) silently produced wrong results on large files: for example, output that only contained platforms whose names started with the first few letters of the alphabet, as if most of the file had vanished.

Why. Another bug in Polars 0.43.1. When a Parquet file has more than one row group (which any file bigger than one chunk does), and you ask its parallel reader for “the rows from position X onwards,” it ignores X and keeps returning rows from the very first row group. So every slice read the same opening chunk over and over, and the rest of the file was never seen. It was invisible in our automated tests because test fixtures are small enough to be a single row group.

Fix: use the sequential reader for sliced reads. The five affected commands now read Parquet with Polars’ non-parallel reader, which honours the slice position correctly. Whole-file operations (like report) are unaffected and keep the faster parallel reader. A regression test now writes a deliberately many-row-group file and slices through it to guard against this coming back.

This bug was also the root cause of a confusing field report where dropqc output looked truncated: the user was running an older build from before this fix. Rebuilding to the current version resolved it.

4. markdup crashed on large files with a “record batch” error

Symptom. markdup panicked on large merged files with RecordBatch requires all its arrays to have an equal number of rows, and left a truncated, unreadable output file. It worked fine on small files.

Why. markdup reads the file in slices, adds a new true/false is_dup column to each slice, and writes it out. When a slice happened to span several row groups of the input, the columns it read back were internally split into several pieces (“chunks”), while the freshly-built is_dup column was a single piece. The Parquet writer writes one batch per internal piece and insists every column be split the same way, the mismatch made it crash. Small files have only one row group, so their columns were never split and the bug never showed.

Fix: line up the pieces before writing. After adding the is_dup column, markdup now re-aligns all columns to the same internal chunk layout before writing each slice. The sibling commands did not need this because their row-filtering step happens to realign the columns for free. A new test feeds markdup a multi-row-group input to keep this covered.

5. Multi-threaded batch runs crashed with a stack overflow

Symptom. Batch conversion processed each file on a separate worker thread and crashed on large files, even though converting the same file on its own worked.

Why. Worker threads get a much smaller memory “stack” by default (about 2 MB) than the program’s main thread (about 8 MB). Polars’ Parquet writer uses deep call chains on large files that need more stack than the 2 MB default, so it overflowed only when run on a worker thread.

Fix: give the workers a bigger stack. Batch mode builds its thread pool with a 16 MB stack per worker, and the program also raises the stack size for Polars’ own internal threads. Single-file conversion never hit this because it runs on the roomier main thread.

6. Asking for N threads actually spawned far more

Symptom. Requesting, say, 10 threads created many more than 10 busy threads, oversubscribing the machine.

Why. Polars keeps its own thread pool, sized to the number of CPU cores. So running our 10 file-workers, each of which calls into Polars, produced roughly 10 + (number of cores) threads all competing for the CPU.

Fix: pin Polars to one internal thread. Batch and concat set Polars to a single internal thread before the first Polars call, so our own --threads setting is the real, honest knob. We already parallelise across files (or across data ranges in concat), so we do not need Polars to add a second layer of threading on top.

7. A single large file stalled all the other threads

Symptom. In a multi-threaded batch run, watching a system monitor showed all cores busy at first, then most of them going idle near the end while one core finished a large file. It looked as though files were processed in fixed groups that waited for the slowest member.

Why. The work scheduler treats every file as equally expensive, it has no idea one file is much bigger than the others, and a single file cannot be split across threads. If a big file happened to be picked up late, the other workers would finish all the small files, find nothing left to do but that one big file (which they cannot help with), and sit idle until it finished.

Fix: start the biggest files first. The batch converter now sorts files largest-first before handing them out, and dispatches them one at a time so a free worker always grabs the next individual file. Starting heavy files at the very beginning lets the many small files fill in around them, so the run finishes close to “total work ÷ threads” instead of being dragged out by a late big file. The one unavoidable limit: if a single file is larger than everything else combined, its own runtime still sets the floor, no scheduler can split one file across threads. Starting it first is simply the best possible placement.

8. Merging many files without loading them all at once

Challenge. concat merges many Parquet files into one and re-numbers the profiles consistently across the whole result. Doing that naively means holding every input in memory at the same time.

Fix: merge one platform at a time. Re-numbering is grouped by platform (each platform’s profiles are numbered together), so concat processes one contiguous block of platforms at a time and writes each block out as it goes. A quick first pass reads only the tiny “platform” column of every file to plan the blocks; the second pass re-reads only the files that contribute to each block. Because a given platform’s data always lands in the same block, profiles that were split across input files are still merged correctly. Producing the blocks in order gives a result identical to merging everything at once, only the on-disk layout differs.

When more than one thread is allowed, whole platform blocks are independent units of work, so concat renumbers several blocks at once, each writing to a temporary file, and then stitches the temporaries together in order. The final file is byte-for-byte identical to the single-threaded result.

9. Missing datasets should not break the pipeline

Symptom / need. One planned dataset (the global product for the Baltic Sea) is not yet published by the data provider, but the workflow already refers to it. We did not want the whole run to fail just because those files do not exist yet.

Fix: treat “no matching files” as a no-op, not an error. When a batch or concat command is pointed at a pattern that matches nothing (or an empty/absent folder), it now prints a short informational message and writes no output, instead of stopping with an error. The helper scripts likewise skip any step whose input file is missing, with a note. The result is that the pipeline runs cleanly today and will automatically pick up the Baltic global data the moment it becomes available, with no further changes.

10. Harmless HDF5 diagnostic messages

Symptom. On some systems, reading certain files prints scary-looking HDF5-DIAG messages to the screen.

Why. These come from an older version of the underlying HDF5 library (1.10.x, used on our continuous-integration runner) encountering a newer file attribute it does not recognise. It is only a diagnostic, the data is read correctly.

What to do. Nothing. The messages are harmless and the output is unaffected. Using a newer HDF5 library avoids them.

A recurring lesson

Most of the hard bugs above share a shape: they only appear at scale. A file big enough to have multiple row groups, or a run long enough for a small leak to matter, or a workload uneven enough for scheduling to bite. Small test fixtures pass happily while the real data fails. Wherever we fixed one of these, we also added a test that deliberately recreates the “large” condition (many row groups, tiny chunks, mixed sizes) so the fix stays fixed.

Helper scripts

The scripts/ directory ships six Bash scripts that automate the full regional pipeline, the same steps documented one command at a time in the Regional workflows. They run in six phases, each consuming the previous phase’s output:

Each phase handles the Arctic, Baltic, and Mediterranean regions. The scripts only orchestrate the ctddump binary, so it must be on your PATH, except download_data.sh, which instead needs the copernicusmarine toolbox (and a free Copernicus Marine account) and does not use ctddump at all, and summary_site.sh, which needs mdBook (cargo install mdbook).

Running a script

Run the scripts from a working directory (e.g. ctddump). Source NetCDF is read from source/, data products are written under output/, and summary reports under report/; all are created as needed. They share one interface:

scripts/<script>.sh [options] [command] [region ...]

• command: the step (or all) to run; each script defaults to its most common command when omitted (see below).
• region: one or more of arctic, baltic, mediterranean. Omitting them (or passing all) runs every region.
• options: may appear anywhere on the line, in either --out DIR or --out=DIR form.

Before doing any work, a script prints the resolved configuration and asks for confirmation:

$ scripts/clean_data.sh dropqc
Configuration:
  command : dropqc
  regions : arctic baltic mediterranean
  files   : 8
  out     : output
  report  : report
  chunk   : default
  mode    : parallel (per file)
Run with -h/--help to see all options.
Proceed? [y/N]

Answer y to proceed; anything else (including a bare Enter) aborts. Pass -y/--yes to skip the prompt, required when running non-interactively (a pipe or CI job), where the script otherwise aborts with a hint rather than hang. Every script closes its configuration block with the -h/--help reminder shown above, so the full option list is always one step away. While running, each step is announced with a timestamp so you can see what is currently in progress.

Parallelism

The selected work runs in parallel by default, since the units are independent. Each worker tags its log lines with its region so the interleaved output stays readable:

[12:00:01] [arctic] dropqc nrt_ar_ar
[12:00:01] [arctic] dropqc nrt_ar_gl
[12:00:01] [baltic] dropqc nrt_bo_bo

The granularity differs by script:

• clean_data.sh and dedup_data.sh default to per file: one worker per merged file (a stem within a region), the finest granularity (8 files across the three regions). Each file runs its whole stage chain (e.g. dropqc → dropna → filter → report) in order. Pass --by-region for one worker per region instead (coarser), or --sequential for no parallelism.
• convert_data.sh defaults to per unit: one worker per (region, product), where each region’s three products are its regional NRT, the Global (GL), and CORA (9 units across the three regions). Each unit runs its own convert → merge → header → merge-header chain in order. Pass --by-region for one worker per region instead (its three products in order), or --sequential for no parallelism.
• download_data.sh parallelises per region; --sequential disables it.

If any unit fails, the others still finish and the script exits non-zero after reporting which unit failed. A run with only one unit is always sequential. Note that convert/clean/dedup each also use multiple threads within a unit, so parallel units multiply the load accordingly, convert_data.sh’s -t/--threads therefore defaults to just 2, since up to 9 units may run at once. Throttle further with --by-region / --sequential. For download_data.sh, --sequential also helps if concurrent Copernicus transfers hit rate limits.

If memory is tight (parallel units each hold a chunk in memory at once), lower the streaming chunk size with --chunk-rows N on convert/clean/dedup, it trades a smaller memory footprint for more Parquet row groups without changing the output. See Configuration for the full list of tuning variables.

Options

Timing steps

convert_data.sh, clean_data.sh, and dedup_data.sh accept --time (off by default). When on, every ctddump step is wrapped in GNU time and its wall-clock seconds and peak resident memory are logged as a timed <step>: …s, … MiB peak RSS, …% CPU line after the step completes:

[12:00:03] dropqc nrt_ar_ar
[12:00:15] timed dropqc nrt_ar_ar: 11.72s, 512 MiB peak RSS, 148% CPU

Notes:

• It requires GNU time (the time package: sudo apt-get install time), not the shell time builtin, which cannot report memory. The scripts resolve /usr/bin/time (or gtime), and check it up front so a missing tool fails fast rather than mid-run. Override the binary with the CTDDUMP_TIME_BIN environment variable.
• Peak RSS is measured per ctddump process. A multi-step stage such as the multi-box filter logs one line per underlying process (… (include), … (exclude 1), …).
• Under the default parallelism, wall-clock times of concurrent steps overlap and CPU% can exceed 100% (each ctddump is itself multi-threaded). For clean, comparable per-step wall times, pair --time with --sequential; peak RSS is per process and meaningful either way.
• The measurements share the log stream (stderr) with the normal progress, so a plain redirect captures both (... --time --sequential -y all 2> run.log). To keep them apart, pass --time-log FILE: the timed … lines go to FILE (which implies --time and is truncated fresh each run) while the screen keeps only the normal progress. In parallel mode each worker appends its own lines, so the file holds every step’s timing, ordered by completion.

Missing datasets

Every stage tolerates missing inputs, so an unavailable dataset does not fail the run. A ctddump batch or concat step that matches no files reports this and writes nothing; clean_data.sh / dedup_data.sh skip any file whose input is missing (with a note). The concrete case today is the Global (GL) product for the Baltic Sea, which Copernicus does not yet publish: the nrt_bo_gl outputs are simply skipped, and appear automatically once the data becomes available.

download_data.sh

Downloads the source NetCDF from Copernicus Marine. Commands: login, download (default). Needs the copernicusmarine toolbox (not ctddump).

scripts/download_data.sh login             # one-time Copernicus login
scripts/download_data.sh                   # download every region into source/
scripts/download_data.sh download arctic   # just the Arctic

Each product’s directory is downloaded under source/ (override with -s/--src), ready for convert_data.sh.

convert_data.sh

Converts the downloaded NetCDF to Parquet and metadata. Commands: process (default), report, all.

scripts/convert_data.sh all                # process + report, all regions
scripts/convert_data.sh process arctic     # just convert + merge the Arctic

Reads the source NetCDF from source/ (-s/--src); writes merged Parquet to output/convert/, merged headers to output/header/, and summaries to report/convert/ (Parquet data) and report/header/ (header YAML).

clean_data.sh

Cleans the merged Parquet from phase 1. Commands: dropqc, dropna, filter, report, all (default). The stages chain output/convert → clean/dropqc → clean/dropna → clean/filter, with a summary of each stage in report/clean/{dropqc,dropna,filter}/. The filter step applies each region’s bounding box(es).

scripts/clean_data.sh                       # dropqc → dropna → filter → report, all regions
scripts/clean_data.sh -y baltic             # skip the prompt, Baltic only

dedup_data.sh

Removes duplicate profiles from the cleaned data. Commands: markdup, report, dedup, all (default). It reads output/clean/filter, writes marked data to output/dedup/markdup/ (plus a duplicates TSV) and de-duplicated data to output/dedup/dedup/, with summaries under report/dedup/{markdup,dedup}/.

scripts/dedup_data.sh                        # markdup → report → dedup → report, all regions

summary_data.sh

Builds one report summary page per (region, product) unit from the TSV reports the earlier phases produced: Markdown by default, --format html for HTML. It reads reports from report/ (-r/--report) and the markdup .dups.tsv from output/ (-o/--out), and writes one page per stem to summary/ (-d/--dest). A stem with no report files (e.g. the not-yet-published Baltic GL product) is skipped, not an error.

scripts/summary_data.sh                       # summary/<stem>.md for every unit
scripts/summary_data.sh -y -f html baltic     # HTML pages, Baltic only, no prompt

The script gives each page a human-readable title and any product-specific notes (passed to report summary as --title / --note). Both live in the Page text section near the top of the script, title_for and notes_for, one case arm per stem, and that is the place to edit what a page says about a region or dataset. The section prose on the page is generic and comes from ctddump itself.

summary_site.sh

Renders the Markdown pages from summary_data.sh into a static local web site with mdBook. Pages are read from summary/ (-s/--src) and the built site is written to site/ (-d/--dest); open site/index.html in a browser, or serve the directory as-is.

Chapters are grouped into one part per region, in the same order the other scripts use. Each chapter’s name comes from the page’s own top-level heading (the title summary_data.sh gave it), so titles are defined in exactly one place. A region whose pages are absent is skipped rather than left as an empty part; if no pages are found at all, that is an error rather than an empty site.

The book is assembled in a temporary directory, so only the rendered site is left behind. This phase needs the Markdown pages, so run summary_data.sh with its default -f md (an HTML run has nothing for mdBook to render).

The built directory is made self-contained and ready to publish: mdBook writes a .nojekyll (so GitHub Pages serves its FontAwesome/ and other asset folders), and the script adds a LICENSE (the project’s own by default, -l/--license FILE to override or --license "" to skip) and a short README.md describing the site. This is what makes the output directory suitable to push straight to a publishing repository such as ctddump-report-example.

A live example of a built site is published at https://aiqc-hub.github.io/ctddump-report-example/ (a static, point-in-time sample of this phase’s output).

scripts/summary_site.sh                      # site/ from summary/, every region
scripts/summary_site.sh -y arctic            # Arctic pages only, no prompt
scripts/summary_site.sh -t "Arctic CTD QC"   # override the book title

By default the script writes a book.toml for you. Pass -c/--config FILE to use your own instead; it is used verbatim, so it must keep mdBook’s default src = "src", the script assembles the chapters into a src/ directory beside it. With --config, --title is ignored (your file sets the title).

scripts/summary_site.sh -c my-book.toml -y all

Full pipeline

Run the six phases in order (skipping prompts) for every region. Log in once first if you have not already (scripts/download_data.sh login):

scripts/download_data.sh -y            # download every region into source/
scripts/convert_data.sh  -y all
scripts/clean_data.sh    -y all
scripts/dedup_data.sh    -y all
scripts/summary_data.sh  -y all
scripts/summary_site.sh  -y all        # site/index.html

For the equivalent commands spelled out step by step, see the Regional workflows.

Arctic Sea

An end-to-end workflow for the Arctic Sea in two phases: data preparation (download, convert, merge, and export the metadata) and data cleaning (drop low-quality profiles and restrict the data to the region).

Data preparation

Prerequisites

Downloading requires a free Copernicus Marine account and the Copernicus Marine Toolbox (documentation), which provides the copernicusmarine command used below.

Run everything from a working directory (e.g. ctddump). Downloads land under source/; ctddump writes data products under output/ and summary reports under report/ (all created as needed). Create source, enter it, and log in once:

mkdir source
cd source
copernicusmarine login

1. Download the data

# NRT: Arctic (AR) and Global (GL)
copernicusmarine get -i cmems_obs-ins_arc_phybgcwav_mynrt_na_irr --dataset-part "history" --filter "*/CT/*"

# CORA: Arctic
copernicusmarine get -i cmems_obs-ins_glo_phy-temp-sal_my_cora_irr --filter "arctic/*/*_PR_CT.nc"

# Back to the working root; the steps below use source/, output/, and report/ relative to it.
cd ..

2. Convert NetCDF to Parquet

# NRT AR
ctddump batch convert nrt_ar --threads 10 --output output/convert/ar/ar source

# NRT GL
ctddump batch convert nrt_gl --threads 10 --output output/convert/ar/gl source/INSITU_ARC_PHYBGCWAV_DISCRETE_MYNRT_013_031

# CORA AR
ctddump batch convert cora --threads 10 --output output/convert/ar/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/arctic

3. Merge the Parquet files

# NRT AR
ctddump concat convert --threads 10 output/convert/ar/ar output/convert/nrt_ar_ar.parquet

# NRT GL
ctddump concat convert --threads 10 output/convert/ar/gl output/convert/nrt_ar_gl.parquet

# CORA AR
ctddump concat convert --threads 10 output/convert/ar/cora output/convert/cora_ar.parquet

4. Export the metadata (headers)

# NRT AR
ctddump batch header nrt --threads 10 --pattern "AR_PR_CT_*.nc" --output output/header/ar/ar source/INSITU_ARC_PHYBGCWAV_DISCRETE_MYNRT_013_031

# NRT GL
ctddump batch header nrt --threads 10 --pattern "GL_PR_CT_*.nc" --output output/header/ar/gl source/INSITU_ARC_PHYBGCWAV_DISCRETE_MYNRT_013_031

# CORA AR
ctddump batch header cora --threads 10 --output output/header/ar/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/arctic

5. Merge the header files

# NRT AR
ctddump concat header output/header/ar/ar output/header/nrt_ar_ar.yaml

# NRT GL
ctddump concat header output/header/ar/gl output/header/nrt_ar_gl.yaml

# CORA AR
ctddump concat header output/header/ar/cora output/header/cora_ar.yaml

6. Summarise the results

Write a platform-level summary of each merged Parquet file and a per-file summary of each merged header YAML (as TSV).

mkdir -p report/convert report/header

# NRT AR
ctddump report parquet --level platform output/convert/nrt_ar_ar.parquet report/convert/nrt_ar_ar.parquet.tsv
ctddump report yaml output/header/nrt_ar_ar.yaml report/header/nrt_ar_ar.yaml.tsv

# NRT GL
ctddump report parquet --level platform output/convert/nrt_ar_gl.parquet report/convert/nrt_ar_gl.parquet.tsv
ctddump report yaml output/header/nrt_ar_gl.yaml report/header/nrt_ar_gl.yaml.tsv

# CORA AR
ctddump report parquet --level platform output/convert/cora_ar.parquet report/convert/cora_ar.parquet.tsv
ctddump report yaml output/header/cora_ar.yaml report/header/cora_ar.yaml.tsv

Data cleaning

Clean the merged Parquet from the preparation phase by dropping low-quality profiles and restricting the data to the region. Each step reads the previous step’s output, so the stages chain dropqc → dropna → filter.

Create the output directories:

mkdir -p output/clean/dropqc output/clean/dropna output/clean/filter \
         report/clean/dropqc report/clean/dropna report/clean/filter

1. Drop profiles with bad profile-level QC

Drop profiles whose time_qc or position_qc is a present, non-OK flag; profiles that are OK ("1") or have missing QC are kept.

# NRT AR
ctddump dropqc output/convert/nrt_ar_ar.parquet output/clean/dropqc/nrt_ar_ar.parquet

# NRT GL
ctddump dropqc output/convert/nrt_ar_gl.parquet output/clean/dropqc/nrt_ar_gl.parquet

# CORA AR
ctddump dropqc output/convert/cora_ar.parquet output/clean/dropqc/cora_ar.parquet

2. Drop profiles with no usable data

Drop profiles that are entirely NA in any of temp, psal, or pres.

# NRT AR
ctddump dropna output/clean/dropqc/nrt_ar_ar.parquet output/clean/dropna/nrt_ar_ar.parquet

# NRT GL
ctddump dropna output/clean/dropqc/nrt_ar_gl.parquet output/clean/dropna/nrt_ar_gl.parquet

# CORA AR
ctddump dropna output/clean/dropqc/cora_ar.parquet output/clean/dropna/cora_ar.parquet

3. Filter to the Arctic region

Keep only profiles inside the Arctic bounding box (longitude -180 to 180, latitude 60 to 90).

# NRT AR
ctddump filter --min-lon -180 --max-lon 180 --min-lat 60 --max-lat 90 output/clean/dropna/nrt_ar_ar.parquet output/clean/filter/nrt_ar_ar.parquet

# NRT GL
ctddump filter --min-lon -180 --max-lon 180 --min-lat 60 --max-lat 90 output/clean/dropna/nrt_ar_gl.parquet output/clean/filter/nrt_ar_gl.parquet

# CORA AR
ctddump filter --min-lon -180 --max-lon 180 --min-lat 60 --max-lat 90 output/clean/dropna/cora_ar.parquet output/clean/filter/cora_ar.parquet

4. Summarise the cleaned data

Summarise each cleaning stage (as TSV), mirroring the data layout under report/clean/.

# after dropqc
ctddump report parquet --level platform output/clean/dropqc/nrt_ar_ar.parquet report/clean/dropqc/nrt_ar_ar.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/nrt_ar_gl.parquet report/clean/dropqc/nrt_ar_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/cora_ar.parquet   report/clean/dropqc/cora_ar.parquet.tsv

# after dropna
ctddump report parquet --level platform output/clean/dropna/nrt_ar_ar.parquet report/clean/dropna/nrt_ar_ar.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/nrt_ar_gl.parquet report/clean/dropna/nrt_ar_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/cora_ar.parquet   report/clean/dropna/cora_ar.parquet.tsv

# after filter
ctddump report parquet --level platform output/clean/filter/nrt_ar_ar.parquet report/clean/filter/nrt_ar_ar.parquet.tsv
ctddump report parquet --level platform output/clean/filter/nrt_ar_gl.parquet report/clean/filter/nrt_ar_gl.parquet.tsv
ctddump report parquet --level platform output/clean/filter/cora_ar.parquet   report/clean/filter/cora_ar.parquet.tsv

Data de-duplication

De-duplicate the cleaned Parquet from the previous phase. Two profiles are duplicates when they share the same date and position (longitude/latitude rounded to 3 decimals), ctddump’s defaults, across platforms. markdup flags them (and lists them in a TSV); dedup removes them, keeping the profile with the most observations.

Create the output directories:

mkdir -p output/dedup/markdup output/dedup/dedup report/dedup/markdup report/dedup/dedup

1. Mark duplicate profiles

# NRT AR
ctddump markdup output/clean/filter/nrt_ar_ar.parquet output/dedup/markdup/nrt_ar_ar.parquet output/dedup/markdup/nrt_ar_ar.dups.tsv

# NRT GL
ctddump markdup output/clean/filter/nrt_ar_gl.parquet output/dedup/markdup/nrt_ar_gl.parquet output/dedup/markdup/nrt_ar_gl.dups.tsv

# CORA AR
ctddump markdup output/clean/filter/cora_ar.parquet output/dedup/markdup/cora_ar.parquet output/dedup/markdup/cora_ar.dups.tsv

2. Summarise the marked data (duplicate counts)

# NRT AR
ctddump report parquet --level platform output/dedup/markdup/nrt_ar_ar.parquet report/dedup/markdup/nrt_ar_ar.parquet.tsv

# NRT GL
ctddump report parquet --level platform output/dedup/markdup/nrt_ar_gl.parquet report/dedup/markdup/nrt_ar_gl.parquet.tsv

# CORA AR
ctddump report parquet --level platform output/dedup/markdup/cora_ar.parquet report/dedup/markdup/cora_ar.parquet.tsv

3. Remove duplicate profiles

# NRT AR
ctddump dedup output/dedup/markdup/nrt_ar_ar.parquet output/dedup/dedup/nrt_ar_ar.parquet

# NRT GL
ctddump dedup output/dedup/markdup/nrt_ar_gl.parquet output/dedup/dedup/nrt_ar_gl.parquet

# CORA AR
ctddump dedup output/dedup/markdup/cora_ar.parquet output/dedup/dedup/cora_ar.parquet

4. Summarise the de-duplicated data

# NRT AR
ctddump report parquet --level platform output/dedup/dedup/nrt_ar_ar.parquet report/dedup/dedup/nrt_ar_ar.parquet.tsv

# NRT GL
ctddump report parquet --level platform output/dedup/dedup/nrt_ar_gl.parquet report/dedup/dedup/nrt_ar_gl.parquet.tsv

# CORA AR
ctddump report parquet --level platform output/dedup/dedup/cora_ar.parquet report/dedup/dedup/cora_ar.parquet.tsv

The pipeline is automated by scripts/download_data.sh, scripts/convert_data.sh, scripts/clean_data.sh, and scripts/dedup_data.sh. See Helper scripts for their commands and options.

Baltic Sea

An end-to-end workflow for the Baltic Sea in two phases: data preparation (download, convert, merge, and export the metadata) and data cleaning (drop low-quality profiles and restrict the data to the region).

This workflow uses the regional NRT (BO), Global (GL), and CORA products. Copernicus does not yet publish the Global (GL) data for the Baltic, so the GL steps currently match no files, ctddump reports this and writes nothing, and the cleaning / de-duplication steps skip the missing nrt_bo_gl outputs, activating automatically once GL becomes available. The manual commands below therefore cover BO and CORA; the helper scripts additionally run the (currently empty) GL steps.

Data preparation

Prerequisites

Downloading requires a free Copernicus Marine account and the Copernicus Marine Toolbox (documentation), which provides the copernicusmarine command used below.

Run everything from a working directory (e.g. ctddump). Downloads land under source/; ctddump writes data products under output/ and summary reports under report/ (all created as needed). Create source, enter it, and log in once:

mkdir source
cd source
copernicusmarine login

1. Download the data

# NRT: Baltic (BO)
copernicusmarine get -i cmems_obs-ins_bal_phybgcwav_mynrt_na_irr --dataset-part "history" --filter "*/CT/*"

# CORA: Baltic
copernicusmarine get -i cmems_obs-ins_glo_phy-temp-sal_my_cora_irr --filter "baltic/*/*_PR_CT.nc"

# Back to the working root; the steps below use source/, output/, and report/ relative to it.
cd ..

2. Convert NetCDF to Parquet

# NRT BO
ctddump batch convert nrt_bo --threads 10 --output output/convert/bo/bo source

# CORA BO
ctddump batch convert cora --threads 10 --output output/convert/bo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/baltic

3. Merge the Parquet files

# NRT BO
ctddump concat convert --threads 10 output/convert/bo/bo output/convert/nrt_bo_bo.parquet

# CORA BO
ctddump concat convert --threads 10 output/convert/bo/cora output/convert/cora_bo.parquet

4. Export the metadata (headers)

# NRT BO
ctddump batch header nrt --threads 10 --pattern "BO_PR_CT_*.nc" --output output/header/bo/bo source/INSITU_BAL_PHYBGCWAV_DISCRETE_MYNRT_013_032

# CORA BO
ctddump batch header cora --threads 10 --output output/header/bo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/baltic

5. Merge the header files

# NRT BO
ctddump concat header output/header/bo/bo output/header/nrt_bo_bo.yaml

# CORA BO
ctddump concat header output/header/bo/cora output/header/cora_bo.yaml

6. Summarise the results

Write a platform-level summary of each merged Parquet file and a per-file summary of each merged header YAML (as TSV).

mkdir -p report/convert report/header

# NRT BO
ctddump report parquet --level platform output/convert/nrt_bo_bo.parquet report/convert/nrt_bo_bo.parquet.tsv
ctddump report yaml output/header/nrt_bo_bo.yaml report/header/nrt_bo_bo.yaml.tsv

# CORA BO
ctddump report parquet --level platform output/convert/cora_bo.parquet report/convert/cora_bo.parquet.tsv
ctddump report yaml output/header/cora_bo.yaml report/header/cora_bo.yaml.tsv

Data cleaning

Clean the merged Parquet from the preparation phase by dropping low-quality profiles and restricting the data to the region. Each step reads the previous step’s output, so the stages chain dropqc → dropna → filter.

Create the output directories:

mkdir -p output/clean/dropqc output/clean/dropna output/clean/filter \
         report/clean/dropqc report/clean/dropna report/clean/filter

1. Drop profiles with bad profile-level QC

Drop profiles whose time_qc or position_qc is a present, non-OK flag; profiles that are OK ("1") or have missing QC are kept.

# NRT BO
ctddump dropqc output/convert/nrt_bo_bo.parquet output/clean/dropqc/nrt_bo_bo.parquet

# CORA BO
ctddump dropqc output/convert/cora_bo.parquet output/clean/dropqc/cora_bo.parquet

2. Drop profiles with no usable data

Drop profiles that are entirely NA in any of temp, psal, or pres.

# NRT BO
ctddump dropna output/clean/dropqc/nrt_bo_bo.parquet output/clean/dropna/nrt_bo_bo.parquet

# CORA BO
ctddump dropna output/clean/dropqc/cora_bo.parquet output/clean/dropna/cora_bo.parquet

3. Filter to the Baltic region

Keep profiles inside the Baltic bounding box (longitude 6 to 30, latitude 53 to 66), then exclude the sub-box (longitude 6 to 15, latitude 60 to 66). The include step writes an intermediate .box.parquet file that the exclude step consumes to produce the final cleaned file.

# NRT BO
ctddump filter --min-lon 6 --max-lon 30 --min-lat 53 --max-lat 66 output/clean/dropna/nrt_bo_bo.parquet output/clean/filter/nrt_bo_bo.box.parquet
ctddump filter --mode exclude --min-lon 6 --max-lon 15 --min-lat 60 --max-lat 66 output/clean/filter/nrt_bo_bo.box.parquet output/clean/filter/nrt_bo_bo.parquet

# CORA BO
ctddump filter --min-lon 6 --max-lon 30 --min-lat 53 --max-lat 66 output/clean/dropna/cora_bo.parquet output/clean/filter/cora_bo.box.parquet
ctddump filter --mode exclude --min-lon 6 --max-lon 15 --min-lat 60 --max-lat 66 output/clean/filter/cora_bo.box.parquet output/clean/filter/cora_bo.parquet

4. Summarise the cleaned data

Summarise each cleaning stage (as TSV), mirroring the data layout under report/clean/.

# after dropqc
ctddump report parquet --level platform output/clean/dropqc/nrt_bo_bo.parquet report/clean/dropqc/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/cora_bo.parquet   report/clean/dropqc/cora_bo.parquet.tsv

# after dropna
ctddump report parquet --level platform output/clean/dropna/nrt_bo_bo.parquet report/clean/dropna/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/cora_bo.parquet   report/clean/dropna/cora_bo.parquet.tsv

# after filter
ctddump report parquet --level platform output/clean/filter/nrt_bo_bo.parquet report/clean/filter/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/filter/cora_bo.parquet   report/clean/filter/cora_bo.parquet.tsv

Data de-duplication

De-duplicate the cleaned Parquet from the previous phase. Two profiles are duplicates when they share the same date and position (longitude/latitude rounded to 3 decimals), ctddump’s defaults, across platforms. markdup flags them (and lists them in a TSV); dedup removes them, keeping the profile with the most observations.

Create the output directories:

mkdir -p output/dedup/markdup output/dedup/dedup report/dedup/markdup report/dedup/dedup

1. Mark duplicate profiles

# NRT BO
ctddump markdup output/clean/filter/nrt_bo_bo.parquet output/dedup/markdup/nrt_bo_bo.parquet output/dedup/markdup/nrt_bo_bo.dups.tsv

# CORA BO
ctddump markdup output/clean/filter/cora_bo.parquet output/dedup/markdup/cora_bo.parquet output/dedup/markdup/cora_bo.dups.tsv

2. Summarise the marked data (duplicate counts)

# NRT BO
ctddump report parquet --level platform output/dedup/markdup/nrt_bo_bo.parquet report/dedup/markdup/nrt_bo_bo.parquet.tsv

# CORA BO
ctddump report parquet --level platform output/dedup/markdup/cora_bo.parquet report/dedup/markdup/cora_bo.parquet.tsv

3. Remove duplicate profiles

# NRT BO
ctddump dedup output/dedup/markdup/nrt_bo_bo.parquet output/dedup/dedup/nrt_bo_bo.parquet

# CORA BO
ctddump dedup output/dedup/markdup/cora_bo.parquet output/dedup/dedup/cora_bo.parquet

4. Summarise the de-duplicated data

# NRT BO
ctddump report parquet --level platform output/dedup/dedup/nrt_bo_bo.parquet report/dedup/dedup/nrt_bo_bo.parquet.tsv

# CORA BO
ctddump report parquet --level platform output/dedup/dedup/cora_bo.parquet report/dedup/dedup/cora_bo.parquet.tsv

The pipeline is automated by scripts/download_data.sh, scripts/convert_data.sh, scripts/clean_data.sh, and scripts/dedup_data.sh. See Helper scripts for their commands and options.

Mediterranean Sea

An end-to-end workflow for the Mediterranean Sea in two phases: data preparation (download, convert, merge, and export the metadata) and data cleaning (drop low-quality profiles and restrict the data to the region).

Data preparation

Prerequisites

Downloading requires a free Copernicus Marine account and the Copernicus Marine Toolbox (documentation), which provides the copernicusmarine command used below.

Run everything from a working directory (e.g. ctddump). Downloads land under source/; ctddump writes data products under output/ and summary reports under report/ (all created as needed). Create source, enter it, and log in once:

mkdir source
cd source
copernicusmarine login

1. Download the data

# NRT: Mediterranean (MO) and Global (GL)
copernicusmarine get -i cmems_obs-ins_med_phybgcwav_mynrt_na_irr --dataset-part "history" --filter "*/CT/*"

# CORA: Mediterranean
copernicusmarine get -i cmems_obs-ins_glo_phy-temp-sal_my_cora_irr --filter "mediterrane/*/*_PR_CT.nc"

# Back to the working root; the steps below use source/, output/, and report/ relative to it.
cd ..

2. Convert NetCDF to Parquet

# NRT MO
ctddump batch convert nrt_mo --threads 10 --output output/convert/mo/mo source

# NRT GL
ctddump batch convert nrt_gl --threads 10 --output output/convert/mo/gl source/INSITU_MED_PHYBGCWAV_DISCRETE_MYNRT_013_035

# CORA MO
ctddump batch convert cora --threads 10 --output output/convert/mo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/mediterrane

3. Merge the Parquet files

# NRT MO
ctddump concat convert --threads 10 output/convert/mo/mo output/convert/nrt_mo_mo.parquet

# NRT GL
ctddump concat convert --threads 10 output/convert/mo/gl output/convert/nrt_mo_gl.parquet

# CORA MO
ctddump concat convert --threads 10 output/convert/mo/cora output/convert/cora_mo.parquet

4. Export the metadata (headers)

# NRT MO
ctddump batch header nrt --threads 10 --pattern "MO_PR_CT_*.nc" --output output/header/mo/mo source/INSITU_MED_PHYBGCWAV_DISCRETE_MYNRT_013_035

# NRT GL
ctddump batch header nrt --threads 10 --pattern "GL_PR_CT_*.nc" --output output/header/mo/gl source/INSITU_MED_PHYBGCWAV_DISCRETE_MYNRT_013_035

# CORA MO
ctddump batch header cora --threads 10 --output output/header/mo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/mediterrane

5. Merge the header files

# NRT MO
ctddump concat header output/header/mo/mo output/header/nrt_mo_mo.yaml

# NRT GL
ctddump concat header output/header/mo/gl output/header/nrt_mo_gl.yaml

# CORA MO
ctddump concat header output/header/mo/cora output/header/cora_mo.yaml

6. Summarise the results

Write a platform-level summary of each merged Parquet file and a per-file summary of each merged header YAML (as TSV).

mkdir -p report/convert report/header

# NRT MO
ctddump report parquet --level platform output/convert/nrt_mo_mo.parquet report/convert/nrt_mo_mo.parquet.tsv
ctddump report yaml output/header/nrt_mo_mo.yaml report/header/nrt_mo_mo.yaml.tsv

# NRT GL
ctddump report parquet --level platform output/convert/nrt_mo_gl.parquet report/convert/nrt_mo_gl.parquet.tsv
ctddump report yaml output/header/nrt_mo_gl.yaml report/header/nrt_mo_gl.yaml.tsv

# CORA MO
ctddump report parquet --level platform output/convert/cora_mo.parquet report/convert/cora_mo.parquet.tsv
ctddump report yaml output/header/cora_mo.yaml report/header/cora_mo.yaml.tsv

Data cleaning

Clean the merged Parquet from the preparation phase by dropping low-quality profiles and restricting the data to the region. Each step reads the previous step’s output, so the stages chain dropqc → dropna → filter.

Create the output directories:

mkdir -p output/clean/dropqc output/clean/dropna output/clean/filter \
         report/clean/dropqc report/clean/dropna report/clean/filter

1. Drop profiles with bad profile-level QC

Drop profiles whose time_qc or position_qc is a present, non-OK flag; profiles that are OK ("1") or have missing QC are kept.

# NRT MO
ctddump dropqc output/convert/nrt_mo_mo.parquet output/clean/dropqc/nrt_mo_mo.parquet

# NRT GL
ctddump dropqc output/convert/nrt_mo_gl.parquet output/clean/dropqc/nrt_mo_gl.parquet

# CORA MO
ctddump dropqc output/convert/cora_mo.parquet output/clean/dropqc/cora_mo.parquet

2. Drop profiles with no usable data

Drop profiles that are entirely NA in any of temp, psal, or pres.

# NRT MO
ctddump dropna output/clean/dropqc/nrt_mo_mo.parquet output/clean/dropna/nrt_mo_mo.parquet

# NRT GL
ctddump dropna output/clean/dropqc/nrt_mo_gl.parquet output/clean/dropna/nrt_mo_gl.parquet

# CORA MO
ctddump dropna output/clean/dropqc/cora_mo.parquet output/clean/dropna/cora_mo.parquet

3. Filter to the Mediterranean region

Keep profiles inside the Mediterranean bounding box (longitude -5.61 to 35.567, latitude 28.378 to 45.755), then exclude two sub-boxes: (longitude 27 to 36, latitude 41 to 46) and (longitude -5.61 to 0, latitude 42 to 46). Each stage chains through an intermediate .box*.parquet file to produce the final cleaned file.

# NRT MO
ctddump filter --min-lon -5.61 --max-lon 35.567 --min-lat 28.378 --max-lat 45.755 output/clean/dropna/nrt_mo_mo.parquet output/clean/filter/nrt_mo_mo.box1.parquet
ctddump filter --mode exclude --min-lon 27 --max-lon 36 --min-lat 41 --max-lat 46 output/clean/filter/nrt_mo_mo.box1.parquet output/clean/filter/nrt_mo_mo.box2.parquet
ctddump filter --mode exclude --min-lon -5.61 --max-lon 0 --min-lat 42 --max-lat 46 output/clean/filter/nrt_mo_mo.box2.parquet output/clean/filter/nrt_mo_mo.parquet

# NRT GL
ctddump filter --min-lon -5.61 --max-lon 35.567 --min-lat 28.378 --max-lat 45.755 output/clean/dropna/nrt_mo_gl.parquet output/clean/filter/nrt_mo_gl.box1.parquet
ctddump filter --mode exclude --min-lon 27 --max-lon 36 --min-lat 41 --max-lat 46 output/clean/filter/nrt_mo_gl.box1.parquet output/clean/filter/nrt_mo_gl.box2.parquet
ctddump filter --mode exclude --min-lon -5.61 --max-lon 0 --min-lat 42 --max-lat 46 output/clean/filter/nrt_mo_gl.box2.parquet output/clean/filter/nrt_mo_gl.parquet

# CORA MO
ctddump filter --min-lon -5.61 --max-lon 35.567 --min-lat 28.378 --max-lat 45.755 output/clean/dropna/cora_mo.parquet output/clean/filter/cora_mo.box1.parquet
ctddump filter --mode exclude --min-lon 27 --max-lon 36 --min-lat 41 --max-lat 46 output/clean/filter/cora_mo.box1.parquet output/clean/filter/cora_mo.box2.parquet
ctddump filter --mode exclude --min-lon -5.61 --max-lon 0 --min-lat 42 --max-lat 46 output/clean/filter/cora_mo.box2.parquet output/clean/filter/cora_mo.parquet

4. Summarise the cleaned data

Summarise each cleaning stage (as TSV), mirroring the data layout under report/clean/.

# after dropqc
ctddump report parquet --level platform output/clean/dropqc/nrt_mo_mo.parquet report/clean/dropqc/nrt_mo_mo.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/nrt_mo_gl.parquet report/clean/dropqc/nrt_mo_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/cora_mo.parquet   report/clean/dropqc/cora_mo.parquet.tsv

# after dropna
ctddump report parquet --level platform output/clean/dropna/nrt_mo_mo.parquet report/clean/dropna/nrt_mo_mo.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/nrt_mo_gl.parquet report/clean/dropna/nrt_mo_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/cora_mo.parquet   report/clean/dropna/cora_mo.parquet.tsv

# after filter
ctddump report parquet --level platform output/clean/filter/nrt_mo_mo.parquet report/clean/filter/nrt_mo_mo.parquet.tsv
ctddump report parquet --level platform output/clean/filter/nrt_mo_gl.parquet report/clean/filter/nrt_mo_gl.parquet.tsv
ctddump report parquet --level platform output/clean/filter/cora_mo.parquet   report/clean/filter/cora_mo.parquet.tsv

Data de-duplication

De-duplicate the cleaned Parquet from the previous phase. Two profiles are duplicates when they share the same date and position (longitude/latitude rounded to 3 decimals), ctddump’s defaults, across platforms. markdup flags them (and lists them in a TSV); dedup removes them, keeping the profile with the most observations.

Create the output directories:

mkdir -p output/dedup/markdup output/dedup/dedup report/dedup/markdup report/dedup/dedup

1. Mark duplicate profiles

# NRT MO
ctddump markdup output/clean/filter/nrt_mo_mo.parquet output/dedup/markdup/nrt_mo_mo.parquet output/dedup/markdup/nrt_mo_mo.dups.tsv

# NRT GL
ctddump markdup output/clean/filter/nrt_mo_gl.parquet output/dedup/markdup/nrt_mo_gl.parquet output/dedup/markdup/nrt_mo_gl.dups.tsv

# CORA MO
ctddump markdup output/clean/filter/cora_mo.parquet output/dedup/markdup/cora_mo.parquet output/dedup/markdup/cora_mo.dups.tsv

2. Summarise the marked data (duplicate counts)

# NRT MO
ctddump report parquet --level platform output/dedup/markdup/nrt_mo_mo.parquet report/dedup/markdup/nrt_mo_mo.parquet.tsv

# NRT GL
ctddump report parquet --level platform output/dedup/markdup/nrt_mo_gl.parquet report/dedup/markdup/nrt_mo_gl.parquet.tsv

# CORA MO
ctddump report parquet --level platform output/dedup/markdup/cora_mo.parquet report/dedup/markdup/cora_mo.parquet.tsv

3. Remove duplicate profiles

# NRT MO
ctddump dedup output/dedup/markdup/nrt_mo_mo.parquet output/dedup/dedup/nrt_mo_mo.parquet

# NRT GL
ctddump dedup output/dedup/markdup/nrt_mo_gl.parquet output/dedup/dedup/nrt_mo_gl.parquet

# CORA MO
ctddump dedup output/dedup/markdup/cora_mo.parquet output/dedup/dedup/cora_mo.parquet

4. Summarise the de-duplicated data

# NRT MO
ctddump report parquet --level platform output/dedup/dedup/nrt_mo_mo.parquet report/dedup/dedup/nrt_mo_mo.parquet.tsv

# NRT GL
ctddump report parquet --level platform output/dedup/dedup/nrt_mo_gl.parquet report/dedup/dedup/nrt_mo_gl.parquet.tsv

# CORA MO
ctddump report parquet --level platform output/dedup/dedup/cora_mo.parquet report/dedup/dedup/cora_mo.parquet.tsv

The pipeline is automated by scripts/download_data.sh, scripts/convert_data.sh, scripts/clean_data.sh, and scripts/dedup_data.sh. See Helper scripts for their commands and options.