ctddump
ctddump is a small, fast command-line tool for converting oceanographic CTD (Conductivity, Temperature, Depth) data from NetCDF into analysis-ready formats:
- Parquet: the observation data, as a uniform flat table.
- YAML: the file metadata (dimensions, variables, global attributes).
It is written in Rust, streams data in bounded memory, and processes whole directory trees in parallel.
Data sources
Two families of source files are supported:
| Source | Description |
|---|---|
| NRT | Near Real Time: Arctic Sea (nrt_ar), Baltic Sea (nrt_bo), Mediterranean Sea (nrt_mo), Global (nrt_gl) |
| CORA | Copernicus Ocean Reanalysis: current format (cora), legacy format (cora_legacy) |
| Your own | Any NetCDF laid out like the Copernicus files, described by a mapping file (generic); see Your own NetCDF files |
What you can do
| Command | Purpose |
|---|---|
convert | Convert a single NetCDF file to Parquet. |
batch | Convert a whole directory tree to Parquet or YAML, in parallel. |
header | Extract a single file’s metadata to YAML. |
checkpres | Find profiles whose pressure does not increase in observation order. |
concat | Merge many Parquet (or YAML) files into one. |
report | Summarise a Parquet or YAML file as a text report. |
filter | Filter a Parquet file’s profiles by a geographic bounding box. |
dropna | Drop profiles that are entirely NA in any of temp/psal/pres. |
dropqc | Drop profiles flagged bad in time_qc or position_qc. |
markdup | Mark duplicate profiles (by date + position) with an is_dup column. |
dedup | Remove duplicate profiles, keeping the one with the most observations. |
compare | Report how far each of two Parquet files covers the other’s platforms and profiles. |
sqlite | Export a Parquet file to a SQLite database (platform, profile, observation tables). |
completions | Print a shell tab-completion script for bash, zsh, fish, elvish, or powershell. |
Quick example
# One file to Parquet
ctddump convert nrt_ar input.nc output.parquet
# A whole Arctic directory, then merge into a single Parquet file
ctddump batch convert nrt_ar --output ./parquet ./netcdf
ctddump concat convert ./parquet arctic.parquet
To see what the full pipeline produces, browse the live example report site, built with the helper scripts.
New here? Start with Installation, skim the Commands, then follow an end-to-end regional workflow.
Installation
There are three ways to install ctddump. Pick the first one that applies:
| Route | Needs | Best for |
|---|---|---|
| Prebuilt binary | nothing | most users |
| crates.io | Rust, HDF5 headers | Rust users, other platforms |
| From source | Rust, HDF5 headers, git | contributors |
Prebuilt binary
Each release ships an archive per
platform. It contains the ctddump executable with HDF5 and netCDF built in, so
no Rust toolchain and no system libraries are required:
| Platform | Archive |
|---|---|
| Linux, Intel/AMD 64-bit | ctddump-vX.Y.Z-x86_64-unknown-linux-gnu.tar.gz |
| Linux, ARM 64-bit | ctddump-vX.Y.Z-aarch64-unknown-linux-gnu.tar.gz |
| macOS, Apple Silicon | ctddump-vX.Y.Z-aarch64-apple-darwin.tar.gz |
| macOS, Intel | ctddump-vX.Y.Z-x86_64-apple-darwin.tar.gz |
The Linux binaries are linked against glibc 2.17, so they run on any distribution with glibc 2.17 or newer: RHEL/CentOS 7, Rocky Linux 8, Ubuntu 18.04, Debian 9, and everything more recent.
Download, extract, and put the binary somewhere on your PATH:
VERSION=v0.37.2
TARGET=x86_64-unknown-linux-gnu
curl -LO "https://github.com/AIQC-Hub/ctddump/releases/download/$VERSION/ctddump-$VERSION-$TARGET.tar.gz"
tar -xzf "ctddump-$VERSION-$TARGET.tar.gz"
cd "ctddump-$VERSION-$TARGET"
./ctddump --version
To install it for your user, move it onto your PATH:
mkdir -p ~/.local/bin
mv ctddump ~/.local/bin/
If ctddump is still not found afterwards, ~/.local/bin is not on your PATH;
add export PATH="$HOME/.local/bin:$PATH" to your shell profile.
Every archive is listed in SHA256SUMS on the same release page. To verify a
download before trusting it:
curl -LO "https://github.com/AIQC-Hub/ctddump/releases/download/$VERSION/SHA256SUMS"
sha256sum -c SHA256SUMS --ignore-missing
What else is in the archive
Alongside the binary are the helper scripts, plus README.md,
LICENSE, and CHANGELOG.md. Five scripts (convert_data.sh, clean_data.sh,
dedup_data.sh, compare_data.sh, and summary_data.sh) need only ctddump on
your PATH, so they work as soon as the step above is done. Three others call
external tools that are not bundled and must be installed separately:
download_data.sh needs copernicusmarine, summary_site.sh needs mdbook,
and fetch_test_data.sh needs gh and unzip.
The scripts are bash, so on Windows they need WSL or Git Bash.
System dependencies
The two routes below compile ctddump themselves and link the HDF5 C library, so the development headers must be installed first. (The prebuilt binaries above need none of this.)
# Ubuntu / Debian
sudo apt-get install libhdf5-dev libnetcdf-dev
# macOS (Homebrew)
brew install hdf5
Install from crates.io
With the system dependencies in place, install the published crate:
cargo install ctddump
This builds the binary and puts it in ~/.cargo/bin, which is normally already
on your PATH. Check it works:
ctddump --help
If the build stops with “A system version of libnetcdf could not be found”, the development headers are missing or are somewhere the build script does not look. Install them as shown above, or point at them explicitly:
NETCDF_DIR=/path/to/netcdf HDF5_DIR=/path/to/hdf5 cargo install ctddump
Failing that, the static-netcdf feature builds HDF5 and netCDF from source
instead of looking for system ones. It is much slower but needs no headers:
cargo install ctddump --features static-netcdf
Build from source
To work on ctddump itself, or to run a version that is not published yet:
git clone https://github.com/AIQC-Hub/ctddump.git
cd ctddump
cargo build --release
The binary is placed at target/release/ctddump. Copy it somewhere on your
PATH (or run it directly) and check it works:
ctddump --help
Every command and subcommand supports -h / --help, so you can always
discover the available options interactively:
ctddump convert --help
ctddump batch convert nrt_ar --help
Note: On systems with HDF5 ≤ 1.10 you may see harmless
HDF5-DIAGmessages in the output. The data is read correctly and results are unaffected.
convert
Convert a single NetCDF file into a Parquet file.
ctddump convert <subcommand> [OPTIONS] <src_file> <target_file>
The subcommand selects the source format and its defaults:
| Subcommand | Source |
|---|---|
nrt_ar | NRT: Arctic Sea |
nrt_bo | NRT: Baltic Sea |
nrt_mo | NRT: Mediterranean Sea |
nrt_gl | NRT: Global |
cora | CORA: current format |
cora_legacy | CORA: legacy format |
generic | Your own NetCDF, described by a mapping file (details) |
Examples
ctddump convert nrt_ar input.nc output.parquet
ctddump convert nrt_bo input.nc output.parquet
ctddump convert cora input.nc output.parquet
ctddump convert cora_legacy input.nc output.parquet
ctddump convert generic --config my_map.toml input.nc output.parquet
Use a saved TOML preset, or override individual fields with flags:
# Apply a config preset
ctddump convert nrt_ar --config my_preset.toml input.nc output.parquet
# Override a single field
ctddump convert nrt_bo --no-deph-source input.nc output.parquet
See Configuration for the available flags and the TOML format, and Output schema for the columns produced.
batch
Process a whole directory tree in parallel. batch recursively finds every
matching .nc file and converts each one, using all CPU cores by default.
ctddump batch convert <subcommand> [OPTIONS] <src_dir>
ctddump batch header <subcommand> [OPTIONS] <src_dir>
batch convertwrites one Parquet file per source file.batch headerwrites one YAML metadata file per source file.
Each output keeps its source’s filename stem with a new extension. If
--output is omitted, each result is written alongside its source; with
--output <dir> all results land flat in that directory.
Examples
# Convert all NRT Arctic files to Parquet, flat into ./output
ctddump batch convert nrt_ar --output ./output ./data/arctic
# Limit to 4 threads
ctddump batch convert nrt_ar --threads 4 --output ./output ./data/arctic
# Override the filename pattern
ctddump batch convert nrt_ar --pattern "AR_PR_CT_ITP-*.nc" --output ./output ./data
# Extract YAML metadata for all NRT files
ctddump batch header nrt --output ./output ./data/arctic
# Convert your own files with a mapping file
ctddump batch convert generic --config my_map.toml --output ./output ./data/mine
Default filename patterns
--pattern matches the filename only (not the path) and supports *, ?
and […]. When omitted, each subcommand uses a sensible default:
| Subcommand | Pattern |
|---|---|
nrt_ar | AR_PR_CT_*.nc |
nrt_bo | BO_PR_CT_*.nc |
nrt_mo | MO_PR_CT_*.nc |
nrt_gl | GL_PR_CT_*.nc |
cora, cora_legacy, generic | *.nc |
batch header nrt, batch header cora, batch header generic | *.nc |
Threads.
--threads Ncaps the worker count; omit it to use all logical cores. Memory scales with the thread count, so lower it if you are processing very large files on a memory-constrained machine.
header
Extract the metadata of a single NetCDF file to a YAML file, dimensions, variables, and global attributes. No observation data is read.
ctddump header <subcommand> <src_file> <target_file>
| Subcommand | Source |
|---|---|
nrt | NRT files |
cora | CORA files |
generic | Any NetCDF file; keeps all global attributes |
Examples
ctddump header nrt input.nc output.yaml
ctddump header cora input.nc output.yaml
ctddump header generic input.nc output.yaml
To extract metadata for a whole directory at once, use
batch header; to combine many YAML files into one, use
concat header.
checkpres
Find profiles whose pressure does not increase in observation order, and write three TSV reports: the profiles, their platforms, and totals.
ctddump checkpres [OPTIONS] <src> <out_prefix>
<src> is a directory of Parquet files, read the same way as by
concat convert, or a single Parquet file.
When to run it
Run it on the per-file Parquet written by batch convert,
before concat convert merges them. The merge sorts every
profile by increasing pressure (see
Row order and numbering), so
afterwards the original order can no longer be checked. Before the merge, rows
are still in the source file’s order and observation_no is the level index in
the file.
Run it once per merge, on the same directory, so the reports cover exactly what ends up in one merged file:
ctddump checkpres output/convert/ar/ar report/convert/checkpres/nrt_ar_ar
ctddump concat convert output/convert/ar/ar output/convert/nrt_ar_ar.parquet
What is checked
- A profile is one
(filename, platform_code, profile_no). A file may hold many profiles and many platforms; each profile is checked on its own. - Within a profile, each observation that has a pressure is compared with the previous observation that had one. Missing pressures are skipped.
- A decrease is a step to a lower pressure. A repeated pressure is not a decrease.
- A profile with no usable pressure at all has nothing to check. It is left out
of the counts and reported only as
n_no_pres, so the counts match the profilesconcat convertkeeps (it drops observations without pressure). - A profile with at least one decrease is reported, as one of two kinds:
| Kind | Meaning |
|---|---|
reversed | Pressure never goes up: the profile is stored from the bottom up. Sorting by pressure turns it the right way round. |
non_monotonic | Pressure goes both up and down: loops, spikes, or levels stored out of order. These are the profiles to look at. |
Options
| Option | Default | Meaning |
|---|---|---|
--pattern <GLOB> | *.parquet | Filename glob used when <src> is a directory |
--min-drop <DBAR> | 0 | Ignore decreases no larger than this, for sensor noise |
--pres-qc <FLAGS> | every pressure | Use only pressures whose pres_qc is one of these flags, e.g. 1,2 |
-t, --threads <N> | all cores | Number of files checked in parallel; the reports are the same for any value |
Outputs
Three tab-separated files, named after <out_prefix>:
-
<out_prefix>.profiles.tsv: one row per reported profile, sorted by platform and time.Column Meaning filename,platform_code,profile_nothe profile, as in the per-file Parquet profile_timestamp,profile_time,longitude,latitudewhen and where; with platform_code, these also find the profile after the mergekindreversedornon_monotonicn_obs,n_presobservations, and observations with a pressure used by the check n_decreasesnumber of decreasing steps first_decrease_obsobservation_nowhere the pressure first dropsmax_decreaselargest drop, in dbar pres_first,pres_last,pres_min,pres_maxpressure range, in dbar pres_conv1when some pressure in the profile was derived from depth -
<out_prefix>.platforms.tsv: one row per platform with at least one reported profile:platform_code, n_files, n_profiles, n_bad_profiles, pct_bad, n_reversed, n_non_monotonic, first_date, last_date. The dates are those of its first and last reported profile. -
<out_prefix>.summary.tsv: one row for the whole source:n_files, n_platforms, n_profiles, n_no_pres, n_bad_profiles, pct_bad, n_reversed, n_non_monotonic, n_bad_platforms, n_bad_files.n_profilesandn_platformscount only what was checked;n_no_presis the number of profiles without any usable pressure.
The output directory is created if needed. A one-line summary is printed to
standard error. Finding problems is not an error: the exit status is 0. When
no file matches, or the directory does not exist, the command says so and
writes nothing.
Examples
# Check a directory of converted files
ctddump checkpres ./parquet reports/arctic
# Ignore drops of 0.1 dbar or less, and pressures not flagged good
ctddump checkpres --min-drop 0.1 --pres-qc 1,2 ./parquet reports/arctic
# Check one file
ctddump checkpres AR_PR_CT_58KN.parquet reports/58kn
Files read one at a time, in fixed row windows (CTDDUMP_CHUNK_ROWS), so memory
follows the size of one file plus the reported profiles.
concat
Merge many files from a directory tree into a single file.
ctddump concat convert [OPTIONS] <src_dir> <output_file>
ctddump concat header [OPTIONS] <src_dir> <output_file>
concat convertmerges Parquet data files.concat headermerges YAML metadata files.
concat convert
By default the merge re-assigns profile_no and observation_no so that
profile numbers are unique and sequential within each platform, and
sorts each profile’s observations by increasing pressure. Its behaviour
is controlled by a few flags:
| Flag | Effect |
|---|---|
| (default) | Renumber; sort by platform_code, profile_timestamp, longitude, latitude, pres; drop rows with missing pres; use all cores. |
--no-renumber | Merge as-is, without re-assigning profile_no / observation_no. |
--no-pres-sort | Sort without pres, keeping each profile’s observations in their original source order instead of reordering by pressure. |
--keep-na-pres | Keep rows whose pres is null/NaN (dropped by default). |
--threads N | Cap the worker count (default: all cores). --threads 1 is the sequential, lowest-memory path. |
--pattern <GLOB> | Merge only files whose name matches the pattern. |
Renumbering processes platform ranges in parallel via temporary files in the output folder. Higher thread counts are faster but use more memory, the merged result is identical either way.
Sorting by pressure
The files written by convert keep the order of the source NetCDF. The default
merge reorders them: within each profile, observations run from the lowest to
the highest pres, and observation_no is renumbered 1..N in that order.
Every other column moves with its row.
- Rows with missing
presare dropped before sorting. With--keep-na-presthey are kept and placed at the end of their profile. - The sort is stable: observations with the same pressure keep their source order (by file path, then by position in the file).
- Profiles are identified by
(platform_code, profile_timestamp, longitude, latitude). Two profiles from different files with the same four values become oneprofile_no, and their observations are interleaved by pressure (filenamestill tells them apart). --no-pres-sortkeeps each profile’s observations in their source order;--no-renumberdoes not sort at all.
See Row order and numbering for
the order each step produces. To list the profiles whose source order is not
increasing in pressure, run checkpres on the same directory
before merging.
Examples
# Merge all Parquet files, with profile renumbering (the default)
ctddump concat convert ./parquet merged.parquet
# Merge without renumbering
ctddump concat convert --no-renumber ./parquet merged.parquet
# Keep each profile's observations in their original order
ctddump concat convert --no-pres-sort ./parquet merged.parquet
# Keep rows with missing pres
ctddump concat convert --keep-na-pres ./parquet merged.parquet
# Sequential, lowest-memory merge
ctddump concat convert --threads 1 ./parquet merged.parquet
# Merge only a subset
ctddump concat convert --pattern "AR_PR_CT_*.parquet" ./parquet merged.parquet
concat header
Merges YAML header files, each file contributes its top-level keys to the combined output. An error is raised if any two files share the same key.
ctddump concat header ./yaml merged.yaml
report
Summarise a produced Parquet data file or YAML header file as a
text report, TSV (default), plain text, or JSON, or assemble a multi-section
summary page (report summary) from the per-stage TSV reports.
ctddump report parquet [--level global|platform|profile] [--format tsv|text|json] <src.parquet> [dest]
ctddump report yaml [--format tsv|text|json] <src.yaml> [dest]
ctddump report summary [--report-dir report] [--out-dir output] [--format md|html]
[--title TEXT] [--note TEXT]... [-o dest] <stem>
The report is written to dest, or to stdout when dest is omitted (so it
pipes cleanly into other tools).
report parquet
Aggregates a data file at one of three levels (--level, default platform):
| Level | One row per | Identifier columns |
|---|---|---|
global | the whole file | n_platforms, n_profiles |
platform | platform_code | platform_code, n_profiles |
profile | (platform_code, profile_no) | platform_code, profile_no, profile_timestamp, longitude, latitude |
Every row also reports:
n_obs: number of observations.time_qc_good/position_qc_good: number of profiles whose flag is"1"(CMEMS “good”). These flags are per-profile; at--level profilethe single flag value is shown instead.na_temp/na_psal/na_pres: count of missing (null/NaN) values.{temp,psal,pres}_{min,max,mean}: statistics over the valid (non-NaN) values. (Median is intentionally not computed.){longitude,latitude}_{min,max}: geographic bounding box over the valid positions (globalandplatformlevels only; atprofilelevel the singlelongitude/latitudeis already shown).{temp,psal,pres}_dm_{r,a,d}: number of observations whose data mode is real-time (R), adjusted in real time (A), or delayed mode (D), placed after that parameter’s statistics, followed by{temp,psal,pres}_dm_unknown, the observations with no data mode (""). Each parameter’s counts appear only when the file has its<param>_dmcolumn, so a file converted before the data-mode columns existed reports as before.dup_profiles(global/platform) oris_dup(profile): only when the file has amarkdupis_dupcolumn.
# Per-platform summary
ctddump report parquet --level platform merged.parquet report.tsv
# Whole-file summary, human-readable, to stdout
ctddump report parquet --level global --format text merged.parquet
report yaml
Summarises a merged header YAML (as produced by concat header):
one row per source file.
| Column | Meaning |
|---|---|
filename | source file stem |
has_temp, has_psal, has_pres, has_deph | presence of each core measurement variable |
has_time_qc, has_position_qc | presence of each profile-level QC flag (TIME_QC / POSITION_QC) |
extra_params | ;-joined list of the extra measurement parameters present |
extra_params is detected automatically: any Float variable dimensioned
(TIME, DEPTH) that is not a _QC flag and not a core physical
(TEMP/PSAL/PRES/DEPH). This captures biogeochemical/biological parameters
(DOXY, FLU2, TUR3, CPHL, NTRA, …) as well as other non-core
measurements (CNDC, SVEL, …) without a hard-coded list.
# YAML header summary as JSON
ctddump report yaml --format json merged.yaml report.json
report summary
Assembles a single Markdown (default) or HTML (--format html) page for
one file stem (e.g. nrt_ar_ar) by reading the TSV reports the pipeline already
produced. It does not scan any Parquet itself, it aggregates the small per-stage
report TSVs, so it is instant.
Given a stem, the source files are auto-located at their standard pipeline
paths under --report-dir (default report) and --out-dir (default output):
| Section | Source file(s) |
|---|---|
| File summary | report/header/<stem>.yaml.tsv |
| Conversion | report/convert/<stem>.parquet.tsv |
| Pressure order | report/convert/checkpres/<stem>.summary.tsv and .platforms.tsv |
| Cleaning → Drop bad QC | report/clean/dropqc/<stem>.parquet.tsv |
| Cleaning → Drop all-NA profiles | report/clean/dropna/<stem>.parquet.tsv |
| Cleaning → Filter by region | report/clean/filter/<stem>.parquet.tsv |
| Deduplication → Mark duplicates | report/dedup/markdup/<stem>.parquet.tsv and output/dedup/markdup/<stem>.dups.tsv |
| Deduplication → Remove duplicates | report/dedup/dedup/<stem>.parquet.tsv |
| Comparison | report/compare/*.tsv (rows whose reference is <stem>) |
Any section whose file is absent is skipped, so a partially-run pipeline still
produces a valid page (a parent heading appears only when it has at least one
present subsection). The Mark-duplicates section shows its within/across tables
only when the .dups.tsv is also present.
Each stage table reports platforms / profiles / observations. The
% of original and Deleted columns compare against the Conversion stage
(the baseline “original”); if Conversion is absent, the earliest present stage is
used.
The two Deduplication stages ran on the cleaned data, not on the original, so
their tables carry three further columns, Cleaned, % of cleaned and
Deleted (cleaned), comparing them against the last cleaning stage present
(Filter, else Drop all-NA, else Drop bad QC). When no cleaning stage ran there is
nothing to compare against and the columns are omitted.
When the Conversion report has the {temp,psal,pres}_dm_{r,a,d} columns, the
Conversion section carries a second table, Data mode: one row per
parameter with the number of observations in real-time (R), adjusted (A),
and delayed mode (D), each with its share of all converted observations,
then Unknown for those with no data mode. Reports made before the
_dm_unknown counts existed have no Unknown columns, so their rows can sum to
less than 100%. Reports of files converted before the data-mode columns existed
have no data-mode columns at all, and the table is omitted.
The Pressure order section comes from checkpres, run on
the per-file Parquet before the merge. A first table gives the profiles checked (and
those without any pressure, which are not), those with decreasing pressure (split into reversed and non-monotonic) as shares
of all profiles, and the platforms and files that have any. When there are such
profiles, a Platforms table lists the 10 platforms with the most, with their
share of the platform’s profiles and the dates of the first and last one. The
full lists stay in the checkpres TSVs.
The Filter by region section carries a second table, Bounding box, with the
minimum and maximum longitude and latitude of the profiles that survived the
filter, in decimal degrees to three places. The extremes are aggregated from the
per-platform longitude_min / longitude_max / latitude_min / latitude_max
columns of the stage TSV, so they span every platform in the file. When the
Conversion report is present a second column, Original, gives the same extremes
for the converted data before any cleaning ran, which shows how far the filter
tightened the box. Profiles with a missing position are ignored, and if no profile
has a valid position the table is omitted entirely.
The Mark-duplicates section additionally lists the duplicate profiles that dedup
would remove, split into:
- within a platform: a
dup_groupconfined to a singleplatform_code; - across platforms: a group spanning two or more.
(A dup_group is a set of profiles sharing the duplicate key, date + rounded
position, so an across-platform group is the same cast reported by more than one
platform.)
A third table, Duplicate group sizes, shows how many profiles the groups hold:
one row per distinct size up to 10, then a single 11+ row for the tail.
The Comparison section appears when report/compare/ holds any
compare TSV whose reference column names this stem, i.e. a
comparison run with this product as the reference (the pipeline writes these with
compare_data.sh). It gathers every such row across all TSVs in
that directory. A first table, This product, gives the reference’s own totals
(platforms, profiles, profiles without a matchable key, observations), the
denominators the coverage is measured against; a second, Coverage of other
products, has one row per compared product: the common platforms and platform
coverage, the matched profiles and profile coverage, and, among the matched
profiles, how many carry the same versus a different observation count
(Same obs / Diff obs / Obs agreement). All percentages are recomputed from
the counts, so they do not depend on how the compare TSV formatted its own
percentage columns.
Title and notes
Each section carries a short, generic explanation of what the stage did, the same prose for every region and dataset. Anything region- or product-specific goes in:
--title TEXT: replaces the defaultSummary: <stem>heading.--note TEXT: a note rendered under the title; repeat for several notes.
Both are plain text and are escaped in HTML output. In the pipeline these are
supplied per stem by summary_data.sh, which is the place to edit
what a page says about a given region or product.
# Markdown page for the Arctic AR stem, to stdout
ctddump report summary nrt_ar_ar
# HTML page, with a title, notes, and non-default roots, to a file
ctddump report summary nrt_bo_bo --report-dir report --out-dir output \
--format html --title "Baltic Sea: Near Real Time, regional product (BO)" \
--note "Each source file holds a single platform." \
-o summary/nrt_bo_bo.html
filter
Filter a produced Parquet data file by a geographic bounding box, keeping or dropping whole profiles, and write the result to a new Parquet file.
ctddump filter [--mode include|exclude] \
--min-lon <W> --max-lon <E> --min-lat <S> --max-lat <N> \
<src.parquet> <dest.parquet>
The box is inclusive on all four edges, and --min-* must not exceed the
matching --max-* (an inverted box is rejected; antimeridian wrap is not
supported).
Because a profile’s longitude/latitude are constant across its observations,
the box acts on whole profiles, every observation of a matching profile is
kept or dropped together.
--mode (default include):
| Mode | Effect |
|---|---|
include | keep only profiles inside the box |
exclude | drop profiles inside the box, keep everything else |
A profile whose position is NaN (unknown) is treated as outside the box:
dropped by include, kept by exclude.
The file is streamed one row group at a time, so peak memory stays bounded
regardless of file size (tune with CTDDUMP_CHUNK_ROWS, as for convert).
# Keep only profiles inside a Mediterranean sub-box
ctddump filter \
--min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 \
merged.parquet med_box.parquet
# Remove profiles inside that box, keep the rest
ctddump filter --mode exclude \
--min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 \
merged.parquet outside_box.parquet
dropna
Filter a produced Parquet data file, dropping whole profiles that carry no usable data in one of the core measurement parameters, and write the survivors to a new Parquet file.
ctddump dropna <src.parquet> <dest.parquet>
A profile is kept only if each of temp, psal, and pres has at
least one non-NA observation. It is dropped if any one of those
parameters is entirely NA (null or NaN) across the profile. Partial NAs within a
parameter are fine, a kept profile retains all of its observations, including
the NA ones.
| Profile | temp | psal | pres | Result |
|---|---|---|---|---|
| some valid, some NaN | ✅ has data | ✅ has data | ✅ has data | kept |
| all NaN in one param | ✅ | ❌ all NaN | ✅ | dropped |
The file is processed in two streaming passes (build the keep-set, then re-emit
the kept rows), so peak memory stays bounded regardless of file size and the
result is independent of how the file is chunked (tune with CTDDUMP_CHUNK_ROWS,
as for convert).
# Drop profiles with an all-NA temp, psal, or pres
ctddump dropna merged.parquet cleaned.parquet
dropqc
Filter a produced Parquet data file, dropping whole profiles whose profile-level quality-control flags mark them as bad, and write the survivors to a new Parquet file.
ctddump dropqc <src.parquet> <dest.parquet>
A profile is kept only if both time_qc and position_qc are either
"1" (OK) or missing. A missing flag is one that is absent from the source
NetCDF, or stored as the NA byte -128, both render as the empty string ""
in the Parquet output. Any other flag ("0", "2"…"9", or a non-numeric char
code) drops the whole profile.
Missing QC is treated as acceptable on purpose: several source files ship no
profile-level position_qc (or time_qc) at all, and those profiles must be
kept rather than discarded. Note that the Argo “missing value” flag 9 is a
present value, not the NA byte, so a "9" flag is dropped.
time_qc | position_qc | Result |
|---|---|---|
1 (OK) | 1 (OK) | kept |
1 (OK) | "" (missing / NA) | kept |
"" (missing) | "" (missing) | kept |
4 (bad) | 1 (OK) | dropped |
1 (OK) | 0 (present, not OK) | dropped |
9 (present) | 9 (present) | dropped |
Because time_qc/position_qc are constant within a profile, this is a plain
per-row predicate that keeps or drops whole profiles. The file is streamed one
row group at a time (set_parallel(false)), so peak memory stays bounded
regardless of file size (tune with CTDDUMP_CHUNK_ROWS, as for convert).
# Drop profiles flagged bad in time_qc or position_qc
ctddump dropqc merged.parquet cleaned.parquet
markdup
Mark duplicate profiles in a produced Parquet data file with an is_dup
column, and write a TSV listing the duplicated profiles.
ctddump markdup [OPTIONS] <src.parquet> <dest.parquet> <dups.tsv>
Two profiles are duplicates when they share the same key, built from:
profile_timestamp: formatted with a strftime string; date only (%Y-%m-%d) by default.longitudeandlatitude: rounded to 3 decimals by default.
platform_code is deliberately not part of the key, so duplicates are
detected across platforms. A profile whose position is NaN or whose timestamp
is null has no key and is never marked.
The is_dup column (Boolean) is true for every profile whose key is shared by
at least one other profile. Two streaming passes bound memory regardless of file
size.
Options
| Option | Default | Meaning |
|---|---|---|
--time-format <FMT> | %Y-%m-%d | strftime format applied to profile_timestamp for the key |
--decimals <N> | 3 | decimal places longitude/latitude are rounded to |
--round-mode <MODE> | round | round, floor, ceil, or trunc |
Outputs
dest.parquet: the input plus theis_dupcolumn.dups.tsv: one row per duplicated profile, grouped bydup_group:dup_group, platform_code, profile_no, profile_time, profile_timestamp, longitude, latitude, n_obs.
# Default key: same date + coordinates rounded to 3 decimals
ctddump markdup merged.parquet marked.parquet duplicates.tsv
# Match on the full hour and 2-decimal coordinates
ctddump markdup --time-format "%Y-%m-%d %H" --decimals 2 merged.parquet marked.parquet dups.tsv
Use dedup to then drop the duplicates, and
report parquet to summarise the duplicate counts.
dedup
Remove duplicate profiles from a produced Parquet data file, keeping one profile per duplicate group.
ctddump dedup [OPTIONS] <src.parquet> <dest.parquet>
dedup re-derives the same duplicate key as markdup, from
profile_timestamp (date only by default) and longitude/latitude (3-decimal
rounding by default), so it can run standalone with the same options. Within
each group of profiles sharing a key, it keeps the profile with the most
observation rows; ties are broken by first appearance. Profiles with no key
(NaN position or null timestamp) are always kept.
If the input has an is_dup column (from markdup), it is reset to false:
the survivors are unique, so the schema stays stable across the
markdup → dedup pipeline and a follow-up report shows zero duplicates.
Two streaming passes bound memory regardless of file size.
Options
The key options match markdup and should be given the same values used
there:
| Option | Default | Meaning |
|---|---|---|
--time-format <FMT> | %Y-%m-%d | strftime format applied to profile_timestamp for the key |
--decimals <N> | 3 | decimal places longitude/latitude are rounded to |
--round-mode <MODE> | round | round, floor, ceil, or trunc |
# Drop duplicates using the default key (same date + 3-dp coordinates)
ctddump dedup marked.parquet deduped.parquet
compare
Compare two produced Parquet data files and report how far each one covers the other’s platforms and profiles.
ctddump compare [OPTIONS] <a.parquet> <b.parquet> [dest]
Without dest the report goes to stdout.
What it answers
Two questions, for a pair of files:
- How much do they have in common? Platforms present in both, and profiles present in both.
- Do the matched profiles agree? For every profile found in both files, whether it has the same number of observation rows in each.
Coverage is reported in both directions, because it is not symmetric. A small file fully contained in a large one is 100% covered while covering only a fraction of the large one. The first output row uses the second file as the reference, the second row uses the first.
How profiles are matched
A profile’s key is built from its platform code, its time reduced to a date,
and its longitude and latitude rounded to 3 decimals. This is the same key
markdup uses for duplicates, with one difference: markdup
deliberately leaves the platform code out, while compare includes it by
default. Two products covering the same waters usually share platform codes, so
the platform code is normally a wanted part of the identity. Pass
--no-platform-key to match on time and position alone, which finds the same
cast recorded under two different platform codes.
Profiles whose time or position is missing get no key and can never match. They
are counted separately as ref_unkeyed_profiles so a low coverage figure can be
told apart from a file full of unusable positions.
The time column may be either a datetime (profile_timestamp) or a float of days
since 1950 (profile_time); the type is detected and handled automatically.
Output columns
One row per direction:
| Column | Meaning |
|---|---|
reference | File the percentages are relative to |
compared | The other file |
ref_platforms | Distinct platform codes in the reference |
common_platforms | Platform codes present in both files |
platform_cov_pct | common_platforms as a percentage of ref_platforms |
ref_profiles | Profiles in the reference |
ref_unkeyed_profiles | Reference profiles with no usable time or position |
matched_profiles | Reference profiles whose key is present in the other file |
profile_cov_pct | matched_profiles as a percentage of ref_profiles |
same_nobs | Matched profiles with the same observation count in both files |
diff_nobs | Matched profiles whose observation count differs |
nobs_agree_pct | same_nobs as a percentage of matched_profiles |
ref_observations | Observation rows in the reference |
matched_observations | Observation rows in the reference’s matched profiles |
profile_cov_pct is taken over all reference profiles, including unkeyed
ones, since a profile that cannot be matched is still a profile the other file
does not demonstrably have. A percentage with no denominator (comparing against
an empty file, or nobs_agree_pct when nothing matched) is left empty rather
than shown as zero.
When several profiles share one key, the observation counts agree if any of the profiles carrying that key in the other file has the same count.
Options
| Option | Default | Meaning |
|---|---|---|
--time-format <FMT> | %Y-%m-%d | strftime format applied to the time column for the key |
--decimals <N> | 3 | decimal places longitude/latitude are rounded to |
--round-mode <MODE> | round | round, floor, ceil, or trunc |
--platform-col <NAME> | platform_code | column holding the platform code |
--time-col <NAME> | profile_time | column holding the profile time |
--lon-col <NAME> | longitude | column holding the longitude |
--lat-col <NAME> | latitude | column holding the latitude |
--no-platform-key | off | match on time and position only |
--format <FMT> | tsv | tsv, text, or json |
Examples
Compares are usually between two products covering the same waters, for example the CORA reanalysis against a regional NRT extract.
# How much of the Arctic NRT product does CORA cover, and vice versa
ctddump compare --format text cora.parquet nrt_ar.parquet
# Save the summary
ctddump compare cora.parquet nrt_ar.parquet compare.tsv
# Ignore platform codes, so the same cast filed under two codes still matches
ctddump compare --no-platform-key cora.parquet nrt_ar.parquet
# Match to the hour and to 4 decimals instead of the date and 3
ctddump compare --time-format '%Y-%m-%dT%H' --decimals 4 cora.parquet nrt_ar.parquet
Reading the result: if the smaller product shows profile_cov_pct near 100
while the larger shows much less, the smaller is close to a subset of the larger.
If both are low, the two products overlap only partly. A high profile_cov_pct
with a low nobs_agree_pct means the same profiles are present in both but carry
different numbers of observations, which usually points at different QC or
cleaning in the two products.
Memory is bounded: each file is streamed and reduced to one record per profile, so peak use follows the profile count rather than the file size.
sqlite
Export a produced Parquet data file to a SQLite database, normalised into three related tables. This turns the flat observation table into something you can query with SQL, index, and open in any SQLite-aware tool.
ctddump sqlite [OPTIONS] <src.parquet> [dest.sqlite]
Without dest the output is the source path with its extension changed to
.sqlite. The command refuses to overwrite an existing output unless --force
is given.
The schema
The flat Parquet repeats every profile-level value on each observation row.
sqlite splits that back apart by grain:
platformholds one row perplatform_code.profileholds one row per(platform_code, profile_no), carrying every column that is constant within a profile (time, position, QC, filename), with a foreign key toplatform.observationholds one row per observation (the measurements and their QC and conversion flags), with a foreign key toprofile.
CREATE TABLE platform (
platform_id INTEGER PRIMARY KEY,
platform_code TEXT UNIQUE NOT NULL
);
CREATE TABLE profile (
profile_id INTEGER PRIMARY KEY,
platform_id INTEGER NOT NULL REFERENCES platform(platform_id),
profile_no INTEGER NOT NULL,
profile_time REAL, -- days since 1950-01-01
profile_timestamp INTEGER, -- Unix milliseconds
longitude REAL,
latitude REAL,
profile_longitude REAL,
profile_latitude REAL,
time_qc TEXT,
position_qc TEXT,
filename TEXT,
UNIQUE(platform_id, profile_no)
);
CREATE TABLE observation (
observation_id INTEGER PRIMARY KEY,
profile_id INTEGER NOT NULL REFERENCES profile(profile_id),
observation_no INTEGER NOT NULL,
temp REAL, temp_qc TEXT, temp_dm TEXT,
psal REAL, psal_qc TEXT, psal_dm TEXT,
pres REAL, pres_qc TEXT, pres_dm TEXT, pres_conv INTEGER,
deph REAL, deph_qc TEXT, deph_conv INTEGER
);
Indexes are created on profile(platform_id), observation(profile_id), and
profile(profile_timestamp) so the common joins and time queries are fast.
Only the columns of the standard output schema are
exported; any other column in the Parquet (for example markdup’s is_dup) is
ignored unless you ask for it with --add-col. A column that is absent from the
input is simply left out of its table, so a file that never had deph produces a
schema without the depth columns.
NULL handling
The Parquet uses sentinels for missing values: NaN for floats and an empty
string for absent QC flags and data modes. Both are stored as SQL NULL, so queries like
WHERE temp IS NOT NULL behave the way SQL expects.
Adding your own columns
Two options attach extra columns to a chosen table. Both name their target as
TABLE.COLUMN, where TABLE is platform, profile, or observation.
--add TABLE.COLUMN=VALUE adds a constant column filled with one value on
every row of the table. It is handy for tagging an export with where it came
from. The value type is inferred: a whole number becomes an integer, a decimal
becomes a real, anything else is stored as text. Repeat the flag for several
columns.
--add-col TABLE.COLUMN copies an existing Parquet column of that name into
the table. For a column added to observation the value is taken per row. For a
column added to profile or platform the value must be constant within each
group (each profile, or each platform): if it varies, the export stops with an
error naming the column and the group. Nulls are ignored when checking, so a
column that is set on some rows of a profile and null on others is accepted.
A column name must be a plain identifier and must not clash with a column the table already has.
Examples
# Straight export next to the source (writes arctic.sqlite)
ctddump sqlite arctic.parquet
# Explicit output, overwriting any earlier file
ctddump sqlite --force arctic.parquet /data/arctic.sqlite
# Tag every row with the source product and region
ctddump sqlite --add profile.dataset=NRT_AR --add platform.source=copernicus \
arctic.parquet arctic.sqlite
# Carry a per-profile institution and a per-observation extra parameter through
ctddump sqlite --add-col profile.institution --add-col observation.chla \
arctic.parquet arctic.sqlite
A typical query against the result, joining all three tables:
SELECT pl.platform_code, p.profile_no, p.profile_timestamp,
o.pres, o.temp, o.psal
FROM observation o
JOIN profile p ON o.profile_id = p.profile_id
JOIN platform pl ON p.platform_id = pl.platform_id
WHERE o.temp IS NOT NULL
ORDER BY pl.platform_code, p.profile_no, o.pres;
Memory
The export streams in two passes and never loads the whole file at once. The
first pass builds the platform and profile rows, so its memory follows the
number of profiles rather than the file size (the same assumption markdup and
dedup rely on). The second pass streams the observations in fixed windows and
inserts them, so an arbitrarily large observation body loads a window at a time.
Configuration
All convert and batch convert subcommands accept a --config TOML file plus
individual flag overrides. The priority order is:
built-in default < --config file < individual CLI flags
--config may set any subset of fields; a per-field flag then overrides the
config for that field only.
NRT flags
| Flag | Field | Default |
|---|---|---|
--deph-source / --no-deph-source | has_deph_source | true for BO/GL, false for AR/MO |
--profile-coords / --no-profile-coords | has_profile_coords | true for BO, false otherwise |
--pattern <GLOB> | pattern | see the batch patterns |
DEPH is auto-detected. NRT always performs the bidirectional PRES↔DEPH conversion when the file contains a
DEPHvariable, regardless ofhas_deph_source. The flag only forces DEPH handling on for files where the variable might otherwise be skipped.
CORA flags
| Flag | Field | cora default | cora_legacy default |
|---|---|---|---|
--time-var <VAR> | time_var | TIME | JULD |
--qc-type <int|char> | qc_type | int | char |
--time-qc / --no-time-qc | has_time_qc | true | false |
--deph-source / --no-deph-source | has_deph_source | true | false |
--pattern <GLOB> | pattern | *.nc | *.nc |
Generic flags
The generic format reads a mapping file that names your file’s dimensions and
variables; see Your own NetCDF files
for its keys.
| Flag | Field | Default |
|---|---|---|
--layout <time-depth|profile-level> | layout | time-depth |
--time-units <UNITS> | time_units | the time variable’s units attribute |
--platform-code <CODE> | [defaults] platform_code | none |
--pattern <GLOB> | pattern | *.nc |
TOML config format
# NRT
has_deph_source = true
has_profile_coords = false
pattern = "AR_PR_CT_*.nc" # optional
# CORA
time_var = "TIME"
qc_type = "int" # "int" or "char"
has_time_qc = true
has_deph_source = true
pattern = "*.nc" # optional
Environment variables
ctddump honours a few environment variables for operational tuning. They do
not change the contents of the output, only memory use, speed, and the
on-disk row-group layout, which is why they live in the environment rather than
as per-command flags.
| Variable | Effect | When unset |
|---|---|---|
CTDDUMP_CHUNK_ROWS | Observation rows assembled per streamed chunk (each written as one Parquet row group). Lower = less memory but more row groups; higher = more memory but fewer. Applies to convert, batch, concat, and the streaming filters (filter / dropqc / dropna / markdup / dedup). | 1000000 |
POLARS_MAX_THREADS | Size of Polars’ internal thread pool. batch / concat pin this to 1 so their own --threads is the real knob, but a value you set is respected. | pinned to 1 by batch / concat |
RUST_MIN_STACK | Worker-thread stack size, in bytes. Raised to 16 MiB for the deep call chains in Polars’ Parquet writer; a larger value you set is respected. | raised to 16 MiB |
CTDDUMP_CHUNK_ROWS is the only one most users would ever change, see
Technical notes for why streaming in chunks matters. The
helper scripts expose it directly as --chunk-rows N, which
exports it for every ctddump process they launch.
Shell completion
ctddump completions <shell> prints a completion script to standard output. Once
your shell loads that script, typing ctddump and pressing Tab suggests the
subcommands, and Tab-completing a partial word (a subcommand, a flag, or an enum
value such as a shell name) fills it in when only one candidate remains.
Supported shells: bash, zsh, fish, elvish, powershell.
The command only writes the script; where you put it is what makes completion permanent. Pick the section for your shell.
Bash
# Try it in the current shell
source <(ctddump completions bash)
# Install for every new shell (user-local)
mkdir -p ~/.local/share/bash-completion/completions
ctddump completions bash > ~/.local/share/bash-completion/completions/ctddump
This relies on the bash-completion package being installed and sourced from
your ~/.bashrc. Without it, add source <(ctddump completions bash) to
~/.bashrc directly.
Zsh
# A directory on your $fpath, created if needed
mkdir -p ~/.zsh/completions
ctddump completions zsh > ~/.zsh/completions/_ctddump
Make sure that directory is on $fpath and completion is initialised, before the
compinit call in your ~/.zshrc:
fpath=(~/.zsh/completions $fpath)
autoload -Uz compinit && compinit
Start a new shell (or run compinit) to pick up the change.
Fish
mkdir -p ~/.config/fish/completions
ctddump completions fish > ~/.config/fish/completions/ctddump.fish
Fish loads it automatically in new shells.
PowerShell
# Add to your profile so it loads every session
ctddump completions powershell | Out-String | Invoke-Expression
To make it permanent, append that line to the file at $PROFILE.
Elvish
ctddump completions elvish > ~/.config/elvish/lib/ctddump.elv
# then, in rc.elv:
use ctddump
Regenerating after an upgrade
The script encodes the commands and flags of the version that produced it. After
upgrading ctddump, regenerate it with the same command so new subcommands and
options complete correctly.
Output schema
Every converter produces the same observation-level flat table, so files from different sources can be merged directly.
| Column | Type | Notes |
|---|---|---|
platform_code | String | |
profile_no | u32 | see Row order and numbering |
profile_time | f64 | days since 1950-01-01 |
profile_timestamp | Datetime(ms) | Unix milliseconds |
observation_no | u32 | see Row order and numbering |
longitude / latitude | f32 (NRT) / f64 (CORA) | generic: as NRT for time-depth, as CORA for profile-level |
profile_longitude / profile_latitude | f32 (NRT) / f64 (CORA) | from PRECISE_* or expanded DEPLOY_*; NaN when unavailable (always for generic) |
time_qc / position_qc | String | "" if absent |
filename | String | source file stem |
temp, psal, pres, deph | f32 | NaN where missing |
temp_qc, psal_qc, pres_qc, deph_qc | String | single-char flag; "" if missing |
temp_dm, psal_dm, pres_dm | String | data mode: R, A, or D; "" when unknown (see below) |
pres_conv, deph_conv | i8 | 1 = value derived by conversion |
Pressure and depth are converted to each other with the TEOS-10 (gsw)
routines when only one is present, and the *_conv columns flag which values
were derived rather than measured.
Row order and numbering
concat convert sorts each profile’s observations by increasing pressure.
A single convert (or batch convert) does not: it keeps the order of the
source file. Which order a Parquet file has depends on which step wrote it.
| Step | Observation order | profile_no | observation_no |
|---|---|---|---|
convert, batch convert | as in the source file (its level dimension) | the profile’s index in the file, from 1 | the level’s index in the file, from 1; levels with no valid measurement are dropped, leaving gaps |
concat convert (default) | increasing pres within each profile | renumbered from 1 within each platform | renumbered 1..N within each profile, following increasing pres |
concat convert --no-pres-sort | as in the source files | renumbered | renumbered 1..N, following the source order |
concat convert --no-renumber | as read, unsorted | kept from the input | kept from the input |
Details of the default concat convert:
- Rows are sorted by
platform_code, profile_timestamp, longitude, latitude, pres. Profiles are therefore in time order within each platform, and the observations of a profile run from the surface down. - Rows whose
presis missing are dropped first, so a profile with no pressure at all disappears here. With--keep-na-presthey are kept and sorted to the end of their profile, after the valid pressures. - Observations with the same pressure keep their source order (by file path, then by position in the file).
profile_nois assigned per distinct(platform_code, profile_timestamp, longitude, latitude). Two profiles from different files that share all four become one profile, and the pressure sort interleaves their observations.filenamestill tells them apart.- All other columns move with their row; nothing is interpolated or averaged.
The pipeline scripts and the regional workflows merge with the default, so
their merged Parquet files, and everything cleaned, de-duplicated, or
summarised from them, are sorted by pressure. Pass --no-pres-sort to keep the
acquisition order instead (see concat). To find the
profiles whose source order is not increasing in pressure, run
checkpres on the per-file Parquet before the merge.
Data mode
The *_dm columns record how each value was processed:
| Code | Meaning |
|---|---|
R | real-time |
A | adjusted in real time |
D | delayed mode |
The code is copied from the source file; ctddump never infers it. Where it comes from depends on the source:
- NRT, with a
<PARAM>_DMvariable (TEMP_DM,PSAL_DM,PRES_DM; found in some Arctic files, whose parameters then carrydata_mode = "M"for “mixed”): the per-observation code is copied through. A fill cell becomes"". - NRT, without one: every observation gets the parameter’s
data_modeattribute when it isR,A, orD. Any other attribute, notably"M", says the mode varies per observation without saying how, so the column is""; a missing attribute gives""too. - NRT pressure derived from depth (
pres_conv = 1):pres_dmis taken fromDEPH_DM(or DEPH’sdata_modeattribute), the same waypres_qctakesdeph_qc. - CORA: there are no per-parameter data-mode variables, so all three columns
hold the profile’s
DATA_MODE, repeated on each of its observations (""if the file has noDATA_MODE, or for a profile whose cell is a fill value). generic: as NRT for thetime-depthlayout and as CORA forprofile-level, except that[defaults] data_modereplaces""when the file has no data mode at all (see Your own NetCDF files). A"M"attribute stays""even with a default.
A parameter the file does not have at all (all NaN) has "" as well.
convert and batch convert print one warning per file whose data mode is
left empty for a whole variable (a "M" attribute without <PARAM>_DM, no
data-mode information at all, or a CORA file without DATA_MODE), naming the
variables. Fill cells inside a data-mode variable are not reported.
Keep in mind when using these columns:
""means “no code in the file”, not “no measurement”: a valid value can have an empty data mode, and a missing value can have a code.- The values and flags are always the ones in
TEMP/TEMP_QC(and so on);*_ADJUSTEDvariables are never read. In Copernicus NRT files the delayed-mode values and flags are in those variables, with<PARAM>_DMsaying which observations they are. - CORA’s
DATA_MODEcomes from the original observation. CORA re-processes the QC of every profile, soRthere does not mean the flags are real-time ones.
Technical notes
This page collects the non-obvious technical problems we ran into while building
ctddump and how each was solved. It is written for a general audience, you do
not need to know Rust to follow it. The recurring themes are memory use,
multi-threading, how data is read and written in chunks, and a handful of
quirks in the Polars data-frame library we depend on.
Each note is laid out the same way: what the symptom looked like, why it happened, and what we did about it.
1. Converting a file used far too much memory
Symptom. Converting a single large NetCDF file (a few hundred MB on disk) pushed memory use to ~8–9 GB, enough to fail on modest machines.
Why. A NetCDF CTD file is stored as a dense rectangular grid: every profile (a cast of the instrument) times every depth level. Most of that grid is empty: a shallow cast still reserves a slot for every deep level other casts reach. If you load the whole grid into memory at once and only then throw away the empty cells, you have already paid for the full rectangle.
Fix: stream the file in chunks. Instead of loading everything, the converter walks the file in slices along the profile dimension (about one million data rows at a time). Each slice is turned into a small table and written straight to disk as one “row group” of the output Parquet file, then discarded before the next slice is read. The file is only ever partially in memory. On a 230 MB test file this cut peak memory from ~8.8 GB to ~0.7 GB, and lower still with smaller chunks.
The chunk size defaults to one million rows and can be tuned with the
CTDDUMP_CHUNK_ROWS environment variable, smaller means less memory but more
row groups. The helper scripts expose it as --chunk-rows N, and
it is listed alongside the other tuning variables under
Configuration. Importantly, the
chunking only changes how the output is laid out on disk; the actual data and
its order are identical no matter what chunk size you pick.
2. Empty rows were dropped the slow, memory-leaky way
Symptom. Running the batch converter over thousands of small files caused memory to climb steadily and never come back down, roughly 0.2 MB lost per file, with no upper limit. Converting 7,905 tiny files eventually consumed 88 GB.
Why. This is a bug in the version of the Polars library we use (0.43.1). Several of Polars’ parallel operations, filtering rows, gathering rows, and the default parallel way of writing Parquet columns, leak a small amount of memory on every call that is never released. On one file it is invisible; across thousands of files in one long-running process it adds up without bound. The leak is independent of the memory allocator and cannot be reclaimed manually.
Fix: avoid the leaky parallel paths. Two changes, both inside the converters:
- When dropping the empty (all-missing) rows, we do it on the plain raw arrays before handing the data to Polars, so Polars’ leaky parallel filter is never called and the empty rows never enter Polars at all.
- Every Parquet write in the converters (and in
concat) turns off Polars’ parallel column encoding.
Together these keep batch memory flat regardless of how many files are processed (3,000 small files went from ~0.9 GB down to ~40 MB), and the output is verified identical to the old parallel path. If a future Polars release fixes the leak, these workarounds can be revisited.
3. Reading a big Parquet file in slices returned the wrong rows
Symptom. Commands that re-read a Parquet file in slices (filter, dropqc,
dropna, markdup, dedup) silently produced wrong results on large files:
for example, output that only contained platforms whose names started with the
first few letters of the alphabet, as if most of the file had vanished.
Why. Another bug in Polars 0.43.1. When a Parquet file has more than one row group (which any file bigger than one chunk does), and you ask its parallel reader for “the rows from position X onwards,” it ignores X and keeps returning rows from the very first row group. So every slice read the same opening chunk over and over, and the rest of the file was never seen. It was invisible in our automated tests because test fixtures are small enough to be a single row group.
Fix: use the sequential reader for sliced reads. The five affected commands
now read Parquet with Polars’ non-parallel reader, which honours the slice
position correctly. Whole-file operations (like report) are unaffected and keep
the faster parallel reader. A regression test now writes a deliberately
many-row-group file and slices through it to guard against this coming back.
This bug was also the root cause of a confusing field report where
dropqcoutput looked truncated: the user was running an older build from before this fix. Rebuilding to the current version resolved it.
4. markdup crashed on large files with a “record batch” error
Symptom. markdup panicked on large merged files with
RecordBatch requires all its arrays to have an equal number of rows, and left a
truncated, unreadable output file. It worked fine on small files.
Why. markdup reads the file in slices, adds a new true/false is_dup
column to each slice, and writes it out. When a slice happened to span several
row groups of the input, the columns it read back were internally split into
several pieces (“chunks”), while the freshly-built is_dup column was a single
piece. The Parquet writer writes one batch per internal piece and insists every
column be split the same way, the mismatch made it crash. Small files have only
one row group, so their columns were never split and the bug never showed.
Fix: line up the pieces before writing. After adding the is_dup column,
markdup now re-aligns all columns to the same internal chunk layout before
writing each slice. The sibling commands did not need this because their
row-filtering step happens to realign the columns for free. A new test feeds
markdup a multi-row-group input to keep this covered.
5. Multi-threaded batch runs crashed with a stack overflow
Symptom. Batch conversion processed each file on a separate worker thread and crashed on large files, even though converting the same file on its own worked.
Why. Worker threads get a much smaller memory “stack” by default (about 2 MB) than the program’s main thread (about 8 MB). Polars’ Parquet writer uses deep call chains on large files that need more stack than the 2 MB default, so it overflowed only when run on a worker thread.
Fix: give the workers a bigger stack. Batch mode builds its thread pool with a 16 MB stack per worker, and the program also raises the stack size for Polars’ own internal threads. Single-file conversion never hit this because it runs on the roomier main thread.
6. Asking for N threads actually spawned far more
Symptom. Requesting, say, 10 threads created many more than 10 busy threads, oversubscribing the machine.
Why. Polars keeps its own thread pool, sized to the number of CPU cores. So running our 10 file-workers, each of which calls into Polars, produced roughly 10 + (number of cores) threads all competing for the CPU.
Fix: pin Polars to one internal thread. Batch and concat set Polars to a
single internal thread before the first Polars call, so our own --threads
setting is the real, honest knob. We already parallelise across files (or across
data ranges in concat), so we do not need Polars to add a second layer of
threading on top.
7. A single large file stalled all the other threads
Symptom. In a multi-threaded batch run, watching a system monitor showed all cores busy at first, then most of them going idle near the end while one core finished a large file. It looked as though files were processed in fixed groups that waited for the slowest member.
Why. The work scheduler treats every file as equally expensive, it has no idea one file is much bigger than the others, and a single file cannot be split across threads. If a big file happened to be picked up late, the other workers would finish all the small files, find nothing left to do but that one big file (which they cannot help with), and sit idle until it finished.
Fix: start the biggest files first. The batch converter now sorts files largest-first before handing them out, and dispatches them one at a time so a free worker always grabs the next individual file. Starting heavy files at the very beginning lets the many small files fill in around them, so the run finishes close to “total work ÷ threads” instead of being dragged out by a late big file. The one unavoidable limit: if a single file is larger than everything else combined, its own runtime still sets the floor, no scheduler can split one file across threads. Starting it first is simply the best possible placement.
8. Merging many files without loading them all at once
Challenge. concat merges many Parquet files into one and re-numbers the
profiles consistently across the whole result. Doing that naively means holding
every input in memory at the same time.
Fix: merge one platform at a time. Re-numbering is grouped by platform
(each platform’s profiles are numbered together), so concat processes one
contiguous block of platforms at a time and writes each block out as it goes. A
quick first pass reads only the tiny “platform” column of every file to plan the
blocks; the second pass re-reads only the files that contribute to each block.
Because a given platform’s data always lands in the same block, profiles that
were split across input files are still merged correctly. Producing the blocks in
order gives a result identical to merging everything at once, only the on-disk
layout differs.
When more than one thread is allowed, whole platform blocks are independent units
of work, so concat renumbers several blocks at once, each writing to a
temporary file, and then stitches the temporaries together in order. The final
file is byte-for-byte identical to the single-threaded result.
9. Missing datasets should not break the pipeline
Symptom / need. One planned dataset (the global product for the Baltic Sea) is not yet published by the data provider, but the workflow already refers to it. We did not want the whole run to fail just because those files do not exist yet.
Fix: treat “no matching files” as a no-op, not an error. When a batch or concat command is pointed at a pattern that matches nothing (or an empty/absent folder), it now prints a short informational message and writes no output, instead of stopping with an error. The helper scripts likewise skip any step whose input file is missing, with a note. The result is that the pipeline runs cleanly today and will automatically pick up the Baltic global data the moment it becomes available, with no further changes.
10. Harmless HDF5 diagnostic messages
Symptom. On some systems, reading certain files prints scary-looking
HDF5-DIAG messages to the screen.
Why. These come from an older version of the underlying HDF5 library (1.10.x, used on our continuous-integration runner) encountering a newer file attribute it does not recognise. It is only a diagnostic, the data is read correctly.
What to do. Nothing. The messages are harmless and the output is unaffected. Using a newer HDF5 library avoids them.
A recurring lesson
Most of the hard bugs above share a shape: they only appear at scale. A file big enough to have multiple row groups, or a run long enough for a small leak to matter, or a workload uneven enough for scheduling to bite. Small test fixtures pass happily while the real data fails. Wherever we fixed one of these, we also added a test that deliberately recreates the “large” condition (many row groups, tiny chunks, mixed sizes) so the fix stays fixed.
Your own NetCDF files
ctddump is built around the Copernicus files, but it can also convert your own
NetCDF files, for example from a mooring, a glider campaign, or a research
cruise. Use the generic format with a small mapping file that tells
ctddump what your dimensions and variables are called. The output has the same
schema as the Copernicus conversions, so your data can be
merged, cleaned, de-duplicated, and reported on with the rest of the toolkit.
ctddump convert generic --config my_map.toml my_data.nc my_data.parquet
What your file needs
Your file must be laid out like a Copernicus file: a profile dimension and a
level dimension, with the measurements stored as 2-D variables indexed
[profile, level], in that order.
| Information | Where it can come from | Required? |
|---|---|---|
| Profile and level dimensions | any two dimension names | yes |
| Time | a variable, scalar or [profile], with CF units such as "seconds since 1970-01-01" | yes |
| Longitude, latitude | variables, scalar or [profile], or a default value | yes |
| Platform code | a global attribute, a variable, or a default value | yes |
| Pressure or depth | a [profile, level] variable | at least one |
| Temperature or salinity | a [profile, level] variable | at least one |
| QC flags | [profile, level] variables (integer or char) | no: empty when absent |
| Data mode | variables, a data_mode attribute, or a default value | no: empty when absent |
Two layouts are supported. They differ in what a profile is and where the platform code usually lives:
time-depth (default) profile-level
like Copernicus NRT like Copernicus CORA / Argo
one platform per file many profiles, many platforms
platform: global attribute platform: [profile] variable
level -> level ->
profile [ x x x x ] TEMP[TIME,DEPTH] profile [ x x x . ] TEMP[N_PROF,N_LEVELS]
| [ x x x . ] | [ x x . . ]
v [ x x x x ] v [ x x x x ]
TIME[TIME] TIME[N_PROF], PLATFORM_NUMBER[N_PROF, ...]
Profiles are numbered in file order (profile_no), and levels within each
profile (observation_no). Cells where temperature, salinity, and pressure are
all missing are dropped.
Example: a mooring
Suppose ncdump -h mooring.nc shows:
dimensions:
time = 8760 ;
z = 12 ;
variables:
double time(time) ;
time:units = "seconds since 1970-01-01 00:00:00" ;
float lat ;
float temperature(time, z) ;
temperature:_FillValue = -999.f ;
short temperature_flag(time, z) ;
short sal(time, z) ;
sal:scale_factor = 0.01 ;
sal:add_offset = 30. ;
double pressure(time, z) ;
The file has no longitude and no platform code, so the mapping supplies both as defaults:
# mooring.toml
[dimensions]
profile = "time"
level = "z"
[variables]
time = "time"
latitude = "lat"
temp = "temperature"
temp_qc = "temperature_flag"
psal = "sal"
pres = "pressure"
[defaults]
platform_code = "MOORING_1"
longitude = 20.0
data_mode = "D" # our own, quality-controlled data
ctddump convert generic --config mooring.toml mooring.nc mooring.parquet
ctddump report parquet mooring.parquet # check the result
Time is converted from seconds since 1970 to the output’s days since 1950; the
fill values become NaN; salinity is unpacked with scale_factor and
add_offset; and depth is derived from pressure (flagged with deph_conv = 1).
Example: a cruise
A cruise file with many casts, each with its own ship code, uses the
profile-level layout:
dimensions:
cast = 54 ;
level = 400 ;
strlen = 8 ;
variables:
char ship(cast, strlen) ;
double time(cast) ;
time:units = "days since 2000-01-01T00:00:00Z" ;
double lon(cast) ;
double lat(cast) ;
float depth(cast, level) ;
float TEMP(cast, level) ;
char TEMP_QC(cast, level) ;
float PSAL(cast, level) ;
char PSAL_QC(cast, level) ;
# cruise.toml
layout = "profile-level"
[dimensions]
profile = "cast"
level = "level"
[variables]
time = "time"
longitude = "lon"
latitude = "lat"
deph = "depth"
# TEMP, TEMP_QC, PSAL and PSAL_QC already have the default names.
[platform]
variable = "ship"
Pressure is derived from depth here. The char QC flags are detected automatically.
Many files at once
batch convert generic converts a directory tree with the same mapping, and
header generic / batch header generic export the metadata (dimensions,
variables, and all global attributes). The results feed into the usual
pipeline:
ctddump batch convert generic --config cruise.toml --output output/convert source/cruises
ctddump batch header generic --output output/header source/cruises
ctddump concat convert output/convert output/cruises.parquet
ctddump concat header output/header output/cruises.yaml
ctddump report parquet output/cruises.parquet
batch matches *.nc by default; set --pattern (or pattern in the mapping
file) to narrow it. The helper scripts are written for the
Copernicus downloads and do not run the generic format.
The mapping file
Every key is optional. A missing key takes the Copernicus name, so an empty
mapping file describes a Copernicus NRT file. Setting a variable name to ""
says the file has no such variable. Misspelt keys are rejected rather than
ignored.
layout = "time-depth" # or "profile-level"
time_units = "..." # overrides the time variable's units attribute
qc_type = "int" # or "char"; detected per variable when omitted
pattern = "*.nc" # batch file selection
[dimensions]
profile = "TIME" # profile-level: N_PROF
level = "DEPTH" # profile-level: N_LEVELS
[variables]
time = "TIME"
longitude = "LONGITUDE"
latitude = "LATITUDE"
time_qc = "TIME_QC"
position_qc = "POSITION_QC"
temp = "TEMP" # temp_qc / temp_dm default to "<temp>_QC" / "<temp>_DM"
psal = "PSAL" # likewise psal_qc / psal_dm
pres = "PRES" # likewise pres_qc / pres_dm
deph = "DEPH" # likewise deph_qc / deph_dm
data_mode = "DATA_MODE" # per-profile data mode (profile-level layout)
[platform] # one of the two
attribute = "platform_code" # time-depth default: a global attribute
# variable = "PLATFORM_NUMBER" # profile-level default: a per-profile variable
[defaults] # used when the file does not provide the value
platform_code = "MY_PLATFORM"
longitude = 20.0
latitude = 60.0
data_mode = "D" # R, A or D
A few flags override the mapping file, which helps when one mapping serves several platforms:
| Flag | Mapping key |
|---|---|
--layout <time-depth|profile-level> | layout |
--time-units <UNITS> | time_units |
--platform-code <CODE> | [defaults] platform_code |
--pattern <GLOB> | pattern |
The priority is: built-in default, then the --config file, then the flags.
How values are read
- Missing values.
_FillValuebecomes NaN; without one,missing_valueand then the NetCDF default fill value for the type are used. This applies to time and position as well as to the measurements. - Packed values.
scale_factorandadd_offsetare applied. - Time. The units may be days, hours, minutes, seconds, or milliseconds
since a reference date, with an optional time and time zone
(
"hours since 2000-01-01 06:00 +01:00"). Thecalendarmust bestandard,gregorian, orproleptic_gregorian(or absent). Months and years are not accepted because their length varies. - Platform code. A variable may be
char [profile, strlen],string [profile], an integer[profile](e.g. a WMO number), or a scalar of any of these for the whole file. Blank entries take[defaults] platform_code. - QC flags. Integer flags
0to9are written as"0"to"9", other values (fill values included) as""; any integer type works. Char flags are copied as they are, with blanks written as"". - Data mode. In the
time-depthlayout, a[profile, level]char variable per parameter (TEMP_DM); in theprofile-levellayout, one per-profile variable (DATA_MODE). Without one, the parameter’sdata_modeattribute (R,AorD), then[defaults] data_mode, then""(unknown). Adata_modeattribute of"M"(mixed) gives""whatever the default, since only a per-level variable can say which levels are which. Each file whose data mode is left empty this way gets a warning; set[defaults] data_modewhen you know the mode of your data. - Pressure and depth. Whichever is missing is derived from the other with
TEOS-10 at the profile’s latitude, and flagged in
pres_conv/deph_conv.
Limits and workarounds
| Your file | What to do |
|---|---|
A single cast with only a level dimension, TEMP(depth) | Add a profile dimension of length 1, e.g. with NCO: ncecat -O -u cast in.nc out.nc |
Measurements stored [level, profile] | Reorder the dimensions: ncpdq -a cast,level in.nc out.nc |
| Position or time per observation (a trajectory) | Not supported: the output has one position and one time per profile. |
| A profile name or ID variable | Not carried into the output. Profiles are identified by platform_code, profile_no, and time. |
Troubleshooting
ctddump checks the file against the mapping before converting and names the
problem:
| Message | Fix |
|---|---|
dimension "depth" not found (the file has: ...) | Set [dimensions] profile / level to names from the list. |
time variable "TIME" not found | Set [variables] time. |
time variable "time" has no units attribute | Set time_units (or pass --time-units). |
unsupported time units "..." | Use a "<unit> since <date>" form; see How values are read. |
longitude variable "LONGITUDE" not found | Set [variables] longitude, or [defaults] longitude for a fixed position. |
platform global attribute "platform_code" not found | Set [platform] attribute / variable, or [defaults] platform_code. |
neither pressure "PRES" nor depth "DEPH" found | Set [variables] pres or deph. |
variable "X" has dimensions [...]; expected [...] | The variable has the wrong shape; see Limits and workarounds. |
unknown field ... | A misspelt key in the mapping file. |
ctddump header generic my_data.nc my_data.yaml lists every dimension and
variable in a file, which helps when writing the mapping.
Helper scripts
The scripts/
directory ships seven Bash scripts that automate the full regional pipeline,
the same steps documented one command at a time in the
Regional workflows. They run in seven phases, each
consuming the previous phases’ output:
| Phase | Script | What it does |
|---|---|---|
| 1. Download | download_data.sh | Download the source NetCDF from Copernicus Marine into source/. |
| 2. Convert | convert_data.sh | Convert + merge to Parquet, export + merge headers. |
| 3. Clean | clean_data.sh | Drop bad-QC profiles, drop all-NA profiles, restrict to the region. |
| 4. De-duplicate | dedup_data.sh | Mark duplicate profiles, then remove them. |
| 5. Compare | compare_data.sh | Cross-compare each region’s products (NRT, GL, CORA) for coverage. |
| 6. Summarise | summary_data.sh | Build a per-unit Markdown/HTML summary page from the reports. |
| 7. Publish | summary_site.sh | Render the Markdown summary pages into a static mdBook site. |
Each phase handles the Arctic, Baltic, and Mediterranean regions.
The scripts only orchestrate the ctddump binary, so it must be on your PATH,
except download_data.sh, which instead needs the copernicusmarine toolbox
(and a free Copernicus Marine account) and does not use ctddump at all, and
summary_site.sh, which needs mdBook
(cargo install mdbook).
The Compare phase feeds the Summarise phase: compare_data.sh writes a coverage
summary for each pair of products, and summary_data.sh folds the rows that name
a page’s own product into that page’s Comparison section. It is otherwise
self-contained, so you can also run it on its own to inspect the coverage TSVs
directly.
Running a script
Run the scripts from a working directory (e.g. ctddump). Source NetCDF is read
from source/, data products are written under output/, and summary reports
under report/; all are created as needed. They share one interface:
scripts/<script>.sh [options] [command] [region ...]
- command: the step (or
all) to run; each script defaults to its most common command when omitted (see below). - region: one or more of
arctic,baltic,mediterranean. Omitting them (or passingall) runs every region. - options: may appear anywhere on the line, in either
--out DIRor--out=DIRform.
Before doing any work, a script prints the resolved configuration and asks for confirmation:
$ scripts/clean_data.sh dropqc
Configuration:
command : dropqc
regions : arctic baltic mediterranean
files : 8
out : output
report : report
chunk : default
mode : parallel (per file)
Run with -h/--help to see all options.
Proceed? [y/N]
Answer y to proceed; anything else (including a bare Enter) aborts. Pass
-y/--yes to skip the prompt, required when running non-interactively (a
pipe or CI job), where the script otherwise aborts with a hint rather than hang.
Every script closes its configuration block with the -h/--help reminder shown
above, so the full option list is always one step away.
While running, each step is announced with a timestamp so you can see what is
currently in progress.
Parallelism
The selected work runs in parallel by default, since the units are independent. Each worker tags its log lines with its region so the interleaved output stays readable:
[12:00:01] [arctic] dropqc nrt_ar_ar
[12:00:01] [arctic] dropqc nrt_ar_gl
[12:00:01] [baltic] dropqc nrt_bo_bo
The granularity differs by script:
clean_data.shanddedup_data.shdefault to per file: one worker per merged file (a stem within a region), the finest granularity (8 files across the three regions). Each file runs its whole stage chain (e.g.dropqc → dropna → filter → report) in order. Pass--by-regionfor one worker per region instead (coarser), or--sequentialfor no parallelism.convert_data.shdefaults to per unit: one worker per (region, product), where each region’s three products are its regional NRT, the Global (GL), and CORA (9 units across the three regions). Each unit runs its own convert → merge → header → merge-header chain in order. Pass--by-regionfor one worker per region instead (its three products in order), or--sequentialfor no parallelism.compare_data.shdefaults to per comparison: one worker per (region, comparison), where each region has three comparisons (NRT vs CORA, GL vs CORA, NRT vs GL: 9 across the three regions). Pass--by-regionfor one worker per region instead (its comparisons in order), or--sequentialfor no parallelism.download_data.shparallelises per region;--sequentialdisables it.
If any unit fails, the others still finish and the script exits non-zero after
reporting which unit failed. A run with only one unit is always sequential.
Note that convert/clean/dedup each also use multiple threads within a
unit, so parallel units multiply the load accordingly, convert_data.sh’s
-t/--threads therefore defaults to just 2, since up to 9 units may run at
once. Throttle further with --by-region / --sequential. For
download_data.sh, --sequential also helps if concurrent Copernicus transfers
hit rate limits.
If memory is tight (parallel units each hold a chunk in memory at once), lower
the streaming chunk size with --chunk-rows N on convert/clean/dedup, it
trades a smaller memory footprint for more Parquet row groups without changing
the output. See Configuration for the
full list of tuning variables.
Options
| Option | Scripts | Default | Meaning |
|---|---|---|---|
-t, --threads N | convert | 2 | Worker threads for each ctddump call (kept low because up to 9 units run in parallel by default). |
-s, --src DIR | download, convert | source | Root of the source NetCDF tree (downloaded into / read from). |
-s, --src DIR | site | summary | Directory holding the <stem>.md summary pages to render. |
--src DIR | compare | output/dedup/dedup | Directory holding the input Parquet files to compare. |
-o, --out DIR | convert, clean, dedup, summary, compare | output | Root for the generated / consumed data outputs (summary reads the markdup .dups.tsv from here; compare derives its --src default from it). |
-r, --report DIR | convert, clean, dedup, summary, compare | report | Root for the summary TSV reports (a sibling of output/). |
-d, --dest DIR | summary | summary | Directory the generated summary pages are written to. |
-d, --dest DIR | site | site | Directory the built static site is written to. |
-c, --config FILE | site | built-in | Custom book.toml to use instead of the built-in template. |
-t, --title TEXT | site | ctddump: CTD data summary reports | Book title (built-in template only; ignored with --config). |
-l, --license FILE | site | the LICENSE beside the script | LICENSE copied into the built site. Pass --license "" to skip it. |
--citation FILE | site | the site-CITATION.cff beside the script | CITATION.cff copied into the built site. If it has a top-level doi:, the site README also gets a DOI badge and a Citation section. Pass --citation "" to skip it. |
-f, --format FMT | summary | md | Summary page format: md or html. |
--format FMT | compare | tsv | ctddump compare output format: tsv, text, or json (sets the report file extension). |
--no-platform-key | compare | off | Match profiles on time and position only, ignoring platform_code. |
--chunk-rows N | convert, clean, dedup, compare | ctddump default | Streaming chunk size in rows: lower uses less memory but writes/reads more row groups. Exported as CTDDUMP_CHUNK_ROWS for every ctddump process the script launches. |
--by-region | convert, clean, dedup, compare | off | Parallelise per region instead of per unit/file/comparison. |
--sequential | convert, clean, dedup, compare | off | Process one unit at a time (no parallelism). |
--time | convert, clean, dedup | off | Measure each ctddump step with GNU time and log its wall clock and peak memory (see Timing steps). |
--time-log FILE | convert, clean, dedup | off | Write the --time measurements to FILE (implies --time), so the screen keeps only the normal progress output. |
-y, --yes | all | off | Skip the confirmation prompt. |
-h, --help | all | off | Show the script’s help. |
Timing steps
convert_data.sh, clean_data.sh, and dedup_data.sh accept --time (off by
default). When on, every ctddump step is wrapped in GNU
time and its wall-clock seconds and peak
resident memory are logged as a timed <step>: …s, … MiB peak RSS, …% CPU line
after the step completes:
[12:00:03] dropqc nrt_ar_ar
[12:00:15] timed dropqc nrt_ar_ar: 11.72s, 512 MiB peak RSS, 148% CPU
Notes:
- It requires GNU time (the
timepackage:sudo apt-get install time), not the shelltimebuiltin, which cannot report memory. The scripts resolve/usr/bin/time(orgtime), and check it up front so a missing tool fails fast rather than mid-run. Override the binary with theCTDDUMP_TIME_BINenvironment variable. - Peak RSS is measured per
ctddumpprocess. A multi-step stage such as the multi-boxfilterlogs one line per underlying process (… (include),… (exclude 1), …). - Under the default parallelism, wall-clock times of concurrent steps overlap and
CPU% can exceed 100% (each
ctddumpis itself multi-threaded). For clean, comparable per-step wall times, pair--timewith--sequential; peak RSS is per process and meaningful either way. - The measurements share the log stream (stderr) with the normal progress, so a
plain redirect captures both (
... --time --sequential -y all 2> run.log). To keep them apart, pass--time-log FILE: thetimed …lines go to FILE (which implies--timeand is truncated fresh each run) while the screen keeps only the normal progress. In parallel mode each worker appends its own lines, so the file holds every step’s timing, ordered by completion.
Missing datasets
Every stage tolerates missing inputs, so an unavailable dataset does not fail the
run. A ctddump batch or concat step that matches no files reports this and
writes nothing; clean_data.sh / dedup_data.sh skip any file whose input is
missing (with a note). The concrete case today is the Global (GL) product for
the Baltic Sea, which Copernicus does not yet publish: the nrt_bo_gl outputs
are simply skipped, and appear automatically once the data becomes available.
download_data.sh
Downloads the source NetCDF from Copernicus Marine. Commands: login,
download (default). Needs the copernicusmarine toolbox (not ctddump).
scripts/download_data.sh login # one-time Copernicus login
scripts/download_data.sh # download every region into source/
scripts/download_data.sh download arctic # just the Arctic
Each product’s directory is downloaded under source/ (override with
-s/--src), ready for convert_data.sh.
convert_data.sh
Converts the downloaded NetCDF to Parquet and metadata. Commands: process,
report, all (default). The default runs process and then report, so
the summaries in report/ always match the data just converted; process alone
leaves any earlier reports in place unchanged.
scripts/convert_data.sh # process + report, all regions
scripts/convert_data.sh process arctic # just convert + merge the Arctic
scripts/convert_data.sh report # rewrite the reports from the merged outputs
Reads the source NetCDF from source/ (-s/--src); writes merged Parquet to
output/convert/, merged headers to output/header/, and summaries to
report/convert/ (Parquet data) and report/header/ (header YAML).
The merge runs concat convert with its defaults, so the merged Parquet has
each profile’s observations sorted by increasing pressure, and observations
without a pressure value are dropped. Every later phase works on this order.
See Row order and numbering.
Because the merge hides the source order, the report command also runs
checkpres on the per-file Parquet each merge was
built from (output/convert/<region>/<product>/, kept after the merge) and
writes report/convert/checkpres/<stem>.profiles.tsv, .platforms.tsv and
.summary.tsv: the profiles whose pressure does not increase in the source
order. The summary page shows them in its Pressure order section.
clean_data.sh
Cleans the merged Parquet from phase 1. Commands: dropqc, dropna, filter,
report, all (default). The stages chain
output/convert → clean/dropqc → clean/dropna → clean/filter, with a summary of
each stage in report/clean/{dropqc,dropna,filter}/. The filter step applies
each region’s bounding box(es).
scripts/clean_data.sh # dropqc → dropna → filter → report, all regions
scripts/clean_data.sh -y all baltic # skip the prompt, Baltic only
dedup_data.sh
Removes duplicate profiles from the cleaned data. Commands: markdup, report,
dedup, all (default). It reads output/clean/filter, writes marked data to
output/dedup/markdup/ (plus a duplicates TSV) and de-duplicated data to
output/dedup/dedup/, with summaries under report/dedup/{markdup,dedup}/.
scripts/dedup_data.sh # markdup → report → dedup → report, all regions
compare_data.sh
Cross-compares each region’s de-duplicated products, so the summary pages (and
you) can see how far the products of the same waters overlap. It reads phase 4’s
output from output/dedup/dedup (--src to point elsewhere) and writes a
compare two-way coverage summary per pair to
report/compare/. Three comparisons run per region, selected by command:
nrt-cora: regional NRT (AR/BO/MO) vs CORA.gl-cora: NRT Global (GL) subset vs CORA.nrt-gl: regional NRT vs NRT Global.all(default): all three.
Profiles are matched on ctddump compare’s default key (platform code, the time
as a date, and longitude/latitude to 3 decimals); pass --no-platform-key to
match on time and position alone. Choose the output with --format tsv|text|json
(the report file extension follows). Like the other pipeline scripts it runs the
comparisons in parallel (one worker each), skips a comparison whose input is
missing (so a region’s absent Global product does not fail the run), and honours
--by-region / --sequential / -y.
scripts/compare_data.sh # all three comparisons, every region
scripts/compare_data.sh nrt-cora arctic # just NRT vs CORA for the Arctic
scripts/compare_data.sh --format text -y all # aligned tables, no prompt
The next phase, summary_data.sh, reads these TSVs: each
page’s Comparison section shows the rows whose reference is that page’s own
product. Keep the default --format tsv for that; text and json are for
reading the coverage directly.
summary_data.sh
Builds one report summary page per
(region, product) unit from the TSV reports the earlier phases produced:
Markdown by default, --format html for HTML. It reads reports from report/
(-r/--report) and the markdup .dups.tsv from output/ (-o/--out), and
writes one page per stem to summary/ (-d/--dest). A stem with no report files
(e.g. the not-yet-published Baltic GL product) is skipped, not an error.
Each page includes a Comparison section built from the compare_data.sh
output under report/compare/: the rows whose reference is this page’s own
product, showing how far each other product of the same region covers it. Run
compare_data.sh before this phase to populate it; without those files the
section is simply omitted.
scripts/summary_data.sh # summary/<stem>.md for every unit
scripts/summary_data.sh -y -f html baltic # HTML pages, Baltic only, no prompt
The script gives each page a human-readable title and any product-specific
notes (passed to report summary as --title / --note). Both live in the
Page text section near the top of the script, title_for and notes_for, one
case arm per stem, and that is the place to edit what a page says about a
region or dataset. The section prose on the page is generic and comes from
ctddump itself.
summary_site.sh
Renders the Markdown pages from summary_data.sh into a static local web
site with mdBook. Pages are read from
summary/ (-s/--src) and the built site is written to site/ (-d/--dest);
open site/index.html in a browser, or serve the directory as-is.
Chapters are grouped into one part per region, in the same order the other
scripts use. Each chapter’s name comes from the page’s own top-level heading (the
title summary_data.sh gave it), so titles are defined in exactly one place. A
region whose pages are absent is skipped rather than left as an empty part; if no
pages are found at all, that is an error rather than an empty site.
The book is assembled in a temporary directory, so only the rendered site is left
behind. This phase needs the Markdown pages, so run summary_data.sh with its
default -f md (an HTML run has nothing for mdBook to render).
The built directory is made self-contained and ready to publish: mdBook writes a
.nojekyll (so GitHub Pages serves its FontAwesome/ and other asset folders),
and the script adds a LICENSE (the project’s own by default, -l/--license FILE
to override or --license "" to skip), the site’s CITATION.cff (see below), and a
short README.md describing the site. This is what makes
the output directory suitable to push straight to a publishing repository such as
ctddump-report-example.
mdBook empties the site directory before every build, so anything added to it by
hand, such as a CITATION.cff or an edited README, is gone after the next run.
The site’s citation therefore lives in this repository, as
scripts/site-CITATION.cff, and every build copies it in as CITATION.cff and
regenerates the README’s DOI badge from its doi:. It describes the published
example site (ctddump-report-example, with that site’s Zenodo DOI), so edit it
there when the site’s metadata changes. If you publish a site of your own, pass
your own file with --citation FILE, or --citation "" to leave it out.
A live example of a built site is published at https://aiqc-hub.github.io/ctddump-report-example/ (a static, point-in-time sample of this phase’s output).
scripts/summary_site.sh # site/ from summary/, every region
scripts/summary_site.sh -y arctic # Arctic pages only, no prompt
scripts/summary_site.sh -t "Arctic CTD QC" # override the book title
scripts/summary_site.sh --citation my-CITATION.cff # your own site's citation and DOI badge
By default the script writes a book.toml for you. Pass -c/--config FILE to use
your own instead; it is used verbatim, so it must keep mdBook’s default
src = "src", the script assembles the chapters into a src/ directory beside
it. With --config, --title is ignored (your file sets the title).
scripts/summary_site.sh -c my-book.toml -y all
Full pipeline
Run the seven phases in order (skipping prompts) for every region. Log in once
first if you have not already (scripts/download_data.sh login):
scripts/download_data.sh -y # download every region into source/
scripts/convert_data.sh -y all
scripts/clean_data.sh -y all
scripts/dedup_data.sh -y all
scripts/compare_data.sh -y all # report/compare/<code>_<a>_vs_<b>.tsv
scripts/summary_data.sh -y all # folds the comparison into each page
scripts/summary_site.sh -y all # site/index.html
For the equivalent commands spelled out step by step, see the Regional workflows.
Arctic Sea
An end-to-end workflow for the Arctic Sea in two phases: data preparation (download, convert, merge, and export the metadata) and data cleaning (drop low-quality profiles and restrict the data to the region).
Data preparation
Prerequisites
Downloading requires a free Copernicus Marine
account and the Copernicus Marine Toolbox
(documentation),
which provides the copernicusmarine command used below.
Run everything from a working directory (e.g. ctddump). Downloads land under
source/; ctddump writes data products under output/ and summary
reports under report/ (all created as needed). Create source, enter it, and
log in once:
mkdir source
cd source
copernicusmarine login
1. Download the data
# NRT: Arctic (AR) and Global (GL)
copernicusmarine get -i cmems_obs-ins_arc_phybgcwav_mynrt_na_irr --dataset-part "history" --filter "*/CT/*"
# CORA: Arctic
copernicusmarine get -i cmems_obs-ins_glo_phy-temp-sal_my_cora_irr --filter "arctic/*/*_PR_CT.nc"
# Back to the working root; the steps below use source/, output/, and report/ relative to it.
cd ..
2. Convert NetCDF to Parquet
# NRT AR
ctddump batch convert nrt_ar --threads 10 --output output/convert/ar/ar source
# NRT GL
ctddump batch convert nrt_gl --threads 10 --output output/convert/ar/gl source/INSITU_ARC_PHYBGCWAV_DISCRETE_MYNRT_013_031
# CORA AR
ctddump batch convert cora --threads 10 --output output/convert/ar/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/arctic
3. Merge the Parquet files
First list the profiles whose pressure does not increase in the source order,
since the merge sorts by pressure and hides it (see
checkpres):
# NRT AR
ctddump checkpres --threads 10 output/convert/ar/ar report/convert/checkpres/nrt_ar_ar
# NRT GL
ctddump checkpres --threads 10 output/convert/ar/gl report/convert/checkpres/nrt_ar_gl
# CORA AR
ctddump checkpres --threads 10 output/convert/ar/cora report/convert/checkpres/cora_ar
Then merge:
# NRT AR
ctddump concat convert --threads 10 output/convert/ar/ar output/convert/nrt_ar_ar.parquet
# NRT GL
ctddump concat convert --threads 10 output/convert/ar/gl output/convert/nrt_ar_gl.parquet
# CORA AR
ctddump concat convert --threads 10 output/convert/ar/cora output/convert/cora_ar.parquet
The merge sorts each profile’s observations by increasing pressure and drops
observations without a pressure value; add --no-pres-sort to keep the source
order (see Row order and numbering).
4. Export the metadata (headers)
# NRT AR
ctddump batch header nrt --threads 10 --pattern "AR_PR_CT_*.nc" --output output/header/ar/ar source/INSITU_ARC_PHYBGCWAV_DISCRETE_MYNRT_013_031
# NRT GL
ctddump batch header nrt --threads 10 --pattern "GL_PR_CT_*.nc" --output output/header/ar/gl source/INSITU_ARC_PHYBGCWAV_DISCRETE_MYNRT_013_031
# CORA AR
ctddump batch header cora --threads 10 --output output/header/ar/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/arctic
5. Merge the header files
# NRT AR
ctddump concat header output/header/ar/ar output/header/nrt_ar_ar.yaml
# NRT GL
ctddump concat header output/header/ar/gl output/header/nrt_ar_gl.yaml
# CORA AR
ctddump concat header output/header/ar/cora output/header/cora_ar.yaml
6. Summarise the results
Write a platform-level summary of each merged Parquet file and a per-file summary of each merged header YAML (as TSV).
mkdir -p report/convert report/header
# NRT AR
ctddump report parquet --level platform output/convert/nrt_ar_ar.parquet report/convert/nrt_ar_ar.parquet.tsv
ctddump report yaml output/header/nrt_ar_ar.yaml report/header/nrt_ar_ar.yaml.tsv
# NRT GL
ctddump report parquet --level platform output/convert/nrt_ar_gl.parquet report/convert/nrt_ar_gl.parquet.tsv
ctddump report yaml output/header/nrt_ar_gl.yaml report/header/nrt_ar_gl.yaml.tsv
# CORA AR
ctddump report parquet --level platform output/convert/cora_ar.parquet report/convert/cora_ar.parquet.tsv
ctddump report yaml output/header/cora_ar.yaml report/header/cora_ar.yaml.tsv
Data cleaning
Clean the merged Parquet from the preparation phase by dropping low-quality
profiles and restricting the data to the region. Each step reads the previous
step’s output, so the stages chain dropqc → dropna → filter.
Create the output directories:
mkdir -p output/clean/dropqc output/clean/dropna output/clean/filter \
report/clean/dropqc report/clean/dropna report/clean/filter
1. Drop profiles with bad profile-level QC
Drop profiles whose time_qc or position_qc is a present, non-OK flag;
profiles that are OK ("1") or have missing QC are kept.
# NRT AR
ctddump dropqc output/convert/nrt_ar_ar.parquet output/clean/dropqc/nrt_ar_ar.parquet
# NRT GL
ctddump dropqc output/convert/nrt_ar_gl.parquet output/clean/dropqc/nrt_ar_gl.parquet
# CORA AR
ctddump dropqc output/convert/cora_ar.parquet output/clean/dropqc/cora_ar.parquet
2. Drop profiles with no usable data
Drop profiles that are entirely NA in any of temp, psal, or pres.
# NRT AR
ctddump dropna output/clean/dropqc/nrt_ar_ar.parquet output/clean/dropna/nrt_ar_ar.parquet
# NRT GL
ctddump dropna output/clean/dropqc/nrt_ar_gl.parquet output/clean/dropna/nrt_ar_gl.parquet
# CORA AR
ctddump dropna output/clean/dropqc/cora_ar.parquet output/clean/dropna/cora_ar.parquet
3. Filter to the Arctic region
Keep only profiles inside the Arctic bounding box (longitude -180 to 180, latitude 60 to 90).
# NRT AR
ctddump filter --min-lon -180 --max-lon 180 --min-lat 60 --max-lat 90 output/clean/dropna/nrt_ar_ar.parquet output/clean/filter/nrt_ar_ar.parquet
# NRT GL
ctddump filter --min-lon -180 --max-lon 180 --min-lat 60 --max-lat 90 output/clean/dropna/nrt_ar_gl.parquet output/clean/filter/nrt_ar_gl.parquet
# CORA AR
ctddump filter --min-lon -180 --max-lon 180 --min-lat 60 --max-lat 90 output/clean/dropna/cora_ar.parquet output/clean/filter/cora_ar.parquet
4. Summarise the cleaned data
Summarise each cleaning stage (as TSV), mirroring the data layout under
report/clean/.
# after dropqc
ctddump report parquet --level platform output/clean/dropqc/nrt_ar_ar.parquet report/clean/dropqc/nrt_ar_ar.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/nrt_ar_gl.parquet report/clean/dropqc/nrt_ar_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/cora_ar.parquet report/clean/dropqc/cora_ar.parquet.tsv
# after dropna
ctddump report parquet --level platform output/clean/dropna/nrt_ar_ar.parquet report/clean/dropna/nrt_ar_ar.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/nrt_ar_gl.parquet report/clean/dropna/nrt_ar_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/cora_ar.parquet report/clean/dropna/cora_ar.parquet.tsv
# after filter
ctddump report parquet --level platform output/clean/filter/nrt_ar_ar.parquet report/clean/filter/nrt_ar_ar.parquet.tsv
ctddump report parquet --level platform output/clean/filter/nrt_ar_gl.parquet report/clean/filter/nrt_ar_gl.parquet.tsv
ctddump report parquet --level platform output/clean/filter/cora_ar.parquet report/clean/filter/cora_ar.parquet.tsv
Data de-duplication
De-duplicate the cleaned Parquet from the previous phase. Two profiles are
duplicates when they share the same date and position (longitude/latitude rounded
to 3 decimals), ctddump’s defaults, across platforms. markdup flags them (and
lists them in a TSV); dedup removes them, keeping the profile with the most
observations.
Create the output directories:
mkdir -p output/dedup/markdup output/dedup/dedup report/dedup/markdup report/dedup/dedup
1. Mark duplicate profiles
# NRT AR
ctddump markdup output/clean/filter/nrt_ar_ar.parquet output/dedup/markdup/nrt_ar_ar.parquet output/dedup/markdup/nrt_ar_ar.dups.tsv
# NRT GL
ctddump markdup output/clean/filter/nrt_ar_gl.parquet output/dedup/markdup/nrt_ar_gl.parquet output/dedup/markdup/nrt_ar_gl.dups.tsv
# CORA AR
ctddump markdup output/clean/filter/cora_ar.parquet output/dedup/markdup/cora_ar.parquet output/dedup/markdup/cora_ar.dups.tsv
2. Summarise the marked data (duplicate counts)
# NRT AR
ctddump report parquet --level platform output/dedup/markdup/nrt_ar_ar.parquet report/dedup/markdup/nrt_ar_ar.parquet.tsv
# NRT GL
ctddump report parquet --level platform output/dedup/markdup/nrt_ar_gl.parquet report/dedup/markdup/nrt_ar_gl.parquet.tsv
# CORA AR
ctddump report parquet --level platform output/dedup/markdup/cora_ar.parquet report/dedup/markdup/cora_ar.parquet.tsv
3. Remove duplicate profiles
# NRT AR
ctddump dedup output/dedup/markdup/nrt_ar_ar.parquet output/dedup/dedup/nrt_ar_ar.parquet
# NRT GL
ctddump dedup output/dedup/markdup/nrt_ar_gl.parquet output/dedup/dedup/nrt_ar_gl.parquet
# CORA AR
ctddump dedup output/dedup/markdup/cora_ar.parquet output/dedup/dedup/cora_ar.parquet
4. Summarise the de-duplicated data
# NRT AR
ctddump report parquet --level platform output/dedup/dedup/nrt_ar_ar.parquet report/dedup/dedup/nrt_ar_ar.parquet.tsv
# NRT GL
ctddump report parquet --level platform output/dedup/dedup/nrt_ar_gl.parquet report/dedup/dedup/nrt_ar_gl.parquet.tsv
# CORA AR
ctddump report parquet --level platform output/dedup/dedup/cora_ar.parquet report/dedup/dedup/cora_ar.parquet.tsv
The pipeline is automated by
scripts/download_data.sh,scripts/convert_data.sh,scripts/clean_data.sh, andscripts/dedup_data.sh. See Helper scripts for their commands and options.
Baltic Sea
An end-to-end workflow for the Baltic Sea in two phases: data preparation (download, convert, merge, and export the metadata) and data cleaning (drop low-quality profiles and restrict the data to the region).
This workflow uses the regional NRT (BO), Global (GL), and CORA products. Copernicus does not yet publish the Global (GL) data for the Baltic, so the GL steps currently match no files,
ctddumpreports this and writes nothing, and the cleaning / de-duplication steps skip the missingnrt_bo_gloutputs, activating automatically once GL becomes available. The manual commands below therefore cover BO and CORA; the helper scripts additionally run the (currently empty) GL steps.
Data preparation
Prerequisites
Downloading requires a free Copernicus Marine
account and the Copernicus Marine Toolbox
(documentation),
which provides the copernicusmarine command used below.
Run everything from a working directory (e.g. ctddump). Downloads land under
source/; ctddump writes data products under output/ and summary
reports under report/ (all created as needed). Create source, enter it, and
log in once:
mkdir source
cd source
copernicusmarine login
1. Download the data
# NRT: Baltic (BO)
copernicusmarine get -i cmems_obs-ins_bal_phybgcwav_mynrt_na_irr --dataset-part "history" --filter "*/CT/*"
# CORA: Baltic
copernicusmarine get -i cmems_obs-ins_glo_phy-temp-sal_my_cora_irr --filter "baltic/*/*_PR_CT.nc"
# Back to the working root; the steps below use source/, output/, and report/ relative to it.
cd ..
2. Convert NetCDF to Parquet
# NRT BO
ctddump batch convert nrt_bo --threads 10 --output output/convert/bo/bo source
# CORA BO
ctddump batch convert cora --threads 10 --output output/convert/bo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/baltic
3. Merge the Parquet files
First list the profiles whose pressure does not increase in the source order,
since the merge sorts by pressure and hides it (see
checkpres):
# NRT BO
ctddump checkpres --threads 10 output/convert/bo/bo report/convert/checkpres/nrt_bo_bo
# CORA BO
ctddump checkpres --threads 10 output/convert/bo/cora report/convert/checkpres/cora_bo
Then merge:
# NRT BO
ctddump concat convert --threads 10 output/convert/bo/bo output/convert/nrt_bo_bo.parquet
# CORA BO
ctddump concat convert --threads 10 output/convert/bo/cora output/convert/cora_bo.parquet
The merge sorts each profile’s observations by increasing pressure and drops
observations without a pressure value; add --no-pres-sort to keep the source
order (see Row order and numbering).
4. Export the metadata (headers)
# NRT BO
ctddump batch header nrt --threads 10 --pattern "BO_PR_CT_*.nc" --output output/header/bo/bo source/INSITU_BAL_PHYBGCWAV_DISCRETE_MYNRT_013_032
# CORA BO
ctddump batch header cora --threads 10 --output output/header/bo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/baltic
5. Merge the header files
# NRT BO
ctddump concat header output/header/bo/bo output/header/nrt_bo_bo.yaml
# CORA BO
ctddump concat header output/header/bo/cora output/header/cora_bo.yaml
6. Summarise the results
Write a platform-level summary of each merged Parquet file and a per-file summary of each merged header YAML (as TSV).
mkdir -p report/convert report/header
# NRT BO
ctddump report parquet --level platform output/convert/nrt_bo_bo.parquet report/convert/nrt_bo_bo.parquet.tsv
ctddump report yaml output/header/nrt_bo_bo.yaml report/header/nrt_bo_bo.yaml.tsv
# CORA BO
ctddump report parquet --level platform output/convert/cora_bo.parquet report/convert/cora_bo.parquet.tsv
ctddump report yaml output/header/cora_bo.yaml report/header/cora_bo.yaml.tsv
Data cleaning
Clean the merged Parquet from the preparation phase by dropping low-quality
profiles and restricting the data to the region. Each step reads the previous
step’s output, so the stages chain dropqc → dropna → filter.
Create the output directories:
mkdir -p output/clean/dropqc output/clean/dropna output/clean/filter \
report/clean/dropqc report/clean/dropna report/clean/filter
1. Drop profiles with bad profile-level QC
Drop profiles whose time_qc or position_qc is a present, non-OK flag;
profiles that are OK ("1") or have missing QC are kept.
# NRT BO
ctddump dropqc output/convert/nrt_bo_bo.parquet output/clean/dropqc/nrt_bo_bo.parquet
# CORA BO
ctddump dropqc output/convert/cora_bo.parquet output/clean/dropqc/cora_bo.parquet
2. Drop profiles with no usable data
Drop profiles that are entirely NA in any of temp, psal, or pres.
# NRT BO
ctddump dropna output/clean/dropqc/nrt_bo_bo.parquet output/clean/dropna/nrt_bo_bo.parquet
# CORA BO
ctddump dropna output/clean/dropqc/cora_bo.parquet output/clean/dropna/cora_bo.parquet
3. Filter to the Baltic region
Keep profiles inside the Baltic bounding box (longitude 6 to 30, latitude 53 to
66), then exclude the sub-box (longitude 6 to 15, latitude 60 to 66). The include
step writes an intermediate .box.parquet file that the exclude step consumes to
produce the final cleaned file.
# NRT BO
ctddump filter --min-lon 6 --max-lon 30 --min-lat 53 --max-lat 66 output/clean/dropna/nrt_bo_bo.parquet output/clean/filter/nrt_bo_bo.box.parquet
ctddump filter --mode exclude --min-lon 6 --max-lon 15 --min-lat 60 --max-lat 66 output/clean/filter/nrt_bo_bo.box.parquet output/clean/filter/nrt_bo_bo.parquet
# CORA BO
ctddump filter --min-lon 6 --max-lon 30 --min-lat 53 --max-lat 66 output/clean/dropna/cora_bo.parquet output/clean/filter/cora_bo.box.parquet
ctddump filter --mode exclude --min-lon 6 --max-lon 15 --min-lat 60 --max-lat 66 output/clean/filter/cora_bo.box.parquet output/clean/filter/cora_bo.parquet
4. Summarise the cleaned data
Summarise each cleaning stage (as TSV), mirroring the data layout under
report/clean/.
# after dropqc
ctddump report parquet --level platform output/clean/dropqc/nrt_bo_bo.parquet report/clean/dropqc/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/cora_bo.parquet report/clean/dropqc/cora_bo.parquet.tsv
# after dropna
ctddump report parquet --level platform output/clean/dropna/nrt_bo_bo.parquet report/clean/dropna/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/cora_bo.parquet report/clean/dropna/cora_bo.parquet.tsv
# after filter
ctddump report parquet --level platform output/clean/filter/nrt_bo_bo.parquet report/clean/filter/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/filter/cora_bo.parquet report/clean/filter/cora_bo.parquet.tsv
Data de-duplication
De-duplicate the cleaned Parquet from the previous phase. Two profiles are
duplicates when they share the same date and position (longitude/latitude rounded
to 3 decimals), ctddump’s defaults, across platforms. markdup flags them (and
lists them in a TSV); dedup removes them, keeping the profile with the most
observations.
Create the output directories:
mkdir -p output/dedup/markdup output/dedup/dedup report/dedup/markdup report/dedup/dedup
1. Mark duplicate profiles
# NRT BO
ctddump markdup output/clean/filter/nrt_bo_bo.parquet output/dedup/markdup/nrt_bo_bo.parquet output/dedup/markdup/nrt_bo_bo.dups.tsv
# CORA BO
ctddump markdup output/clean/filter/cora_bo.parquet output/dedup/markdup/cora_bo.parquet output/dedup/markdup/cora_bo.dups.tsv
2. Summarise the marked data (duplicate counts)
# NRT BO
ctddump report parquet --level platform output/dedup/markdup/nrt_bo_bo.parquet report/dedup/markdup/nrt_bo_bo.parquet.tsv
# CORA BO
ctddump report parquet --level platform output/dedup/markdup/cora_bo.parquet report/dedup/markdup/cora_bo.parquet.tsv
3. Remove duplicate profiles
# NRT BO
ctddump dedup output/dedup/markdup/nrt_bo_bo.parquet output/dedup/dedup/nrt_bo_bo.parquet
# CORA BO
ctddump dedup output/dedup/markdup/cora_bo.parquet output/dedup/dedup/cora_bo.parquet
4. Summarise the de-duplicated data
# NRT BO
ctddump report parquet --level platform output/dedup/dedup/nrt_bo_bo.parquet report/dedup/dedup/nrt_bo_bo.parquet.tsv
# CORA BO
ctddump report parquet --level platform output/dedup/dedup/cora_bo.parquet report/dedup/dedup/cora_bo.parquet.tsv
The pipeline is automated by
scripts/download_data.sh,scripts/convert_data.sh,scripts/clean_data.sh, andscripts/dedup_data.sh. See Helper scripts for their commands and options.
Mediterranean Sea
An end-to-end workflow for the Mediterranean Sea in two phases: data preparation (download, convert, merge, and export the metadata) and data cleaning (drop low-quality profiles and restrict the data to the region).
Data preparation
Prerequisites
Downloading requires a free Copernicus Marine
account and the Copernicus Marine Toolbox
(documentation),
which provides the copernicusmarine command used below.
Run everything from a working directory (e.g. ctddump). Downloads land under
source/; ctddump writes data products under output/ and summary
reports under report/ (all created as needed). Create source, enter it, and
log in once:
mkdir source
cd source
copernicusmarine login
1. Download the data
# NRT: Mediterranean (MO) and Global (GL)
copernicusmarine get -i cmems_obs-ins_med_phybgcwav_mynrt_na_irr --dataset-part "history" --filter "*/CT/*"
# CORA: Mediterranean
copernicusmarine get -i cmems_obs-ins_glo_phy-temp-sal_my_cora_irr --filter "mediterrane/*/*_PR_CT.nc"
# Back to the working root; the steps below use source/, output/, and report/ relative to it.
cd ..
2. Convert NetCDF to Parquet
# NRT MO
ctddump batch convert nrt_mo --threads 10 --output output/convert/mo/mo source
# NRT GL
ctddump batch convert nrt_gl --threads 10 --output output/convert/mo/gl source/INSITU_MED_PHYBGCWAV_DISCRETE_MYNRT_013_035
# CORA MO
ctddump batch convert cora --threads 10 --output output/convert/mo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/mediterrane
3. Merge the Parquet files
First list the profiles whose pressure does not increase in the source order,
since the merge sorts by pressure and hides it (see
checkpres):
# NRT MO
ctddump checkpres --threads 10 output/convert/mo/mo report/convert/checkpres/nrt_mo_mo
# NRT GL
ctddump checkpres --threads 10 output/convert/mo/gl report/convert/checkpres/nrt_mo_gl
# CORA MO
ctddump checkpres --threads 10 output/convert/mo/cora report/convert/checkpres/cora_mo
Then merge:
# NRT MO
ctddump concat convert --threads 10 output/convert/mo/mo output/convert/nrt_mo_mo.parquet
# NRT GL
ctddump concat convert --threads 10 output/convert/mo/gl output/convert/nrt_mo_gl.parquet
# CORA MO
ctddump concat convert --threads 10 output/convert/mo/cora output/convert/cora_mo.parquet
The merge sorts each profile’s observations by increasing pressure and drops
observations without a pressure value; add --no-pres-sort to keep the source
order (see Row order and numbering).
4. Export the metadata (headers)
# NRT MO
ctddump batch header nrt --threads 10 --pattern "MO_PR_CT_*.nc" --output output/header/mo/mo source/INSITU_MED_PHYBGCWAV_DISCRETE_MYNRT_013_035
# NRT GL
ctddump batch header nrt --threads 10 --pattern "GL_PR_CT_*.nc" --output output/header/mo/gl source/INSITU_MED_PHYBGCWAV_DISCRETE_MYNRT_013_035
# CORA MO
ctddump batch header cora --threads 10 --output output/header/mo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/mediterrane
5. Merge the header files
# NRT MO
ctddump concat header output/header/mo/mo output/header/nrt_mo_mo.yaml
# NRT GL
ctddump concat header output/header/mo/gl output/header/nrt_mo_gl.yaml
# CORA MO
ctddump concat header output/header/mo/cora output/header/cora_mo.yaml
6. Summarise the results
Write a platform-level summary of each merged Parquet file and a per-file summary of each merged header YAML (as TSV).
mkdir -p report/convert report/header
# NRT MO
ctddump report parquet --level platform output/convert/nrt_mo_mo.parquet report/convert/nrt_mo_mo.parquet.tsv
ctddump report yaml output/header/nrt_mo_mo.yaml report/header/nrt_mo_mo.yaml.tsv
# NRT GL
ctddump report parquet --level platform output/convert/nrt_mo_gl.parquet report/convert/nrt_mo_gl.parquet.tsv
ctddump report yaml output/header/nrt_mo_gl.yaml report/header/nrt_mo_gl.yaml.tsv
# CORA MO
ctddump report parquet --level platform output/convert/cora_mo.parquet report/convert/cora_mo.parquet.tsv
ctddump report yaml output/header/cora_mo.yaml report/header/cora_mo.yaml.tsv
Data cleaning
Clean the merged Parquet from the preparation phase by dropping low-quality
profiles and restricting the data to the region. Each step reads the previous
step’s output, so the stages chain dropqc → dropna → filter.
Create the output directories:
mkdir -p output/clean/dropqc output/clean/dropna output/clean/filter \
report/clean/dropqc report/clean/dropna report/clean/filter
1. Drop profiles with bad profile-level QC
Drop profiles whose time_qc or position_qc is a present, non-OK flag;
profiles that are OK ("1") or have missing QC are kept.
# NRT MO
ctddump dropqc output/convert/nrt_mo_mo.parquet output/clean/dropqc/nrt_mo_mo.parquet
# NRT GL
ctddump dropqc output/convert/nrt_mo_gl.parquet output/clean/dropqc/nrt_mo_gl.parquet
# CORA MO
ctddump dropqc output/convert/cora_mo.parquet output/clean/dropqc/cora_mo.parquet
2. Drop profiles with no usable data
Drop profiles that are entirely NA in any of temp, psal, or pres.
# NRT MO
ctddump dropna output/clean/dropqc/nrt_mo_mo.parquet output/clean/dropna/nrt_mo_mo.parquet
# NRT GL
ctddump dropna output/clean/dropqc/nrt_mo_gl.parquet output/clean/dropna/nrt_mo_gl.parquet
# CORA MO
ctddump dropna output/clean/dropqc/cora_mo.parquet output/clean/dropna/cora_mo.parquet
3. Filter to the Mediterranean region
Keep profiles inside the Mediterranean bounding box (longitude -5.61 to 35.567,
latitude 28.378 to 45.755), then exclude two sub-boxes: (longitude 27 to 36,
latitude 41 to 46) and (longitude -5.61 to 0, latitude 42 to 46). Each stage
chains through an intermediate .box*.parquet file to produce the final cleaned
file.
# NRT MO
ctddump filter --min-lon -5.61 --max-lon 35.567 --min-lat 28.378 --max-lat 45.755 output/clean/dropna/nrt_mo_mo.parquet output/clean/filter/nrt_mo_mo.box1.parquet
ctddump filter --mode exclude --min-lon 27 --max-lon 36 --min-lat 41 --max-lat 46 output/clean/filter/nrt_mo_mo.box1.parquet output/clean/filter/nrt_mo_mo.box2.parquet
ctddump filter --mode exclude --min-lon -5.61 --max-lon 0 --min-lat 42 --max-lat 46 output/clean/filter/nrt_mo_mo.box2.parquet output/clean/filter/nrt_mo_mo.parquet
# NRT GL
ctddump filter --min-lon -5.61 --max-lon 35.567 --min-lat 28.378 --max-lat 45.755 output/clean/dropna/nrt_mo_gl.parquet output/clean/filter/nrt_mo_gl.box1.parquet
ctddump filter --mode exclude --min-lon 27 --max-lon 36 --min-lat 41 --max-lat 46 output/clean/filter/nrt_mo_gl.box1.parquet output/clean/filter/nrt_mo_gl.box2.parquet
ctddump filter --mode exclude --min-lon -5.61 --max-lon 0 --min-lat 42 --max-lat 46 output/clean/filter/nrt_mo_gl.box2.parquet output/clean/filter/nrt_mo_gl.parquet
# CORA MO
ctddump filter --min-lon -5.61 --max-lon 35.567 --min-lat 28.378 --max-lat 45.755 output/clean/dropna/cora_mo.parquet output/clean/filter/cora_mo.box1.parquet
ctddump filter --mode exclude --min-lon 27 --max-lon 36 --min-lat 41 --max-lat 46 output/clean/filter/cora_mo.box1.parquet output/clean/filter/cora_mo.box2.parquet
ctddump filter --mode exclude --min-lon -5.61 --max-lon 0 --min-lat 42 --max-lat 46 output/clean/filter/cora_mo.box2.parquet output/clean/filter/cora_mo.parquet
4. Summarise the cleaned data
Summarise each cleaning stage (as TSV), mirroring the data layout under
report/clean/.
# after dropqc
ctddump report parquet --level platform output/clean/dropqc/nrt_mo_mo.parquet report/clean/dropqc/nrt_mo_mo.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/nrt_mo_gl.parquet report/clean/dropqc/nrt_mo_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/cora_mo.parquet report/clean/dropqc/cora_mo.parquet.tsv
# after dropna
ctddump report parquet --level platform output/clean/dropna/nrt_mo_mo.parquet report/clean/dropna/nrt_mo_mo.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/nrt_mo_gl.parquet report/clean/dropna/nrt_mo_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/cora_mo.parquet report/clean/dropna/cora_mo.parquet.tsv
# after filter
ctddump report parquet --level platform output/clean/filter/nrt_mo_mo.parquet report/clean/filter/nrt_mo_mo.parquet.tsv
ctddump report parquet --level platform output/clean/filter/nrt_mo_gl.parquet report/clean/filter/nrt_mo_gl.parquet.tsv
ctddump report parquet --level platform output/clean/filter/cora_mo.parquet report/clean/filter/cora_mo.parquet.tsv
Data de-duplication
De-duplicate the cleaned Parquet from the previous phase. Two profiles are
duplicates when they share the same date and position (longitude/latitude rounded
to 3 decimals), ctddump’s defaults, across platforms. markdup flags them (and
lists them in a TSV); dedup removes them, keeping the profile with the most
observations.
Create the output directories:
mkdir -p output/dedup/markdup output/dedup/dedup report/dedup/markdup report/dedup/dedup
1. Mark duplicate profiles
# NRT MO
ctddump markdup output/clean/filter/nrt_mo_mo.parquet output/dedup/markdup/nrt_mo_mo.parquet output/dedup/markdup/nrt_mo_mo.dups.tsv
# NRT GL
ctddump markdup output/clean/filter/nrt_mo_gl.parquet output/dedup/markdup/nrt_mo_gl.parquet output/dedup/markdup/nrt_mo_gl.dups.tsv
# CORA MO
ctddump markdup output/clean/filter/cora_mo.parquet output/dedup/markdup/cora_mo.parquet output/dedup/markdup/cora_mo.dups.tsv
2. Summarise the marked data (duplicate counts)
# NRT MO
ctddump report parquet --level platform output/dedup/markdup/nrt_mo_mo.parquet report/dedup/markdup/nrt_mo_mo.parquet.tsv
# NRT GL
ctddump report parquet --level platform output/dedup/markdup/nrt_mo_gl.parquet report/dedup/markdup/nrt_mo_gl.parquet.tsv
# CORA MO
ctddump report parquet --level platform output/dedup/markdup/cora_mo.parquet report/dedup/markdup/cora_mo.parquet.tsv
3. Remove duplicate profiles
# NRT MO
ctddump dedup output/dedup/markdup/nrt_mo_mo.parquet output/dedup/dedup/nrt_mo_mo.parquet
# NRT GL
ctddump dedup output/dedup/markdup/nrt_mo_gl.parquet output/dedup/dedup/nrt_mo_gl.parquet
# CORA MO
ctddump dedup output/dedup/markdup/cora_mo.parquet output/dedup/dedup/cora_mo.parquet
4. Summarise the de-duplicated data
# NRT MO
ctddump report parquet --level platform output/dedup/dedup/nrt_mo_mo.parquet report/dedup/dedup/nrt_mo_mo.parquet.tsv
# NRT GL
ctddump report parquet --level platform output/dedup/dedup/nrt_mo_gl.parquet report/dedup/dedup/nrt_mo_gl.parquet.tsv
# CORA MO
ctddump report parquet --level platform output/dedup/dedup/cora_mo.parquet report/dedup/dedup/cora_mo.parquet.tsv
The pipeline is automated by
scripts/download_data.sh,scripts/convert_data.sh,scripts/clean_data.sh, andscripts/dedup_data.sh. See Helper scripts for their commands and options.