Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ctddump

ctddump is a small, fast command-line tool for converting oceanographic CTD (Conductivity, Temperature, Depth) data from NetCDF into analysis-ready formats:

  • Parquet: the observation data, as a uniform flat table.
  • YAML: the file metadata (dimensions, variables, global attributes).

It is written in Rust, streams data in bounded memory, and processes whole directory trees in parallel.

Data sources

Two families of source files are supported:

SourceDescription
NRTNear Real Time: Arctic Sea (nrt_ar), Baltic Sea (nrt_bo), Mediterranean Sea (nrt_mo), Global (nrt_gl)
CORACopernicus Ocean Reanalysis: current format (cora), legacy format (cora_legacy)
Your ownAny NetCDF laid out like the Copernicus files, described by a mapping file (generic); see Your own NetCDF files

What you can do

CommandPurpose
convertConvert a single NetCDF file to Parquet.
batchConvert a whole directory tree to Parquet or YAML, in parallel.
headerExtract a single file’s metadata to YAML.
checkpresFind profiles whose pressure does not increase in observation order.
concatMerge many Parquet (or YAML) files into one.
reportSummarise a Parquet or YAML file as a text report.
filterFilter a Parquet file’s profiles by a geographic bounding box.
dropnaDrop profiles that are entirely NA in any of temp/psal/pres.
dropqcDrop profiles flagged bad in time_qc or position_qc.
markdupMark duplicate profiles (by date + position) with an is_dup column.
dedupRemove duplicate profiles, keeping the one with the most observations.
compareReport how far each of two Parquet files covers the other’s platforms and profiles.
sqliteExport a Parquet file to a SQLite database (platform, profile, observation tables).
completionsPrint a shell tab-completion script for bash, zsh, fish, elvish, or powershell.

Quick example

# One file to Parquet
ctddump convert nrt_ar input.nc output.parquet

# A whole Arctic directory, then merge into a single Parquet file
ctddump batch convert nrt_ar --output ./parquet ./netcdf
ctddump concat convert ./parquet arctic.parquet

To see what the full pipeline produces, browse the live example report site, built with the helper scripts.

New here? Start with Installation, skim the Commands, then follow an end-to-end regional workflow.

Installation

There are three ways to install ctddump. Pick the first one that applies:

RouteNeedsBest for
Prebuilt binarynothingmost users
crates.ioRust, HDF5 headersRust users, other platforms
From sourceRust, HDF5 headers, gitcontributors

Prebuilt binary

Each release ships an archive per platform. It contains the ctddump executable with HDF5 and netCDF built in, so no Rust toolchain and no system libraries are required:

PlatformArchive
Linux, Intel/AMD 64-bitctddump-vX.Y.Z-x86_64-unknown-linux-gnu.tar.gz
Linux, ARM 64-bitctddump-vX.Y.Z-aarch64-unknown-linux-gnu.tar.gz
macOS, Apple Siliconctddump-vX.Y.Z-aarch64-apple-darwin.tar.gz
macOS, Intelctddump-vX.Y.Z-x86_64-apple-darwin.tar.gz

The Linux binaries are linked against glibc 2.17, so they run on any distribution with glibc 2.17 or newer: RHEL/CentOS 7, Rocky Linux 8, Ubuntu 18.04, Debian 9, and everything more recent.

Download, extract, and put the binary somewhere on your PATH:

VERSION=v0.37.2
TARGET=x86_64-unknown-linux-gnu

curl -LO "https://github.com/AIQC-Hub/ctddump/releases/download/$VERSION/ctddump-$VERSION-$TARGET.tar.gz"
tar -xzf "ctddump-$VERSION-$TARGET.tar.gz"
cd "ctddump-$VERSION-$TARGET"

./ctddump --version

To install it for your user, move it onto your PATH:

mkdir -p ~/.local/bin
mv ctddump ~/.local/bin/

If ctddump is still not found afterwards, ~/.local/bin is not on your PATH; add export PATH="$HOME/.local/bin:$PATH" to your shell profile.

Every archive is listed in SHA256SUMS on the same release page. To verify a download before trusting it:

curl -LO "https://github.com/AIQC-Hub/ctddump/releases/download/$VERSION/SHA256SUMS"
sha256sum -c SHA256SUMS --ignore-missing

What else is in the archive

Alongside the binary are the helper scripts, plus README.md, LICENSE, and CHANGELOG.md. Five scripts (convert_data.sh, clean_data.sh, dedup_data.sh, compare_data.sh, and summary_data.sh) need only ctddump on your PATH, so they work as soon as the step above is done. Three others call external tools that are not bundled and must be installed separately: download_data.sh needs copernicusmarine, summary_site.sh needs mdbook, and fetch_test_data.sh needs gh and unzip.

The scripts are bash, so on Windows they need WSL or Git Bash.

System dependencies

The two routes below compile ctddump themselves and link the HDF5 C library, so the development headers must be installed first. (The prebuilt binaries above need none of this.)

# Ubuntu / Debian
sudo apt-get install libhdf5-dev libnetcdf-dev

# macOS (Homebrew)
brew install hdf5

Install from crates.io

With the system dependencies in place, install the published crate:

cargo install ctddump

This builds the binary and puts it in ~/.cargo/bin, which is normally already on your PATH. Check it works:

ctddump --help

If the build stops with “A system version of libnetcdf could not be found”, the development headers are missing or are somewhere the build script does not look. Install them as shown above, or point at them explicitly:

NETCDF_DIR=/path/to/netcdf HDF5_DIR=/path/to/hdf5 cargo install ctddump

Failing that, the static-netcdf feature builds HDF5 and netCDF from source instead of looking for system ones. It is much slower but needs no headers:

cargo install ctddump --features static-netcdf

Build from source

To work on ctddump itself, or to run a version that is not published yet:

git clone https://github.com/AIQC-Hub/ctddump.git
cd ctddump
cargo build --release

The binary is placed at target/release/ctddump. Copy it somewhere on your PATH (or run it directly) and check it works:

ctddump --help

Every command and subcommand supports -h / --help, so you can always discover the available options interactively:

ctddump convert --help
ctddump batch convert nrt_ar --help

Note: On systems with HDF5 ≤ 1.10 you may see harmless HDF5-DIAG messages in the output. The data is read correctly and results are unaffected.

convert

Convert a single NetCDF file into a Parquet file.

ctddump convert <subcommand> [OPTIONS] <src_file> <target_file>

The subcommand selects the source format and its defaults:

SubcommandSource
nrt_arNRT: Arctic Sea
nrt_boNRT: Baltic Sea
nrt_moNRT: Mediterranean Sea
nrt_glNRT: Global
coraCORA: current format
cora_legacyCORA: legacy format
genericYour own NetCDF, described by a mapping file (details)

Examples

ctddump convert nrt_ar      input.nc output.parquet
ctddump convert nrt_bo      input.nc output.parquet
ctddump convert cora        input.nc output.parquet
ctddump convert cora_legacy input.nc output.parquet
ctddump convert generic --config my_map.toml input.nc output.parquet

Use a saved TOML preset, or override individual fields with flags:

# Apply a config preset
ctddump convert nrt_ar --config my_preset.toml input.nc output.parquet

# Override a single field
ctddump convert nrt_bo --no-deph-source input.nc output.parquet

See Configuration for the available flags and the TOML format, and Output schema for the columns produced.

batch

Process a whole directory tree in parallel. batch recursively finds every matching .nc file and converts each one, using all CPU cores by default.

ctddump batch convert <subcommand> [OPTIONS] <src_dir>
ctddump batch header  <subcommand> [OPTIONS] <src_dir>
  • batch convert writes one Parquet file per source file.
  • batch header writes one YAML metadata file per source file.

Each output keeps its source’s filename stem with a new extension. If --output is omitted, each result is written alongside its source; with --output <dir> all results land flat in that directory.

Examples

# Convert all NRT Arctic files to Parquet, flat into ./output
ctddump batch convert nrt_ar --output ./output ./data/arctic

# Limit to 4 threads
ctddump batch convert nrt_ar --threads 4 --output ./output ./data/arctic

# Override the filename pattern
ctddump batch convert nrt_ar --pattern "AR_PR_CT_ITP-*.nc" --output ./output ./data

# Extract YAML metadata for all NRT files
ctddump batch header nrt --output ./output ./data/arctic

# Convert your own files with a mapping file
ctddump batch convert generic --config my_map.toml --output ./output ./data/mine

Default filename patterns

--pattern matches the filename only (not the path) and supports *, ? and […]. When omitted, each subcommand uses a sensible default:

SubcommandPattern
nrt_arAR_PR_CT_*.nc
nrt_boBO_PR_CT_*.nc
nrt_moMO_PR_CT_*.nc
nrt_glGL_PR_CT_*.nc
cora, cora_legacy, generic*.nc
batch header nrt, batch header cora, batch header generic*.nc

Threads. --threads N caps the worker count; omit it to use all logical cores. Memory scales with the thread count, so lower it if you are processing very large files on a memory-constrained machine.

header

Extract the metadata of a single NetCDF file to a YAML file, dimensions, variables, and global attributes. No observation data is read.

ctddump header <subcommand> <src_file> <target_file>
SubcommandSource
nrtNRT files
coraCORA files
genericAny NetCDF file; keeps all global attributes

Examples

ctddump header nrt  input.nc output.yaml
ctddump header cora input.nc output.yaml
ctddump header generic input.nc output.yaml

To extract metadata for a whole directory at once, use batch header; to combine many YAML files into one, use concat header.

checkpres

Find profiles whose pressure does not increase in observation order, and write three TSV reports: the profiles, their platforms, and totals.

ctddump checkpres [OPTIONS] <src> <out_prefix>

<src> is a directory of Parquet files, read the same way as by concat convert, or a single Parquet file.

When to run it

Run it on the per-file Parquet written by batch convert, before concat convert merges them. The merge sorts every profile by increasing pressure (see Row order and numbering), so afterwards the original order can no longer be checked. Before the merge, rows are still in the source file’s order and observation_no is the level index in the file.

Run it once per merge, on the same directory, so the reports cover exactly what ends up in one merged file:

ctddump checkpres output/convert/ar/ar report/convert/checkpres/nrt_ar_ar
ctddump concat convert output/convert/ar/ar output/convert/nrt_ar_ar.parquet

What is checked

  • A profile is one (filename, platform_code, profile_no). A file may hold many profiles and many platforms; each profile is checked on its own.
  • Within a profile, each observation that has a pressure is compared with the previous observation that had one. Missing pressures are skipped.
  • A decrease is a step to a lower pressure. A repeated pressure is not a decrease.
  • A profile with no usable pressure at all has nothing to check. It is left out of the counts and reported only as n_no_pres, so the counts match the profiles concat convert keeps (it drops observations without pressure).
  • A profile with at least one decrease is reported, as one of two kinds:
KindMeaning
reversedPressure never goes up: the profile is stored from the bottom up. Sorting by pressure turns it the right way round.
non_monotonicPressure goes both up and down: loops, spikes, or levels stored out of order. These are the profiles to look at.

Options

OptionDefaultMeaning
--pattern <GLOB>*.parquetFilename glob used when <src> is a directory
--min-drop <DBAR>0Ignore decreases no larger than this, for sensor noise
--pres-qc <FLAGS>every pressureUse only pressures whose pres_qc is one of these flags, e.g. 1,2
-t, --threads <N>all coresNumber of files checked in parallel; the reports are the same for any value

Outputs

Three tab-separated files, named after <out_prefix>:

  1. <out_prefix>.profiles.tsv: one row per reported profile, sorted by platform and time.

    ColumnMeaning
    filename, platform_code, profile_nothe profile, as in the per-file Parquet
    profile_timestamp, profile_time, longitude, latitudewhen and where; with platform_code, these also find the profile after the merge
    kindreversed or non_monotonic
    n_obs, n_presobservations, and observations with a pressure used by the check
    n_decreasesnumber of decreasing steps
    first_decrease_obsobservation_no where the pressure first drops
    max_decreaselargest drop, in dbar
    pres_first, pres_last, pres_min, pres_maxpressure range, in dbar
    pres_conv1 when some pressure in the profile was derived from depth
  2. <out_prefix>.platforms.tsv: one row per platform with at least one reported profile: platform_code, n_files, n_profiles, n_bad_profiles, pct_bad, n_reversed, n_non_monotonic, first_date, last_date. The dates are those of its first and last reported profile.

  3. <out_prefix>.summary.tsv: one row for the whole source: n_files, n_platforms, n_profiles, n_no_pres, n_bad_profiles, pct_bad, n_reversed, n_non_monotonic, n_bad_platforms, n_bad_files. n_profiles and n_platforms count only what was checked; n_no_pres is the number of profiles without any usable pressure.

The output directory is created if needed. A one-line summary is printed to standard error. Finding problems is not an error: the exit status is 0. When no file matches, or the directory does not exist, the command says so and writes nothing.

Examples

# Check a directory of converted files
ctddump checkpres ./parquet reports/arctic

# Ignore drops of 0.1 dbar or less, and pressures not flagged good
ctddump checkpres --min-drop 0.1 --pres-qc 1,2 ./parquet reports/arctic

# Check one file
ctddump checkpres AR_PR_CT_58KN.parquet reports/58kn

Files read one at a time, in fixed row windows (CTDDUMP_CHUNK_ROWS), so memory follows the size of one file plus the reported profiles.

concat

Merge many files from a directory tree into a single file.

ctddump concat convert [OPTIONS] <src_dir> <output_file>
ctddump concat header  [OPTIONS] <src_dir> <output_file>
  • concat convert merges Parquet data files.
  • concat header merges YAML metadata files.

concat convert

By default the merge re-assigns profile_no and observation_no so that profile numbers are unique and sequential within each platform, and sorts each profile’s observations by increasing pressure. Its behaviour is controlled by a few flags:

FlagEffect
(default)Renumber; sort by platform_code, profile_timestamp, longitude, latitude, pres; drop rows with missing pres; use all cores.
--no-renumberMerge as-is, without re-assigning profile_no / observation_no.
--no-pres-sortSort without pres, keeping each profile’s observations in their original source order instead of reordering by pressure.
--keep-na-presKeep rows whose pres is null/NaN (dropped by default).
--threads NCap the worker count (default: all cores). --threads 1 is the sequential, lowest-memory path.
--pattern <GLOB>Merge only files whose name matches the pattern.

Renumbering processes platform ranges in parallel via temporary files in the output folder. Higher thread counts are faster but use more memory, the merged result is identical either way.

Sorting by pressure

The files written by convert keep the order of the source NetCDF. The default merge reorders them: within each profile, observations run from the lowest to the highest pres, and observation_no is renumbered 1..N in that order. Every other column moves with its row.

  • Rows with missing pres are dropped before sorting. With --keep-na-pres they are kept and placed at the end of their profile.
  • The sort is stable: observations with the same pressure keep their source order (by file path, then by position in the file).
  • Profiles are identified by (platform_code, profile_timestamp, longitude, latitude). Two profiles from different files with the same four values become one profile_no, and their observations are interleaved by pressure (filename still tells them apart).
  • --no-pres-sort keeps each profile’s observations in their source order; --no-renumber does not sort at all.

See Row order and numbering for the order each step produces. To list the profiles whose source order is not increasing in pressure, run checkpres on the same directory before merging.

Examples

# Merge all Parquet files, with profile renumbering (the default)
ctddump concat convert ./parquet merged.parquet

# Merge without renumbering
ctddump concat convert --no-renumber ./parquet merged.parquet

# Keep each profile's observations in their original order
ctddump concat convert --no-pres-sort ./parquet merged.parquet

# Keep rows with missing pres
ctddump concat convert --keep-na-pres ./parquet merged.parquet

# Sequential, lowest-memory merge
ctddump concat convert --threads 1 ./parquet merged.parquet

# Merge only a subset
ctddump concat convert --pattern "AR_PR_CT_*.parquet" ./parquet merged.parquet

concat header

Merges YAML header files, each file contributes its top-level keys to the combined output. An error is raised if any two files share the same key.

ctddump concat header ./yaml merged.yaml

report

Summarise a produced Parquet data file or YAML header file as a text report, TSV (default), plain text, or JSON, or assemble a multi-section summary page (report summary) from the per-stage TSV reports.

ctddump report parquet [--level global|platform|profile] [--format tsv|text|json] <src.parquet> [dest]
ctddump report yaml    [--format tsv|text|json] <src.yaml> [dest]
ctddump report summary [--report-dir report] [--out-dir output] [--format md|html]
                       [--title TEXT] [--note TEXT]... [-o dest] <stem>

The report is written to dest, or to stdout when dest is omitted (so it pipes cleanly into other tools).

report parquet

Aggregates a data file at one of three levels (--level, default platform):

LevelOne row perIdentifier columns
globalthe whole filen_platforms, n_profiles
platformplatform_codeplatform_code, n_profiles
profile(platform_code, profile_no)platform_code, profile_no, profile_timestamp, longitude, latitude

Every row also reports:

  • n_obs: number of observations.
  • time_qc_good / position_qc_good: number of profiles whose flag is "1" (CMEMS “good”). These flags are per-profile; at --level profile the single flag value is shown instead.
  • na_temp / na_psal / na_pres: count of missing (null/NaN) values.
  • {temp,psal,pres}_{min,max,mean}: statistics over the valid (non-NaN) values. (Median is intentionally not computed.)
  • {longitude,latitude}_{min,max}: geographic bounding box over the valid positions (global and platform levels only; at profile level the single longitude/latitude is already shown).
  • {temp,psal,pres}_dm_{r,a,d}: number of observations whose data mode is real-time (R), adjusted in real time (A), or delayed mode (D), placed after that parameter’s statistics, followed by {temp,psal,pres}_dm_unknown, the observations with no data mode (""). Each parameter’s counts appear only when the file has its <param>_dm column, so a file converted before the data-mode columns existed reports as before.
  • dup_profiles (global/platform) or is_dup (profile): only when the file has a markdup is_dup column.
# Per-platform summary
ctddump report parquet --level platform merged.parquet report.tsv

# Whole-file summary, human-readable, to stdout
ctddump report parquet --level global --format text merged.parquet

report yaml

Summarises a merged header YAML (as produced by concat header): one row per source file.

ColumnMeaning
filenamesource file stem
has_temp, has_psal, has_pres, has_dephpresence of each core measurement variable
has_time_qc, has_position_qcpresence of each profile-level QC flag (TIME_QC / POSITION_QC)
extra_params;-joined list of the extra measurement parameters present

extra_params is detected automatically: any Float variable dimensioned (TIME, DEPTH) that is not a _QC flag and not a core physical (TEMP/PSAL/PRES/DEPH). This captures biogeochemical/biological parameters (DOXY, FLU2, TUR3, CPHL, NTRA, …) as well as other non-core measurements (CNDC, SVEL, …) without a hard-coded list.

# YAML header summary as JSON
ctddump report yaml --format json merged.yaml report.json

report summary

Assembles a single Markdown (default) or HTML (--format html) page for one file stem (e.g. nrt_ar_ar) by reading the TSV reports the pipeline already produced. It does not scan any Parquet itself, it aggregates the small per-stage report TSVs, so it is instant.

Given a stem, the source files are auto-located at their standard pipeline paths under --report-dir (default report) and --out-dir (default output):

SectionSource file(s)
File summaryreport/header/<stem>.yaml.tsv
Conversionreport/convert/<stem>.parquet.tsv
Pressure orderreport/convert/checkpres/<stem>.summary.tsv and .platforms.tsv
Cleaning → Drop bad QCreport/clean/dropqc/<stem>.parquet.tsv
Cleaning → Drop all-NA profilesreport/clean/dropna/<stem>.parquet.tsv
Cleaning → Filter by regionreport/clean/filter/<stem>.parquet.tsv
Deduplication → Mark duplicatesreport/dedup/markdup/<stem>.parquet.tsv and output/dedup/markdup/<stem>.dups.tsv
Deduplication → Remove duplicatesreport/dedup/dedup/<stem>.parquet.tsv
Comparisonreport/compare/*.tsv (rows whose reference is <stem>)

Any section whose file is absent is skipped, so a partially-run pipeline still produces a valid page (a parent heading appears only when it has at least one present subsection). The Mark-duplicates section shows its within/across tables only when the .dups.tsv is also present.

Each stage table reports platforms / profiles / observations. The % of original and Deleted columns compare against the Conversion stage (the baseline “original”); if Conversion is absent, the earliest present stage is used.

The two Deduplication stages ran on the cleaned data, not on the original, so their tables carry three further columns, Cleaned, % of cleaned and Deleted (cleaned), comparing them against the last cleaning stage present (Filter, else Drop all-NA, else Drop bad QC). When no cleaning stage ran there is nothing to compare against and the columns are omitted.

When the Conversion report has the {temp,psal,pres}_dm_{r,a,d} columns, the Conversion section carries a second table, Data mode: one row per parameter with the number of observations in real-time (R), adjusted (A), and delayed mode (D), each with its share of all converted observations, then Unknown for those with no data mode. Reports made before the _dm_unknown counts existed have no Unknown columns, so their rows can sum to less than 100%. Reports of files converted before the data-mode columns existed have no data-mode columns at all, and the table is omitted.

The Pressure order section comes from checkpres, run on the per-file Parquet before the merge. A first table gives the profiles checked (and those without any pressure, which are not), those with decreasing pressure (split into reversed and non-monotonic) as shares of all profiles, and the platforms and files that have any. When there are such profiles, a Platforms table lists the 10 platforms with the most, with their share of the platform’s profiles and the dates of the first and last one. The full lists stay in the checkpres TSVs.

The Filter by region section carries a second table, Bounding box, with the minimum and maximum longitude and latitude of the profiles that survived the filter, in decimal degrees to three places. The extremes are aggregated from the per-platform longitude_min / longitude_max / latitude_min / latitude_max columns of the stage TSV, so they span every platform in the file. When the Conversion report is present a second column, Original, gives the same extremes for the converted data before any cleaning ran, which shows how far the filter tightened the box. Profiles with a missing position are ignored, and if no profile has a valid position the table is omitted entirely.

The Mark-duplicates section additionally lists the duplicate profiles that dedup would remove, split into:

  • within a platform: a dup_group confined to a single platform_code;
  • across platforms: a group spanning two or more.

(A dup_group is a set of profiles sharing the duplicate key, date + rounded position, so an across-platform group is the same cast reported by more than one platform.)

A third table, Duplicate group sizes, shows how many profiles the groups hold: one row per distinct size up to 10, then a single 11+ row for the tail.

The Comparison section appears when report/compare/ holds any compare TSV whose reference column names this stem, i.e. a comparison run with this product as the reference (the pipeline writes these with compare_data.sh). It gathers every such row across all TSVs in that directory. A first table, This product, gives the reference’s own totals (platforms, profiles, profiles without a matchable key, observations), the denominators the coverage is measured against; a second, Coverage of other products, has one row per compared product: the common platforms and platform coverage, the matched profiles and profile coverage, and, among the matched profiles, how many carry the same versus a different observation count (Same obs / Diff obs / Obs agreement). All percentages are recomputed from the counts, so they do not depend on how the compare TSV formatted its own percentage columns.

Title and notes

Each section carries a short, generic explanation of what the stage did, the same prose for every region and dataset. Anything region- or product-specific goes in:

  • --title TEXT: replaces the default Summary: <stem> heading.
  • --note TEXT: a note rendered under the title; repeat for several notes.

Both are plain text and are escaped in HTML output. In the pipeline these are supplied per stem by summary_data.sh, which is the place to edit what a page says about a given region or product.

# Markdown page for the Arctic AR stem, to stdout
ctddump report summary nrt_ar_ar

# HTML page, with a title, notes, and non-default roots, to a file
ctddump report summary nrt_bo_bo --report-dir report --out-dir output \
  --format html --title "Baltic Sea: Near Real Time, regional product (BO)" \
  --note "Each source file holds a single platform." \
  -o summary/nrt_bo_bo.html

filter

Filter a produced Parquet data file by a geographic bounding box, keeping or dropping whole profiles, and write the result to a new Parquet file.

ctddump filter [--mode include|exclude] \
  --min-lon <W> --max-lon <E> --min-lat <S> --max-lat <N> \
  <src.parquet> <dest.parquet>

The box is inclusive on all four edges, and --min-* must not exceed the matching --max-* (an inverted box is rejected; antimeridian wrap is not supported).

Because a profile’s longitude/latitude are constant across its observations, the box acts on whole profiles, every observation of a matching profile is kept or dropped together.

--mode (default include):

ModeEffect
includekeep only profiles inside the box
excludedrop profiles inside the box, keep everything else

A profile whose position is NaN (unknown) is treated as outside the box: dropped by include, kept by exclude.

The file is streamed one row group at a time, so peak memory stays bounded regardless of file size (tune with CTDDUMP_CHUNK_ROWS, as for convert).

# Keep only profiles inside a Mediterranean sub-box
ctddump filter \
  --min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 \
  merged.parquet med_box.parquet

# Remove profiles inside that box, keep the rest
ctddump filter --mode exclude \
  --min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 \
  merged.parquet outside_box.parquet

dropna

Filter a produced Parquet data file, dropping whole profiles that carry no usable data in one of the core measurement parameters, and write the survivors to a new Parquet file.

ctddump dropna <src.parquet> <dest.parquet>

A profile is kept only if each of temp, psal, and pres has at least one non-NA observation. It is dropped if any one of those parameters is entirely NA (null or NaN) across the profile. Partial NAs within a parameter are fine, a kept profile retains all of its observations, including the NA ones.

ProfiletemppsalpresResult
some valid, some NaN✅ has data✅ has data✅ has datakept
all NaN in one param✅❌ all NaN✅dropped

The file is processed in two streaming passes (build the keep-set, then re-emit the kept rows), so peak memory stays bounded regardless of file size and the result is independent of how the file is chunked (tune with CTDDUMP_CHUNK_ROWS, as for convert).

# Drop profiles with an all-NA temp, psal, or pres
ctddump dropna merged.parquet cleaned.parquet

dropqc

Filter a produced Parquet data file, dropping whole profiles whose profile-level quality-control flags mark them as bad, and write the survivors to a new Parquet file.

ctddump dropqc <src.parquet> <dest.parquet>

A profile is kept only if both time_qc and position_qc are either "1" (OK) or missing. A missing flag is one that is absent from the source NetCDF, or stored as the NA byte -128, both render as the empty string "" in the Parquet output. Any other flag ("0", "2"…"9", or a non-numeric char code) drops the whole profile.

Missing QC is treated as acceptable on purpose: several source files ship no profile-level position_qc (or time_qc) at all, and those profiles must be kept rather than discarded. Note that the Argo “missing value” flag 9 is a present value, not the NA byte, so a "9" flag is dropped.

time_qcposition_qcResult
1 (OK)1 (OK)kept
1 (OK)"" (missing / NA)kept
"" (missing)"" (missing)kept
4 (bad)1 (OK)dropped
1 (OK)0 (present, not OK)dropped
9 (present)9 (present)dropped

Because time_qc/position_qc are constant within a profile, this is a plain per-row predicate that keeps or drops whole profiles. The file is streamed one row group at a time (set_parallel(false)), so peak memory stays bounded regardless of file size (tune with CTDDUMP_CHUNK_ROWS, as for convert).

# Drop profiles flagged bad in time_qc or position_qc
ctddump dropqc merged.parquet cleaned.parquet

markdup

Mark duplicate profiles in a produced Parquet data file with an is_dup column, and write a TSV listing the duplicated profiles.

ctddump markdup [OPTIONS] <src.parquet> <dest.parquet> <dups.tsv>

Two profiles are duplicates when they share the same key, built from:

  • profile_timestamp: formatted with a strftime string; date only (%Y-%m-%d) by default.
  • longitude and latitude: rounded to 3 decimals by default.

platform_code is deliberately not part of the key, so duplicates are detected across platforms. A profile whose position is NaN or whose timestamp is null has no key and is never marked.

The is_dup column (Boolean) is true for every profile whose key is shared by at least one other profile. Two streaming passes bound memory regardless of file size.

Options

OptionDefaultMeaning
--time-format <FMT>%Y-%m-%dstrftime format applied to profile_timestamp for the key
--decimals <N>3decimal places longitude/latitude are rounded to
--round-mode <MODE>roundround, floor, ceil, or trunc

Outputs

  1. dest.parquet: the input plus the is_dup column.
  2. dups.tsv: one row per duplicated profile, grouped by dup_group: dup_group, platform_code, profile_no, profile_time, profile_timestamp, longitude, latitude, n_obs.
# Default key: same date + coordinates rounded to 3 decimals
ctddump markdup merged.parquet marked.parquet duplicates.tsv

# Match on the full hour and 2-decimal coordinates
ctddump markdup --time-format "%Y-%m-%d %H" --decimals 2 merged.parquet marked.parquet dups.tsv

Use dedup to then drop the duplicates, and report parquet to summarise the duplicate counts.

dedup

Remove duplicate profiles from a produced Parquet data file, keeping one profile per duplicate group.

ctddump dedup [OPTIONS] <src.parquet> <dest.parquet>

dedup re-derives the same duplicate key as markdup, from profile_timestamp (date only by default) and longitude/latitude (3-decimal rounding by default), so it can run standalone with the same options. Within each group of profiles sharing a key, it keeps the profile with the most observation rows; ties are broken by first appearance. Profiles with no key (NaN position or null timestamp) are always kept.

If the input has an is_dup column (from markdup), it is reset to false: the survivors are unique, so the schema stays stable across the markdup → dedup pipeline and a follow-up report shows zero duplicates.

Two streaming passes bound memory regardless of file size.

Options

The key options match markdup and should be given the same values used there:

OptionDefaultMeaning
--time-format <FMT>%Y-%m-%dstrftime format applied to profile_timestamp for the key
--decimals <N>3decimal places longitude/latitude are rounded to
--round-mode <MODE>roundround, floor, ceil, or trunc
# Drop duplicates using the default key (same date + 3-dp coordinates)
ctddump dedup marked.parquet deduped.parquet

compare

Compare two produced Parquet data files and report how far each one covers the other’s platforms and profiles.

ctddump compare [OPTIONS] <a.parquet> <b.parquet> [dest]

Without dest the report goes to stdout.

What it answers

Two questions, for a pair of files:

  • How much do they have in common? Platforms present in both, and profiles present in both.
  • Do the matched profiles agree? For every profile found in both files, whether it has the same number of observation rows in each.

Coverage is reported in both directions, because it is not symmetric. A small file fully contained in a large one is 100% covered while covering only a fraction of the large one. The first output row uses the second file as the reference, the second row uses the first.

How profiles are matched

A profile’s key is built from its platform code, its time reduced to a date, and its longitude and latitude rounded to 3 decimals. This is the same key markdup uses for duplicates, with one difference: markdup deliberately leaves the platform code out, while compare includes it by default. Two products covering the same waters usually share platform codes, so the platform code is normally a wanted part of the identity. Pass --no-platform-key to match on time and position alone, which finds the same cast recorded under two different platform codes.

Profiles whose time or position is missing get no key and can never match. They are counted separately as ref_unkeyed_profiles so a low coverage figure can be told apart from a file full of unusable positions.

The time column may be either a datetime (profile_timestamp) or a float of days since 1950 (profile_time); the type is detected and handled automatically.

Output columns

One row per direction:

ColumnMeaning
referenceFile the percentages are relative to
comparedThe other file
ref_platformsDistinct platform codes in the reference
common_platformsPlatform codes present in both files
platform_cov_pctcommon_platforms as a percentage of ref_platforms
ref_profilesProfiles in the reference
ref_unkeyed_profilesReference profiles with no usable time or position
matched_profilesReference profiles whose key is present in the other file
profile_cov_pctmatched_profiles as a percentage of ref_profiles
same_nobsMatched profiles with the same observation count in both files
diff_nobsMatched profiles whose observation count differs
nobs_agree_pctsame_nobs as a percentage of matched_profiles
ref_observationsObservation rows in the reference
matched_observationsObservation rows in the reference’s matched profiles

profile_cov_pct is taken over all reference profiles, including unkeyed ones, since a profile that cannot be matched is still a profile the other file does not demonstrably have. A percentage with no denominator (comparing against an empty file, or nobs_agree_pct when nothing matched) is left empty rather than shown as zero.

When several profiles share one key, the observation counts agree if any of the profiles carrying that key in the other file has the same count.

Options

OptionDefaultMeaning
--time-format <FMT>%Y-%m-%dstrftime format applied to the time column for the key
--decimals <N>3decimal places longitude/latitude are rounded to
--round-mode <MODE>roundround, floor, ceil, or trunc
--platform-col <NAME>platform_codecolumn holding the platform code
--time-col <NAME>profile_timecolumn holding the profile time
--lon-col <NAME>longitudecolumn holding the longitude
--lat-col <NAME>latitudecolumn holding the latitude
--no-platform-keyoffmatch on time and position only
--format <FMT>tsvtsv, text, or json

Examples

Compares are usually between two products covering the same waters, for example the CORA reanalysis against a regional NRT extract.

# How much of the Arctic NRT product does CORA cover, and vice versa
ctddump compare --format text cora.parquet nrt_ar.parquet

# Save the summary
ctddump compare cora.parquet nrt_ar.parquet compare.tsv

# Ignore platform codes, so the same cast filed under two codes still matches
ctddump compare --no-platform-key cora.parquet nrt_ar.parquet

# Match to the hour and to 4 decimals instead of the date and 3
ctddump compare --time-format '%Y-%m-%dT%H' --decimals 4 cora.parquet nrt_ar.parquet

Reading the result: if the smaller product shows profile_cov_pct near 100 while the larger shows much less, the smaller is close to a subset of the larger. If both are low, the two products overlap only partly. A high profile_cov_pct with a low nobs_agree_pct means the same profiles are present in both but carry different numbers of observations, which usually points at different QC or cleaning in the two products.

Memory is bounded: each file is streamed and reduced to one record per profile, so peak use follows the profile count rather than the file size.

sqlite

Export a produced Parquet data file to a SQLite database, normalised into three related tables. This turns the flat observation table into something you can query with SQL, index, and open in any SQLite-aware tool.

ctddump sqlite [OPTIONS] <src.parquet> [dest.sqlite]

Without dest the output is the source path with its extension changed to .sqlite. The command refuses to overwrite an existing output unless --force is given.

The schema

The flat Parquet repeats every profile-level value on each observation row. sqlite splits that back apart by grain:

  • platform holds one row per platform_code.
  • profile holds one row per (platform_code, profile_no), carrying every column that is constant within a profile (time, position, QC, filename), with a foreign key to platform.
  • observation holds one row per observation (the measurements and their QC and conversion flags), with a foreign key to profile.
CREATE TABLE platform (
  platform_id   INTEGER PRIMARY KEY,
  platform_code TEXT UNIQUE NOT NULL
);
CREATE TABLE profile (
  profile_id        INTEGER PRIMARY KEY,
  platform_id       INTEGER NOT NULL REFERENCES platform(platform_id),
  profile_no        INTEGER NOT NULL,
  profile_time      REAL,      -- days since 1950-01-01
  profile_timestamp INTEGER,   -- Unix milliseconds
  longitude         REAL,
  latitude          REAL,
  profile_longitude REAL,
  profile_latitude  REAL,
  time_qc           TEXT,
  position_qc       TEXT,
  filename          TEXT,
  UNIQUE(platform_id, profile_no)
);
CREATE TABLE observation (
  observation_id INTEGER PRIMARY KEY,
  profile_id     INTEGER NOT NULL REFERENCES profile(profile_id),
  observation_no INTEGER NOT NULL,
  temp REAL, temp_qc TEXT, temp_dm TEXT,
  psal REAL, psal_qc TEXT, psal_dm TEXT,
  pres REAL, pres_qc TEXT, pres_dm TEXT, pres_conv INTEGER,
  deph REAL, deph_qc TEXT, deph_conv INTEGER
);

Indexes are created on profile(platform_id), observation(profile_id), and profile(profile_timestamp) so the common joins and time queries are fast.

Only the columns of the standard output schema are exported; any other column in the Parquet (for example markdup’s is_dup) is ignored unless you ask for it with --add-col. A column that is absent from the input is simply left out of its table, so a file that never had deph produces a schema without the depth columns.

NULL handling

The Parquet uses sentinels for missing values: NaN for floats and an empty string for absent QC flags and data modes. Both are stored as SQL NULL, so queries like WHERE temp IS NOT NULL behave the way SQL expects.

Adding your own columns

Two options attach extra columns to a chosen table. Both name their target as TABLE.COLUMN, where TABLE is platform, profile, or observation.

--add TABLE.COLUMN=VALUE adds a constant column filled with one value on every row of the table. It is handy for tagging an export with where it came from. The value type is inferred: a whole number becomes an integer, a decimal becomes a real, anything else is stored as text. Repeat the flag for several columns.

--add-col TABLE.COLUMN copies an existing Parquet column of that name into the table. For a column added to observation the value is taken per row. For a column added to profile or platform the value must be constant within each group (each profile, or each platform): if it varies, the export stops with an error naming the column and the group. Nulls are ignored when checking, so a column that is set on some rows of a profile and null on others is accepted.

A column name must be a plain identifier and must not clash with a column the table already has.

Examples

# Straight export next to the source (writes arctic.sqlite)
ctddump sqlite arctic.parquet

# Explicit output, overwriting any earlier file
ctddump sqlite --force arctic.parquet /data/arctic.sqlite

# Tag every row with the source product and region
ctddump sqlite --add profile.dataset=NRT_AR --add platform.source=copernicus \
  arctic.parquet arctic.sqlite

# Carry a per-profile institution and a per-observation extra parameter through
ctddump sqlite --add-col profile.institution --add-col observation.chla \
  arctic.parquet arctic.sqlite

A typical query against the result, joining all three tables:

SELECT pl.platform_code, p.profile_no, p.profile_timestamp,
       o.pres, o.temp, o.psal
FROM observation o
JOIN profile  p  ON o.profile_id  = p.profile_id
JOIN platform pl ON p.platform_id = pl.platform_id
WHERE o.temp IS NOT NULL
ORDER BY pl.platform_code, p.profile_no, o.pres;

Memory

The export streams in two passes and never loads the whole file at once. The first pass builds the platform and profile rows, so its memory follows the number of profiles rather than the file size (the same assumption markdup and dedup rely on). The second pass streams the observations in fixed windows and inserts them, so an arbitrarily large observation body loads a window at a time.

Configuration

All convert and batch convert subcommands accept a --config TOML file plus individual flag overrides. The priority order is:

built-in default  <  --config file  <  individual CLI flags

--config may set any subset of fields; a per-field flag then overrides the config for that field only.

NRT flags

FlagFieldDefault
--deph-source / --no-deph-sourcehas_deph_sourcetrue for BO/GL, false for AR/MO
--profile-coords / --no-profile-coordshas_profile_coordstrue for BO, false otherwise
--pattern <GLOB>patternsee the batch patterns

DEPH is auto-detected. NRT always performs the bidirectional PRES↔DEPH conversion when the file contains a DEPH variable, regardless of has_deph_source. The flag only forces DEPH handling on for files where the variable might otherwise be skipped.

CORA flags

FlagFieldcora defaultcora_legacy default
--time-var <VAR>time_varTIMEJULD
--qc-type <int|char>qc_typeintchar
--time-qc / --no-time-qchas_time_qctruefalse
--deph-source / --no-deph-sourcehas_deph_sourcetruefalse
--pattern <GLOB>pattern*.nc*.nc

Generic flags

The generic format reads a mapping file that names your file’s dimensions and variables; see Your own NetCDF files for its keys.

FlagFieldDefault
--layout <time-depth|profile-level>layouttime-depth
--time-units <UNITS>time_unitsthe time variable’s units attribute
--platform-code <CODE>[defaults] platform_codenone
--pattern <GLOB>pattern*.nc

TOML config format

# NRT
has_deph_source    = true
has_profile_coords = false
pattern            = "AR_PR_CT_*.nc"  # optional

# CORA
time_var        = "TIME"
qc_type         = "int"   # "int" or "char"
has_time_qc     = true
has_deph_source = true
pattern         = "*.nc"  # optional

Environment variables

ctddump honours a few environment variables for operational tuning. They do not change the contents of the output, only memory use, speed, and the on-disk row-group layout, which is why they live in the environment rather than as per-command flags.

VariableEffectWhen unset
CTDDUMP_CHUNK_ROWSObservation rows assembled per streamed chunk (each written as one Parquet row group). Lower = less memory but more row groups; higher = more memory but fewer. Applies to convert, batch, concat, and the streaming filters (filter / dropqc / dropna / markdup / dedup).1000000
POLARS_MAX_THREADSSize of Polars’ internal thread pool. batch / concat pin this to 1 so their own --threads is the real knob, but a value you set is respected.pinned to 1 by batch / concat
RUST_MIN_STACKWorker-thread stack size, in bytes. Raised to 16 MiB for the deep call chains in Polars’ Parquet writer; a larger value you set is respected.raised to 16 MiB

CTDDUMP_CHUNK_ROWS is the only one most users would ever change, see Technical notes for why streaming in chunks matters. The helper scripts expose it directly as --chunk-rows N, which exports it for every ctddump process they launch.

Shell completion

ctddump completions <shell> prints a completion script to standard output. Once your shell loads that script, typing ctddump and pressing Tab suggests the subcommands, and Tab-completing a partial word (a subcommand, a flag, or an enum value such as a shell name) fills it in when only one candidate remains.

Supported shells: bash, zsh, fish, elvish, powershell.

The command only writes the script; where you put it is what makes completion permanent. Pick the section for your shell.

Bash

# Try it in the current shell
source <(ctddump completions bash)

# Install for every new shell (user-local)
mkdir -p ~/.local/share/bash-completion/completions
ctddump completions bash > ~/.local/share/bash-completion/completions/ctddump

This relies on the bash-completion package being installed and sourced from your ~/.bashrc. Without it, add source <(ctddump completions bash) to ~/.bashrc directly.

Zsh

# A directory on your $fpath, created if needed
mkdir -p ~/.zsh/completions
ctddump completions zsh > ~/.zsh/completions/_ctddump

Make sure that directory is on $fpath and completion is initialised, before the compinit call in your ~/.zshrc:

fpath=(~/.zsh/completions $fpath)
autoload -Uz compinit && compinit

Start a new shell (or run compinit) to pick up the change.

Fish

mkdir -p ~/.config/fish/completions
ctddump completions fish > ~/.config/fish/completions/ctddump.fish

Fish loads it automatically in new shells.

PowerShell

# Add to your profile so it loads every session
ctddump completions powershell | Out-String | Invoke-Expression

To make it permanent, append that line to the file at $PROFILE.

Elvish

ctddump completions elvish > ~/.config/elvish/lib/ctddump.elv
# then, in rc.elv:
use ctddump

Regenerating after an upgrade

The script encodes the commands and flags of the version that produced it. After upgrading ctddump, regenerate it with the same command so new subcommands and options complete correctly.

Output schema

Every converter produces the same observation-level flat table, so files from different sources can be merged directly.

ColumnTypeNotes
platform_codeString
profile_nou32see Row order and numbering
profile_timef64days since 1950-01-01
profile_timestampDatetime(ms)Unix milliseconds
observation_nou32see Row order and numbering
longitude / latitudef32 (NRT) / f64 (CORA)generic: as NRT for time-depth, as CORA for profile-level
profile_longitude / profile_latitudef32 (NRT) / f64 (CORA)from PRECISE_* or expanded DEPLOY_*; NaN when unavailable (always for generic)
time_qc / position_qcString"" if absent
filenameStringsource file stem
temp, psal, pres, dephf32NaN where missing
temp_qc, psal_qc, pres_qc, deph_qcStringsingle-char flag; "" if missing
temp_dm, psal_dm, pres_dmStringdata mode: R, A, or D; "" when unknown (see below)
pres_conv, deph_convi81 = value derived by conversion

Pressure and depth are converted to each other with the TEOS-10 (gsw) routines when only one is present, and the *_conv columns flag which values were derived rather than measured.

Row order and numbering

concat convert sorts each profile’s observations by increasing pressure. A single convert (or batch convert) does not: it keeps the order of the source file. Which order a Parquet file has depends on which step wrote it.

StepObservation orderprofile_noobservation_no
convert, batch convertas in the source file (its level dimension)the profile’s index in the file, from 1the level’s index in the file, from 1; levels with no valid measurement are dropped, leaving gaps
concat convert (default)increasing pres within each profilerenumbered from 1 within each platformrenumbered 1..N within each profile, following increasing pres
concat convert --no-pres-sortas in the source filesrenumberedrenumbered 1..N, following the source order
concat convert --no-renumberas read, unsortedkept from the inputkept from the input

Details of the default concat convert:

  • Rows are sorted by platform_code, profile_timestamp, longitude, latitude, pres. Profiles are therefore in time order within each platform, and the observations of a profile run from the surface down.
  • Rows whose pres is missing are dropped first, so a profile with no pressure at all disappears here. With --keep-na-pres they are kept and sorted to the end of their profile, after the valid pressures.
  • Observations with the same pressure keep their source order (by file path, then by position in the file).
  • profile_no is assigned per distinct (platform_code, profile_timestamp, longitude, latitude). Two profiles from different files that share all four become one profile, and the pressure sort interleaves their observations. filename still tells them apart.
  • All other columns move with their row; nothing is interpolated or averaged.

The pipeline scripts and the regional workflows merge with the default, so their merged Parquet files, and everything cleaned, de-duplicated, or summarised from them, are sorted by pressure. Pass --no-pres-sort to keep the acquisition order instead (see concat). To find the profiles whose source order is not increasing in pressure, run checkpres on the per-file Parquet before the merge.

Data mode

The *_dm columns record how each value was processed:

CodeMeaning
Rreal-time
Aadjusted in real time
Ddelayed mode

The code is copied from the source file; ctddump never infers it. Where it comes from depends on the source:

  • NRT, with a <PARAM>_DM variable (TEMP_DM, PSAL_DM, PRES_DM; found in some Arctic files, whose parameters then carry data_mode = "M" for “mixed”): the per-observation code is copied through. A fill cell becomes "".
  • NRT, without one: every observation gets the parameter’s data_mode attribute when it is R, A, or D. Any other attribute, notably "M", says the mode varies per observation without saying how, so the column is ""; a missing attribute gives "" too.
  • NRT pressure derived from depth (pres_conv = 1): pres_dm is taken from DEPH_DM (or DEPH’s data_mode attribute), the same way pres_qc takes deph_qc.
  • CORA: there are no per-parameter data-mode variables, so all three columns hold the profile’s DATA_MODE, repeated on each of its observations ("" if the file has no DATA_MODE, or for a profile whose cell is a fill value).
  • generic: as NRT for the time-depth layout and as CORA for profile-level, except that [defaults] data_mode replaces "" when the file has no data mode at all (see Your own NetCDF files). A "M" attribute stays "" even with a default.

A parameter the file does not have at all (all NaN) has "" as well.

convert and batch convert print one warning per file whose data mode is left empty for a whole variable (a "M" attribute without <PARAM>_DM, no data-mode information at all, or a CORA file without DATA_MODE), naming the variables. Fill cells inside a data-mode variable are not reported.

Keep in mind when using these columns:

  • "" means “no code in the file”, not “no measurement”: a valid value can have an empty data mode, and a missing value can have a code.
  • The values and flags are always the ones in TEMP / TEMP_QC (and so on); *_ADJUSTED variables are never read. In Copernicus NRT files the delayed-mode values and flags are in those variables, with <PARAM>_DM saying which observations they are.
  • CORA’s DATA_MODE comes from the original observation. CORA re-processes the QC of every profile, so R there does not mean the flags are real-time ones.

Technical notes

This page collects the non-obvious technical problems we ran into while building ctddump and how each was solved. It is written for a general audience, you do not need to know Rust to follow it. The recurring themes are memory use, multi-threading, how data is read and written in chunks, and a handful of quirks in the Polars data-frame library we depend on.

Each note is laid out the same way: what the symptom looked like, why it happened, and what we did about it.


1. Converting a file used far too much memory

Symptom. Converting a single large NetCDF file (a few hundred MB on disk) pushed memory use to ~8–9 GB, enough to fail on modest machines.

Why. A NetCDF CTD file is stored as a dense rectangular grid: every profile (a cast of the instrument) times every depth level. Most of that grid is empty: a shallow cast still reserves a slot for every deep level other casts reach. If you load the whole grid into memory at once and only then throw away the empty cells, you have already paid for the full rectangle.

Fix: stream the file in chunks. Instead of loading everything, the converter walks the file in slices along the profile dimension (about one million data rows at a time). Each slice is turned into a small table and written straight to disk as one “row group” of the output Parquet file, then discarded before the next slice is read. The file is only ever partially in memory. On a 230 MB test file this cut peak memory from ~8.8 GB to ~0.7 GB, and lower still with smaller chunks.

The chunk size defaults to one million rows and can be tuned with the CTDDUMP_CHUNK_ROWS environment variable, smaller means less memory but more row groups. The helper scripts expose it as --chunk-rows N, and it is listed alongside the other tuning variables under Configuration. Importantly, the chunking only changes how the output is laid out on disk; the actual data and its order are identical no matter what chunk size you pick.


2. Empty rows were dropped the slow, memory-leaky way

Symptom. Running the batch converter over thousands of small files caused memory to climb steadily and never come back down, roughly 0.2 MB lost per file, with no upper limit. Converting 7,905 tiny files eventually consumed 88 GB.

Why. This is a bug in the version of the Polars library we use (0.43.1). Several of Polars’ parallel operations, filtering rows, gathering rows, and the default parallel way of writing Parquet columns, leak a small amount of memory on every call that is never released. On one file it is invisible; across thousands of files in one long-running process it adds up without bound. The leak is independent of the memory allocator and cannot be reclaimed manually.

Fix: avoid the leaky parallel paths. Two changes, both inside the converters:

  1. When dropping the empty (all-missing) rows, we do it on the plain raw arrays before handing the data to Polars, so Polars’ leaky parallel filter is never called and the empty rows never enter Polars at all.
  2. Every Parquet write in the converters (and in concat) turns off Polars’ parallel column encoding.

Together these keep batch memory flat regardless of how many files are processed (3,000 small files went from ~0.9 GB down to ~40 MB), and the output is verified identical to the old parallel path. If a future Polars release fixes the leak, these workarounds can be revisited.


3. Reading a big Parquet file in slices returned the wrong rows

Symptom. Commands that re-read a Parquet file in slices (filter, dropqc, dropna, markdup, dedup) silently produced wrong results on large files: for example, output that only contained platforms whose names started with the first few letters of the alphabet, as if most of the file had vanished.

Why. Another bug in Polars 0.43.1. When a Parquet file has more than one row group (which any file bigger than one chunk does), and you ask its parallel reader for “the rows from position X onwards,” it ignores X and keeps returning rows from the very first row group. So every slice read the same opening chunk over and over, and the rest of the file was never seen. It was invisible in our automated tests because test fixtures are small enough to be a single row group.

Fix: use the sequential reader for sliced reads. The five affected commands now read Parquet with Polars’ non-parallel reader, which honours the slice position correctly. Whole-file operations (like report) are unaffected and keep the faster parallel reader. A regression test now writes a deliberately many-row-group file and slices through it to guard against this coming back.

This bug was also the root cause of a confusing field report where dropqc output looked truncated: the user was running an older build from before this fix. Rebuilding to the current version resolved it.


4. markdup crashed on large files with a “record batch” error

Symptom. markdup panicked on large merged files with RecordBatch requires all its arrays to have an equal number of rows, and left a truncated, unreadable output file. It worked fine on small files.

Why. markdup reads the file in slices, adds a new true/false is_dup column to each slice, and writes it out. When a slice happened to span several row groups of the input, the columns it read back were internally split into several pieces (“chunks”), while the freshly-built is_dup column was a single piece. The Parquet writer writes one batch per internal piece and insists every column be split the same way, the mismatch made it crash. Small files have only one row group, so their columns were never split and the bug never showed.

Fix: line up the pieces before writing. After adding the is_dup column, markdup now re-aligns all columns to the same internal chunk layout before writing each slice. The sibling commands did not need this because their row-filtering step happens to realign the columns for free. A new test feeds markdup a multi-row-group input to keep this covered.


5. Multi-threaded batch runs crashed with a stack overflow

Symptom. Batch conversion processed each file on a separate worker thread and crashed on large files, even though converting the same file on its own worked.

Why. Worker threads get a much smaller memory “stack” by default (about 2 MB) than the program’s main thread (about 8 MB). Polars’ Parquet writer uses deep call chains on large files that need more stack than the 2 MB default, so it overflowed only when run on a worker thread.

Fix: give the workers a bigger stack. Batch mode builds its thread pool with a 16 MB stack per worker, and the program also raises the stack size for Polars’ own internal threads. Single-file conversion never hit this because it runs on the roomier main thread.


6. Asking for N threads actually spawned far more

Symptom. Requesting, say, 10 threads created many more than 10 busy threads, oversubscribing the machine.

Why. Polars keeps its own thread pool, sized to the number of CPU cores. So running our 10 file-workers, each of which calls into Polars, produced roughly 10 + (number of cores) threads all competing for the CPU.

Fix: pin Polars to one internal thread. Batch and concat set Polars to a single internal thread before the first Polars call, so our own --threads setting is the real, honest knob. We already parallelise across files (or across data ranges in concat), so we do not need Polars to add a second layer of threading on top.


7. A single large file stalled all the other threads

Symptom. In a multi-threaded batch run, watching a system monitor showed all cores busy at first, then most of them going idle near the end while one core finished a large file. It looked as though files were processed in fixed groups that waited for the slowest member.

Why. The work scheduler treats every file as equally expensive, it has no idea one file is much bigger than the others, and a single file cannot be split across threads. If a big file happened to be picked up late, the other workers would finish all the small files, find nothing left to do but that one big file (which they cannot help with), and sit idle until it finished.

Fix: start the biggest files first. The batch converter now sorts files largest-first before handing them out, and dispatches them one at a time so a free worker always grabs the next individual file. Starting heavy files at the very beginning lets the many small files fill in around them, so the run finishes close to “total work ÷ threads” instead of being dragged out by a late big file. The one unavoidable limit: if a single file is larger than everything else combined, its own runtime still sets the floor, no scheduler can split one file across threads. Starting it first is simply the best possible placement.


8. Merging many files without loading them all at once

Challenge. concat merges many Parquet files into one and re-numbers the profiles consistently across the whole result. Doing that naively means holding every input in memory at the same time.

Fix: merge one platform at a time. Re-numbering is grouped by platform (each platform’s profiles are numbered together), so concat processes one contiguous block of platforms at a time and writes each block out as it goes. A quick first pass reads only the tiny “platform” column of every file to plan the blocks; the second pass re-reads only the files that contribute to each block. Because a given platform’s data always lands in the same block, profiles that were split across input files are still merged correctly. Producing the blocks in order gives a result identical to merging everything at once, only the on-disk layout differs.

When more than one thread is allowed, whole platform blocks are independent units of work, so concat renumbers several blocks at once, each writing to a temporary file, and then stitches the temporaries together in order. The final file is byte-for-byte identical to the single-threaded result.


9. Missing datasets should not break the pipeline

Symptom / need. One planned dataset (the global product for the Baltic Sea) is not yet published by the data provider, but the workflow already refers to it. We did not want the whole run to fail just because those files do not exist yet.

Fix: treat “no matching files” as a no-op, not an error. When a batch or concat command is pointed at a pattern that matches nothing (or an empty/absent folder), it now prints a short informational message and writes no output, instead of stopping with an error. The helper scripts likewise skip any step whose input file is missing, with a note. The result is that the pipeline runs cleanly today and will automatically pick up the Baltic global data the moment it becomes available, with no further changes.


10. Harmless HDF5 diagnostic messages

Symptom. On some systems, reading certain files prints scary-looking HDF5-DIAG messages to the screen.

Why. These come from an older version of the underlying HDF5 library (1.10.x, used on our continuous-integration runner) encountering a newer file attribute it does not recognise. It is only a diagnostic, the data is read correctly.

What to do. Nothing. The messages are harmless and the output is unaffected. Using a newer HDF5 library avoids them.


A recurring lesson

Most of the hard bugs above share a shape: they only appear at scale. A file big enough to have multiple row groups, or a run long enough for a small leak to matter, or a workload uneven enough for scheduling to bite. Small test fixtures pass happily while the real data fails. Wherever we fixed one of these, we also added a test that deliberately recreates the “large” condition (many row groups, tiny chunks, mixed sizes) so the fix stays fixed.

Your own NetCDF files

ctddump is built around the Copernicus files, but it can also convert your own NetCDF files, for example from a mooring, a glider campaign, or a research cruise. Use the generic format with a small mapping file that tells ctddump what your dimensions and variables are called. The output has the same schema as the Copernicus conversions, so your data can be merged, cleaned, de-duplicated, and reported on with the rest of the toolkit.

ctddump convert generic --config my_map.toml my_data.nc my_data.parquet

What your file needs

Your file must be laid out like a Copernicus file: a profile dimension and a level dimension, with the measurements stored as 2-D variables indexed [profile, level], in that order.

InformationWhere it can come fromRequired?
Profile and level dimensionsany two dimension namesyes
Timea variable, scalar or [profile], with CF units such as "seconds since 1970-01-01"yes
Longitude, latitudevariables, scalar or [profile], or a default valueyes
Platform codea global attribute, a variable, or a default valueyes
Pressure or deptha [profile, level] variableat least one
Temperature or salinitya [profile, level] variableat least one
QC flags[profile, level] variables (integer or char)no: empty when absent
Data modevariables, a data_mode attribute, or a default valueno: empty when absent

Two layouts are supported. They differ in what a profile is and where the platform code usually lives:

time-depth (default)                    profile-level
like Copernicus NRT                     like Copernicus CORA / Argo

  one platform per file                   many profiles, many platforms
  platform: global attribute              platform: [profile] variable

          level ->                                level ->
  profile [ x  x  x  x ]  TEMP[TIME,DEPTH]  profile [ x  x  x  . ]  TEMP[N_PROF,N_LEVELS]
     |    [ x  x  x  . ]                       |    [ x  x  .  . ]
     v    [ x  x  x  x ]                       v    [ x  x  x  x ]
  TIME[TIME]                               TIME[N_PROF], PLATFORM_NUMBER[N_PROF, ...]

Profiles are numbered in file order (profile_no), and levels within each profile (observation_no). Cells where temperature, salinity, and pressure are all missing are dropped.

Example: a mooring

Suppose ncdump -h mooring.nc shows:

dimensions:
    time = 8760 ;
    z = 12 ;
variables:
    double time(time) ;
        time:units = "seconds since 1970-01-01 00:00:00" ;
    float lat ;
    float temperature(time, z) ;
        temperature:_FillValue = -999.f ;
    short temperature_flag(time, z) ;
    short sal(time, z) ;
        sal:scale_factor = 0.01 ;
        sal:add_offset = 30. ;
    double pressure(time, z) ;

The file has no longitude and no platform code, so the mapping supplies both as defaults:

# mooring.toml
[dimensions]
profile = "time"
level   = "z"

[variables]
time      = "time"
latitude  = "lat"
temp      = "temperature"
temp_qc   = "temperature_flag"
psal      = "sal"
pres      = "pressure"

[defaults]
platform_code = "MOORING_1"
longitude     = 20.0
data_mode     = "D"       # our own, quality-controlled data
ctddump convert generic --config mooring.toml mooring.nc mooring.parquet
ctddump report parquet mooring.parquet     # check the result

Time is converted from seconds since 1970 to the output’s days since 1950; the fill values become NaN; salinity is unpacked with scale_factor and add_offset; and depth is derived from pressure (flagged with deph_conv = 1).

Example: a cruise

A cruise file with many casts, each with its own ship code, uses the profile-level layout:

dimensions:
    cast = 54 ;
    level = 400 ;
    strlen = 8 ;
variables:
    char ship(cast, strlen) ;
    double time(cast) ;
        time:units = "days since 2000-01-01T00:00:00Z" ;
    double lon(cast) ;
    double lat(cast) ;
    float depth(cast, level) ;
    float TEMP(cast, level) ;
    char TEMP_QC(cast, level) ;
    float PSAL(cast, level) ;
    char PSAL_QC(cast, level) ;
# cruise.toml
layout = "profile-level"

[dimensions]
profile = "cast"
level   = "level"

[variables]
time      = "time"
longitude = "lon"
latitude  = "lat"
deph      = "depth"
# TEMP, TEMP_QC, PSAL and PSAL_QC already have the default names.

[platform]
variable = "ship"

Pressure is derived from depth here. The char QC flags are detected automatically.

Many files at once

batch convert generic converts a directory tree with the same mapping, and header generic / batch header generic export the metadata (dimensions, variables, and all global attributes). The results feed into the usual pipeline:

ctddump batch convert generic --config cruise.toml --output output/convert source/cruises
ctddump batch header  generic --output output/header source/cruises
ctddump concat convert output/convert output/cruises.parquet
ctddump concat header  output/header  output/cruises.yaml
ctddump report parquet output/cruises.parquet

batch matches *.nc by default; set --pattern (or pattern in the mapping file) to narrow it. The helper scripts are written for the Copernicus downloads and do not run the generic format.

The mapping file

Every key is optional. A missing key takes the Copernicus name, so an empty mapping file describes a Copernicus NRT file. Setting a variable name to "" says the file has no such variable. Misspelt keys are rejected rather than ignored.

layout     = "time-depth"   # or "profile-level"
time_units = "..."          # overrides the time variable's units attribute
qc_type    = "int"          # or "char"; detected per variable when omitted
pattern    = "*.nc"         # batch file selection

[dimensions]
profile = "TIME"            # profile-level: N_PROF
level   = "DEPTH"           # profile-level: N_LEVELS

[variables]
time        = "TIME"
longitude   = "LONGITUDE"
latitude    = "LATITUDE"
time_qc     = "TIME_QC"
position_qc = "POSITION_QC"
temp        = "TEMP"        # temp_qc / temp_dm default to "<temp>_QC" / "<temp>_DM"
psal        = "PSAL"        # likewise psal_qc / psal_dm
pres        = "PRES"        # likewise pres_qc / pres_dm
deph        = "DEPH"        # likewise deph_qc / deph_dm
data_mode   = "DATA_MODE"   # per-profile data mode (profile-level layout)

[platform]                  # one of the two
attribute = "platform_code" # time-depth default: a global attribute
# variable = "PLATFORM_NUMBER"  # profile-level default: a per-profile variable

[defaults]                  # used when the file does not provide the value
platform_code = "MY_PLATFORM"
longitude     = 20.0
latitude      = 60.0
data_mode     = "D"         # R, A or D

A few flags override the mapping file, which helps when one mapping serves several platforms:

FlagMapping key
--layout <time-depth|profile-level>layout
--time-units <UNITS>time_units
--platform-code <CODE>[defaults] platform_code
--pattern <GLOB>pattern

The priority is: built-in default, then the --config file, then the flags.

How values are read

  • Missing values. _FillValue becomes NaN; without one, missing_value and then the NetCDF default fill value for the type are used. This applies to time and position as well as to the measurements.
  • Packed values. scale_factor and add_offset are applied.
  • Time. The units may be days, hours, minutes, seconds, or milliseconds since a reference date, with an optional time and time zone ("hours since 2000-01-01 06:00 +01:00"). The calendar must be standard, gregorian, or proleptic_gregorian (or absent). Months and years are not accepted because their length varies.
  • Platform code. A variable may be char [profile, strlen], string [profile], an integer [profile] (e.g. a WMO number), or a scalar of any of these for the whole file. Blank entries take [defaults] platform_code.
  • QC flags. Integer flags 0 to 9 are written as "0" to "9", other values (fill values included) as ""; any integer type works. Char flags are copied as they are, with blanks written as "".
  • Data mode. In the time-depth layout, a [profile, level] char variable per parameter (TEMP_DM); in the profile-level layout, one per-profile variable (DATA_MODE). Without one, the parameter’s data_mode attribute (R, A or D), then [defaults] data_mode, then "" (unknown). A data_mode attribute of "M" (mixed) gives "" whatever the default, since only a per-level variable can say which levels are which. Each file whose data mode is left empty this way gets a warning; set [defaults] data_mode when you know the mode of your data.
  • Pressure and depth. Whichever is missing is derived from the other with TEOS-10 at the profile’s latitude, and flagged in pres_conv / deph_conv.

Limits and workarounds

Your fileWhat to do
A single cast with only a level dimension, TEMP(depth)Add a profile dimension of length 1, e.g. with NCO: ncecat -O -u cast in.nc out.nc
Measurements stored [level, profile]Reorder the dimensions: ncpdq -a cast,level in.nc out.nc
Position or time per observation (a trajectory)Not supported: the output has one position and one time per profile.
A profile name or ID variableNot carried into the output. Profiles are identified by platform_code, profile_no, and time.

Troubleshooting

ctddump checks the file against the mapping before converting and names the problem:

MessageFix
dimension "depth" not found (the file has: ...)Set [dimensions] profile / level to names from the list.
time variable "TIME" not foundSet [variables] time.
time variable "time" has no units attributeSet time_units (or pass --time-units).
unsupported time units "..."Use a "<unit> since <date>" form; see How values are read.
longitude variable "LONGITUDE" not foundSet [variables] longitude, or [defaults] longitude for a fixed position.
platform global attribute "platform_code" not foundSet [platform] attribute / variable, or [defaults] platform_code.
neither pressure "PRES" nor depth "DEPH" foundSet [variables] pres or deph.
variable "X" has dimensions [...]; expected [...]The variable has the wrong shape; see Limits and workarounds.
unknown field ...A misspelt key in the mapping file.

ctddump header generic my_data.nc my_data.yaml lists every dimension and variable in a file, which helps when writing the mapping.

Helper scripts

The scripts/ directory ships seven Bash scripts that automate the full regional pipeline, the same steps documented one command at a time in the Regional workflows. They run in seven phases, each consuming the previous phases’ output:

PhaseScriptWhat it does
1. Downloaddownload_data.shDownload the source NetCDF from Copernicus Marine into source/.
2. Convertconvert_data.shConvert + merge to Parquet, export + merge headers.
3. Cleanclean_data.shDrop bad-QC profiles, drop all-NA profiles, restrict to the region.
4. De-duplicatededup_data.shMark duplicate profiles, then remove them.
5. Comparecompare_data.shCross-compare each region’s products (NRT, GL, CORA) for coverage.
6. Summarisesummary_data.shBuild a per-unit Markdown/HTML summary page from the reports.
7. Publishsummary_site.shRender the Markdown summary pages into a static mdBook site.

Each phase handles the Arctic, Baltic, and Mediterranean regions. The scripts only orchestrate the ctddump binary, so it must be on your PATH, except download_data.sh, which instead needs the copernicusmarine toolbox (and a free Copernicus Marine account) and does not use ctddump at all, and summary_site.sh, which needs mdBook (cargo install mdbook).

The Compare phase feeds the Summarise phase: compare_data.sh writes a coverage summary for each pair of products, and summary_data.sh folds the rows that name a page’s own product into that page’s Comparison section. It is otherwise self-contained, so you can also run it on its own to inspect the coverage TSVs directly.

Running a script

Run the scripts from a working directory (e.g. ctddump). Source NetCDF is read from source/, data products are written under output/, and summary reports under report/; all are created as needed. They share one interface:

scripts/<script>.sh [options] [command] [region ...]
  • command: the step (or all) to run; each script defaults to its most common command when omitted (see below).
  • region: one or more of arctic, baltic, mediterranean. Omitting them (or passing all) runs every region.
  • options: may appear anywhere on the line, in either --out DIR or --out=DIR form.

Before doing any work, a script prints the resolved configuration and asks for confirmation:

$ scripts/clean_data.sh dropqc
Configuration:
  command : dropqc
  regions : arctic baltic mediterranean
  files   : 8
  out     : output
  report  : report
  chunk   : default
  mode    : parallel (per file)
Run with -h/--help to see all options.
Proceed? [y/N]

Answer y to proceed; anything else (including a bare Enter) aborts. Pass -y/--yes to skip the prompt, required when running non-interactively (a pipe or CI job), where the script otherwise aborts with a hint rather than hang. Every script closes its configuration block with the -h/--help reminder shown above, so the full option list is always one step away. While running, each step is announced with a timestamp so you can see what is currently in progress.

Parallelism

The selected work runs in parallel by default, since the units are independent. Each worker tags its log lines with its region so the interleaved output stays readable:

[12:00:01] [arctic] dropqc nrt_ar_ar
[12:00:01] [arctic] dropqc nrt_ar_gl
[12:00:01] [baltic] dropqc nrt_bo_bo

The granularity differs by script:

  • clean_data.sh and dedup_data.sh default to per file: one worker per merged file (a stem within a region), the finest granularity (8 files across the three regions). Each file runs its whole stage chain (e.g. dropqc → dropna → filter → report) in order. Pass --by-region for one worker per region instead (coarser), or --sequential for no parallelism.
  • convert_data.sh defaults to per unit: one worker per (region, product), where each region’s three products are its regional NRT, the Global (GL), and CORA (9 units across the three regions). Each unit runs its own convert → merge → header → merge-header chain in order. Pass --by-region for one worker per region instead (its three products in order), or --sequential for no parallelism.
  • compare_data.sh defaults to per comparison: one worker per (region, comparison), where each region has three comparisons (NRT vs CORA, GL vs CORA, NRT vs GL: 9 across the three regions). Pass --by-region for one worker per region instead (its comparisons in order), or --sequential for no parallelism.
  • download_data.sh parallelises per region; --sequential disables it.

If any unit fails, the others still finish and the script exits non-zero after reporting which unit failed. A run with only one unit is always sequential. Note that convert/clean/dedup each also use multiple threads within a unit, so parallel units multiply the load accordingly, convert_data.sh’s -t/--threads therefore defaults to just 2, since up to 9 units may run at once. Throttle further with --by-region / --sequential. For download_data.sh, --sequential also helps if concurrent Copernicus transfers hit rate limits.

If memory is tight (parallel units each hold a chunk in memory at once), lower the streaming chunk size with --chunk-rows N on convert/clean/dedup, it trades a smaller memory footprint for more Parquet row groups without changing the output. See Configuration for the full list of tuning variables.

Options

OptionScriptsDefaultMeaning
-t, --threads Nconvert2Worker threads for each ctddump call (kept low because up to 9 units run in parallel by default).
-s, --src DIRdownload, convertsourceRoot of the source NetCDF tree (downloaded into / read from).
-s, --src DIRsitesummaryDirectory holding the <stem>.md summary pages to render.
--src DIRcompareoutput/dedup/dedupDirectory holding the input Parquet files to compare.
-o, --out DIRconvert, clean, dedup, summary, compareoutputRoot for the generated / consumed data outputs (summary reads the markdup .dups.tsv from here; compare derives its --src default from it).
-r, --report DIRconvert, clean, dedup, summary, comparereportRoot for the summary TSV reports (a sibling of output/).
-d, --dest DIRsummarysummaryDirectory the generated summary pages are written to.
-d, --dest DIRsitesiteDirectory the built static site is written to.
-c, --config FILEsitebuilt-inCustom book.toml to use instead of the built-in template.
-t, --title TEXTsitectddump: CTD data summary reportsBook title (built-in template only; ignored with --config).
-l, --license FILEsitethe LICENSE beside the scriptLICENSE copied into the built site. Pass --license "" to skip it.
--citation FILEsitethe site-CITATION.cff beside the scriptCITATION.cff copied into the built site. If it has a top-level doi:, the site README also gets a DOI badge and a Citation section. Pass --citation "" to skip it.
-f, --format FMTsummarymdSummary page format: md or html.
--format FMTcomparetsvctddump compare output format: tsv, text, or json (sets the report file extension).
--no-platform-keycompareoffMatch profiles on time and position only, ignoring platform_code.
--chunk-rows Nconvert, clean, dedup, comparectddump defaultStreaming chunk size in rows: lower uses less memory but writes/reads more row groups. Exported as CTDDUMP_CHUNK_ROWS for every ctddump process the script launches.
--by-regionconvert, clean, dedup, compareoffParallelise per region instead of per unit/file/comparison.
--sequentialconvert, clean, dedup, compareoffProcess one unit at a time (no parallelism).
--timeconvert, clean, dedupoffMeasure each ctddump step with GNU time and log its wall clock and peak memory (see Timing steps).
--time-log FILEconvert, clean, dedupoffWrite the --time measurements to FILE (implies --time), so the screen keeps only the normal progress output.
-y, --yesalloffSkip the confirmation prompt.
-h, --helpalloffShow the script’s help.

Timing steps

convert_data.sh, clean_data.sh, and dedup_data.sh accept --time (off by default). When on, every ctddump step is wrapped in GNU time and its wall-clock seconds and peak resident memory are logged as a timed <step>: …s, … MiB peak RSS, …% CPU line after the step completes:

[12:00:03] dropqc nrt_ar_ar
[12:00:15] timed dropqc nrt_ar_ar: 11.72s, 512 MiB peak RSS, 148% CPU

Notes:

  • It requires GNU time (the time package: sudo apt-get install time), not the shell time builtin, which cannot report memory. The scripts resolve /usr/bin/time (or gtime), and check it up front so a missing tool fails fast rather than mid-run. Override the binary with the CTDDUMP_TIME_BIN environment variable.
  • Peak RSS is measured per ctddump process. A multi-step stage such as the multi-box filter logs one line per underlying process (… (include), … (exclude 1), …).
  • Under the default parallelism, wall-clock times of concurrent steps overlap and CPU% can exceed 100% (each ctddump is itself multi-threaded). For clean, comparable per-step wall times, pair --time with --sequential; peak RSS is per process and meaningful either way.
  • The measurements share the log stream (stderr) with the normal progress, so a plain redirect captures both (... --time --sequential -y all 2> run.log). To keep them apart, pass --time-log FILE: the timed … lines go to FILE (which implies --time and is truncated fresh each run) while the screen keeps only the normal progress. In parallel mode each worker appends its own lines, so the file holds every step’s timing, ordered by completion.

Missing datasets

Every stage tolerates missing inputs, so an unavailable dataset does not fail the run. A ctddump batch or concat step that matches no files reports this and writes nothing; clean_data.sh / dedup_data.sh skip any file whose input is missing (with a note). The concrete case today is the Global (GL) product for the Baltic Sea, which Copernicus does not yet publish: the nrt_bo_gl outputs are simply skipped, and appear automatically once the data becomes available.

download_data.sh

Downloads the source NetCDF from Copernicus Marine. Commands: login, download (default). Needs the copernicusmarine toolbox (not ctddump).

scripts/download_data.sh login             # one-time Copernicus login
scripts/download_data.sh                   # download every region into source/
scripts/download_data.sh download arctic   # just the Arctic

Each product’s directory is downloaded under source/ (override with -s/--src), ready for convert_data.sh.

convert_data.sh

Converts the downloaded NetCDF to Parquet and metadata. Commands: process, report, all (default). The default runs process and then report, so the summaries in report/ always match the data just converted; process alone leaves any earlier reports in place unchanged.

scripts/convert_data.sh                    # process + report, all regions
scripts/convert_data.sh process arctic     # just convert + merge the Arctic
scripts/convert_data.sh report             # rewrite the reports from the merged outputs

Reads the source NetCDF from source/ (-s/--src); writes merged Parquet to output/convert/, merged headers to output/header/, and summaries to report/convert/ (Parquet data) and report/header/ (header YAML).

The merge runs concat convert with its defaults, so the merged Parquet has each profile’s observations sorted by increasing pressure, and observations without a pressure value are dropped. Every later phase works on this order. See Row order and numbering.

Because the merge hides the source order, the report command also runs checkpres on the per-file Parquet each merge was built from (output/convert/<region>/<product>/, kept after the merge) and writes report/convert/checkpres/<stem>.profiles.tsv, .platforms.tsv and .summary.tsv: the profiles whose pressure does not increase in the source order. The summary page shows them in its Pressure order section.

clean_data.sh

Cleans the merged Parquet from phase 1. Commands: dropqc, dropna, filter, report, all (default). The stages chain output/convert → clean/dropqc → clean/dropna → clean/filter, with a summary of each stage in report/clean/{dropqc,dropna,filter}/. The filter step applies each region’s bounding box(es).

scripts/clean_data.sh                       # dropqc → dropna → filter → report, all regions
scripts/clean_data.sh -y all baltic         # skip the prompt, Baltic only

dedup_data.sh

Removes duplicate profiles from the cleaned data. Commands: markdup, report, dedup, all (default). It reads output/clean/filter, writes marked data to output/dedup/markdup/ (plus a duplicates TSV) and de-duplicated data to output/dedup/dedup/, with summaries under report/dedup/{markdup,dedup}/.

scripts/dedup_data.sh                        # markdup → report → dedup → report, all regions

compare_data.sh

Cross-compares each region’s de-duplicated products, so the summary pages (and you) can see how far the products of the same waters overlap. It reads phase 4’s output from output/dedup/dedup (--src to point elsewhere) and writes a compare two-way coverage summary per pair to report/compare/. Three comparisons run per region, selected by command:

  • nrt-cora: regional NRT (AR/BO/MO) vs CORA.
  • gl-cora: NRT Global (GL) subset vs CORA.
  • nrt-gl: regional NRT vs NRT Global.
  • all (default): all three.

Profiles are matched on ctddump compare’s default key (platform code, the time as a date, and longitude/latitude to 3 decimals); pass --no-platform-key to match on time and position alone. Choose the output with --format tsv|text|json (the report file extension follows). Like the other pipeline scripts it runs the comparisons in parallel (one worker each), skips a comparison whose input is missing (so a region’s absent Global product does not fail the run), and honours --by-region / --sequential / -y.

scripts/compare_data.sh                        # all three comparisons, every region
scripts/compare_data.sh nrt-cora arctic        # just NRT vs CORA for the Arctic
scripts/compare_data.sh --format text -y all   # aligned tables, no prompt

The next phase, summary_data.sh, reads these TSVs: each page’s Comparison section shows the rows whose reference is that page’s own product. Keep the default --format tsv for that; text and json are for reading the coverage directly.

summary_data.sh

Builds one report summary page per (region, product) unit from the TSV reports the earlier phases produced: Markdown by default, --format html for HTML. It reads reports from report/ (-r/--report) and the markdup .dups.tsv from output/ (-o/--out), and writes one page per stem to summary/ (-d/--dest). A stem with no report files (e.g. the not-yet-published Baltic GL product) is skipped, not an error.

Each page includes a Comparison section built from the compare_data.sh output under report/compare/: the rows whose reference is this page’s own product, showing how far each other product of the same region covers it. Run compare_data.sh before this phase to populate it; without those files the section is simply omitted.

scripts/summary_data.sh                       # summary/<stem>.md for every unit
scripts/summary_data.sh -y -f html baltic     # HTML pages, Baltic only, no prompt

The script gives each page a human-readable title and any product-specific notes (passed to report summary as --title / --note). Both live in the Page text section near the top of the script, title_for and notes_for, one case arm per stem, and that is the place to edit what a page says about a region or dataset. The section prose on the page is generic and comes from ctddump itself.

summary_site.sh

Renders the Markdown pages from summary_data.sh into a static local web site with mdBook. Pages are read from summary/ (-s/--src) and the built site is written to site/ (-d/--dest); open site/index.html in a browser, or serve the directory as-is.

Chapters are grouped into one part per region, in the same order the other scripts use. Each chapter’s name comes from the page’s own top-level heading (the title summary_data.sh gave it), so titles are defined in exactly one place. A region whose pages are absent is skipped rather than left as an empty part; if no pages are found at all, that is an error rather than an empty site.

The book is assembled in a temporary directory, so only the rendered site is left behind. This phase needs the Markdown pages, so run summary_data.sh with its default -f md (an HTML run has nothing for mdBook to render).

The built directory is made self-contained and ready to publish: mdBook writes a .nojekyll (so GitHub Pages serves its FontAwesome/ and other asset folders), and the script adds a LICENSE (the project’s own by default, -l/--license FILE to override or --license "" to skip), the site’s CITATION.cff (see below), and a short README.md describing the site. This is what makes the output directory suitable to push straight to a publishing repository such as ctddump-report-example.

mdBook empties the site directory before every build, so anything added to it by hand, such as a CITATION.cff or an edited README, is gone after the next run. The site’s citation therefore lives in this repository, as scripts/site-CITATION.cff, and every build copies it in as CITATION.cff and regenerates the README’s DOI badge from its doi:. It describes the published example site (ctddump-report-example, with that site’s Zenodo DOI), so edit it there when the site’s metadata changes. If you publish a site of your own, pass your own file with --citation FILE, or --citation "" to leave it out.

A live example of a built site is published at https://aiqc-hub.github.io/ctddump-report-example/ (a static, point-in-time sample of this phase’s output).

scripts/summary_site.sh                      # site/ from summary/, every region
scripts/summary_site.sh -y arctic            # Arctic pages only, no prompt
scripts/summary_site.sh -t "Arctic CTD QC"   # override the book title
scripts/summary_site.sh --citation my-CITATION.cff   # your own site's citation and DOI badge

By default the script writes a book.toml for you. Pass -c/--config FILE to use your own instead; it is used verbatim, so it must keep mdBook’s default src = "src", the script assembles the chapters into a src/ directory beside it. With --config, --title is ignored (your file sets the title).

scripts/summary_site.sh -c my-book.toml -y all

Full pipeline

Run the seven phases in order (skipping prompts) for every region. Log in once first if you have not already (scripts/download_data.sh login):

scripts/download_data.sh -y            # download every region into source/
scripts/convert_data.sh  -y all
scripts/clean_data.sh    -y all
scripts/dedup_data.sh    -y all
scripts/compare_data.sh  -y all        # report/compare/<code>_<a>_vs_<b>.tsv
scripts/summary_data.sh  -y all        # folds the comparison into each page
scripts/summary_site.sh  -y all        # site/index.html

For the equivalent commands spelled out step by step, see the Regional workflows.

Arctic Sea

An end-to-end workflow for the Arctic Sea in two phases: data preparation (download, convert, merge, and export the metadata) and data cleaning (drop low-quality profiles and restrict the data to the region).

Data preparation

Prerequisites

Downloading requires a free Copernicus Marine account and the Copernicus Marine Toolbox (documentation), which provides the copernicusmarine command used below.

Run everything from a working directory (e.g. ctddump). Downloads land under source/; ctddump writes data products under output/ and summary reports under report/ (all created as needed). Create source, enter it, and log in once:

mkdir source
cd source
copernicusmarine login

1. Download the data

# NRT: Arctic (AR) and Global (GL)
copernicusmarine get -i cmems_obs-ins_arc_phybgcwav_mynrt_na_irr --dataset-part "history" --filter "*/CT/*"

# CORA: Arctic
copernicusmarine get -i cmems_obs-ins_glo_phy-temp-sal_my_cora_irr --filter "arctic/*/*_PR_CT.nc"

# Back to the working root; the steps below use source/, output/, and report/ relative to it.
cd ..

2. Convert NetCDF to Parquet

# NRT AR
ctddump batch convert nrt_ar --threads 10 --output output/convert/ar/ar source

# NRT GL
ctddump batch convert nrt_gl --threads 10 --output output/convert/ar/gl source/INSITU_ARC_PHYBGCWAV_DISCRETE_MYNRT_013_031

# CORA AR
ctddump batch convert cora --threads 10 --output output/convert/ar/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/arctic

3. Merge the Parquet files

First list the profiles whose pressure does not increase in the source order, since the merge sorts by pressure and hides it (see checkpres):

# NRT AR
ctddump checkpres --threads 10 output/convert/ar/ar report/convert/checkpres/nrt_ar_ar

# NRT GL
ctddump checkpres --threads 10 output/convert/ar/gl report/convert/checkpres/nrt_ar_gl

# CORA AR
ctddump checkpres --threads 10 output/convert/ar/cora report/convert/checkpres/cora_ar

Then merge:

# NRT AR
ctddump concat convert --threads 10 output/convert/ar/ar output/convert/nrt_ar_ar.parquet

# NRT GL
ctddump concat convert --threads 10 output/convert/ar/gl output/convert/nrt_ar_gl.parquet

# CORA AR
ctddump concat convert --threads 10 output/convert/ar/cora output/convert/cora_ar.parquet

The merge sorts each profile’s observations by increasing pressure and drops observations without a pressure value; add --no-pres-sort to keep the source order (see Row order and numbering).

4. Export the metadata (headers)

# NRT AR
ctddump batch header nrt --threads 10 --pattern "AR_PR_CT_*.nc" --output output/header/ar/ar source/INSITU_ARC_PHYBGCWAV_DISCRETE_MYNRT_013_031

# NRT GL
ctddump batch header nrt --threads 10 --pattern "GL_PR_CT_*.nc" --output output/header/ar/gl source/INSITU_ARC_PHYBGCWAV_DISCRETE_MYNRT_013_031

# CORA AR
ctddump batch header cora --threads 10 --output output/header/ar/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/arctic

5. Merge the header files

# NRT AR
ctddump concat header output/header/ar/ar output/header/nrt_ar_ar.yaml

# NRT GL
ctddump concat header output/header/ar/gl output/header/nrt_ar_gl.yaml

# CORA AR
ctddump concat header output/header/ar/cora output/header/cora_ar.yaml

6. Summarise the results

Write a platform-level summary of each merged Parquet file and a per-file summary of each merged header YAML (as TSV).

mkdir -p report/convert report/header

# NRT AR
ctddump report parquet --level platform output/convert/nrt_ar_ar.parquet report/convert/nrt_ar_ar.parquet.tsv
ctddump report yaml output/header/nrt_ar_ar.yaml report/header/nrt_ar_ar.yaml.tsv

# NRT GL
ctddump report parquet --level platform output/convert/nrt_ar_gl.parquet report/convert/nrt_ar_gl.parquet.tsv
ctddump report yaml output/header/nrt_ar_gl.yaml report/header/nrt_ar_gl.yaml.tsv

# CORA AR
ctddump report parquet --level platform output/convert/cora_ar.parquet report/convert/cora_ar.parquet.tsv
ctddump report yaml output/header/cora_ar.yaml report/header/cora_ar.yaml.tsv

Data cleaning

Clean the merged Parquet from the preparation phase by dropping low-quality profiles and restricting the data to the region. Each step reads the previous step’s output, so the stages chain dropqc → dropna → filter.

Create the output directories:

mkdir -p output/clean/dropqc output/clean/dropna output/clean/filter \
         report/clean/dropqc report/clean/dropna report/clean/filter

1. Drop profiles with bad profile-level QC

Drop profiles whose time_qc or position_qc is a present, non-OK flag; profiles that are OK ("1") or have missing QC are kept.

# NRT AR
ctddump dropqc output/convert/nrt_ar_ar.parquet output/clean/dropqc/nrt_ar_ar.parquet

# NRT GL
ctddump dropqc output/convert/nrt_ar_gl.parquet output/clean/dropqc/nrt_ar_gl.parquet

# CORA AR
ctddump dropqc output/convert/cora_ar.parquet output/clean/dropqc/cora_ar.parquet

2. Drop profiles with no usable data

Drop profiles that are entirely NA in any of temp, psal, or pres.

# NRT AR
ctddump dropna output/clean/dropqc/nrt_ar_ar.parquet output/clean/dropna/nrt_ar_ar.parquet

# NRT GL
ctddump dropna output/clean/dropqc/nrt_ar_gl.parquet output/clean/dropna/nrt_ar_gl.parquet

# CORA AR
ctddump dropna output/clean/dropqc/cora_ar.parquet output/clean/dropna/cora_ar.parquet

3. Filter to the Arctic region

Keep only profiles inside the Arctic bounding box (longitude -180 to 180, latitude 60 to 90).

# NRT AR
ctddump filter --min-lon -180 --max-lon 180 --min-lat 60 --max-lat 90 output/clean/dropna/nrt_ar_ar.parquet output/clean/filter/nrt_ar_ar.parquet

# NRT GL
ctddump filter --min-lon -180 --max-lon 180 --min-lat 60 --max-lat 90 output/clean/dropna/nrt_ar_gl.parquet output/clean/filter/nrt_ar_gl.parquet

# CORA AR
ctddump filter --min-lon -180 --max-lon 180 --min-lat 60 --max-lat 90 output/clean/dropna/cora_ar.parquet output/clean/filter/cora_ar.parquet

4. Summarise the cleaned data

Summarise each cleaning stage (as TSV), mirroring the data layout under report/clean/.

# after dropqc
ctddump report parquet --level platform output/clean/dropqc/nrt_ar_ar.parquet report/clean/dropqc/nrt_ar_ar.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/nrt_ar_gl.parquet report/clean/dropqc/nrt_ar_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/cora_ar.parquet   report/clean/dropqc/cora_ar.parquet.tsv

# after dropna
ctddump report parquet --level platform output/clean/dropna/nrt_ar_ar.parquet report/clean/dropna/nrt_ar_ar.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/nrt_ar_gl.parquet report/clean/dropna/nrt_ar_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/cora_ar.parquet   report/clean/dropna/cora_ar.parquet.tsv

# after filter
ctddump report parquet --level platform output/clean/filter/nrt_ar_ar.parquet report/clean/filter/nrt_ar_ar.parquet.tsv
ctddump report parquet --level platform output/clean/filter/nrt_ar_gl.parquet report/clean/filter/nrt_ar_gl.parquet.tsv
ctddump report parquet --level platform output/clean/filter/cora_ar.parquet   report/clean/filter/cora_ar.parquet.tsv

Data de-duplication

De-duplicate the cleaned Parquet from the previous phase. Two profiles are duplicates when they share the same date and position (longitude/latitude rounded to 3 decimals), ctddump’s defaults, across platforms. markdup flags them (and lists them in a TSV); dedup removes them, keeping the profile with the most observations.

Create the output directories:

mkdir -p output/dedup/markdup output/dedup/dedup report/dedup/markdup report/dedup/dedup

1. Mark duplicate profiles

# NRT AR
ctddump markdup output/clean/filter/nrt_ar_ar.parquet output/dedup/markdup/nrt_ar_ar.parquet output/dedup/markdup/nrt_ar_ar.dups.tsv

# NRT GL
ctddump markdup output/clean/filter/nrt_ar_gl.parquet output/dedup/markdup/nrt_ar_gl.parquet output/dedup/markdup/nrt_ar_gl.dups.tsv

# CORA AR
ctddump markdup output/clean/filter/cora_ar.parquet output/dedup/markdup/cora_ar.parquet output/dedup/markdup/cora_ar.dups.tsv

2. Summarise the marked data (duplicate counts)

# NRT AR
ctddump report parquet --level platform output/dedup/markdup/nrt_ar_ar.parquet report/dedup/markdup/nrt_ar_ar.parquet.tsv

# NRT GL
ctddump report parquet --level platform output/dedup/markdup/nrt_ar_gl.parquet report/dedup/markdup/nrt_ar_gl.parquet.tsv

# CORA AR
ctddump report parquet --level platform output/dedup/markdup/cora_ar.parquet report/dedup/markdup/cora_ar.parquet.tsv

3. Remove duplicate profiles

# NRT AR
ctddump dedup output/dedup/markdup/nrt_ar_ar.parquet output/dedup/dedup/nrt_ar_ar.parquet

# NRT GL
ctddump dedup output/dedup/markdup/nrt_ar_gl.parquet output/dedup/dedup/nrt_ar_gl.parquet

# CORA AR
ctddump dedup output/dedup/markdup/cora_ar.parquet output/dedup/dedup/cora_ar.parquet

4. Summarise the de-duplicated data

# NRT AR
ctddump report parquet --level platform output/dedup/dedup/nrt_ar_ar.parquet report/dedup/dedup/nrt_ar_ar.parquet.tsv

# NRT GL
ctddump report parquet --level platform output/dedup/dedup/nrt_ar_gl.parquet report/dedup/dedup/nrt_ar_gl.parquet.tsv

# CORA AR
ctddump report parquet --level platform output/dedup/dedup/cora_ar.parquet report/dedup/dedup/cora_ar.parquet.tsv

The pipeline is automated by scripts/download_data.sh, scripts/convert_data.sh, scripts/clean_data.sh, and scripts/dedup_data.sh. See Helper scripts for their commands and options.

Baltic Sea

An end-to-end workflow for the Baltic Sea in two phases: data preparation (download, convert, merge, and export the metadata) and data cleaning (drop low-quality profiles and restrict the data to the region).

This workflow uses the regional NRT (BO), Global (GL), and CORA products. Copernicus does not yet publish the Global (GL) data for the Baltic, so the GL steps currently match no files, ctddump reports this and writes nothing, and the cleaning / de-duplication steps skip the missing nrt_bo_gl outputs, activating automatically once GL becomes available. The manual commands below therefore cover BO and CORA; the helper scripts additionally run the (currently empty) GL steps.

Data preparation

Prerequisites

Downloading requires a free Copernicus Marine account and the Copernicus Marine Toolbox (documentation), which provides the copernicusmarine command used below.

Run everything from a working directory (e.g. ctddump). Downloads land under source/; ctddump writes data products under output/ and summary reports under report/ (all created as needed). Create source, enter it, and log in once:

mkdir source
cd source
copernicusmarine login

1. Download the data

# NRT: Baltic (BO)
copernicusmarine get -i cmems_obs-ins_bal_phybgcwav_mynrt_na_irr --dataset-part "history" --filter "*/CT/*"

# CORA: Baltic
copernicusmarine get -i cmems_obs-ins_glo_phy-temp-sal_my_cora_irr --filter "baltic/*/*_PR_CT.nc"

# Back to the working root; the steps below use source/, output/, and report/ relative to it.
cd ..

2. Convert NetCDF to Parquet

# NRT BO
ctddump batch convert nrt_bo --threads 10 --output output/convert/bo/bo source

# CORA BO
ctddump batch convert cora --threads 10 --output output/convert/bo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/baltic

3. Merge the Parquet files

First list the profiles whose pressure does not increase in the source order, since the merge sorts by pressure and hides it (see checkpres):

# NRT BO
ctddump checkpres --threads 10 output/convert/bo/bo report/convert/checkpres/nrt_bo_bo

# CORA BO
ctddump checkpres --threads 10 output/convert/bo/cora report/convert/checkpres/cora_bo

Then merge:

# NRT BO
ctddump concat convert --threads 10 output/convert/bo/bo output/convert/nrt_bo_bo.parquet

# CORA BO
ctddump concat convert --threads 10 output/convert/bo/cora output/convert/cora_bo.parquet

The merge sorts each profile’s observations by increasing pressure and drops observations without a pressure value; add --no-pres-sort to keep the source order (see Row order and numbering).

4. Export the metadata (headers)

# NRT BO
ctddump batch header nrt --threads 10 --pattern "BO_PR_CT_*.nc" --output output/header/bo/bo source/INSITU_BAL_PHYBGCWAV_DISCRETE_MYNRT_013_032

# CORA BO
ctddump batch header cora --threads 10 --output output/header/bo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/baltic

5. Merge the header files

# NRT BO
ctddump concat header output/header/bo/bo output/header/nrt_bo_bo.yaml

# CORA BO
ctddump concat header output/header/bo/cora output/header/cora_bo.yaml

6. Summarise the results

Write a platform-level summary of each merged Parquet file and a per-file summary of each merged header YAML (as TSV).

mkdir -p report/convert report/header

# NRT BO
ctddump report parquet --level platform output/convert/nrt_bo_bo.parquet report/convert/nrt_bo_bo.parquet.tsv
ctddump report yaml output/header/nrt_bo_bo.yaml report/header/nrt_bo_bo.yaml.tsv

# CORA BO
ctddump report parquet --level platform output/convert/cora_bo.parquet report/convert/cora_bo.parquet.tsv
ctddump report yaml output/header/cora_bo.yaml report/header/cora_bo.yaml.tsv

Data cleaning

Clean the merged Parquet from the preparation phase by dropping low-quality profiles and restricting the data to the region. Each step reads the previous step’s output, so the stages chain dropqc → dropna → filter.

Create the output directories:

mkdir -p output/clean/dropqc output/clean/dropna output/clean/filter \
         report/clean/dropqc report/clean/dropna report/clean/filter

1. Drop profiles with bad profile-level QC

Drop profiles whose time_qc or position_qc is a present, non-OK flag; profiles that are OK ("1") or have missing QC are kept.

# NRT BO
ctddump dropqc output/convert/nrt_bo_bo.parquet output/clean/dropqc/nrt_bo_bo.parquet

# CORA BO
ctddump dropqc output/convert/cora_bo.parquet output/clean/dropqc/cora_bo.parquet

2. Drop profiles with no usable data

Drop profiles that are entirely NA in any of temp, psal, or pres.

# NRT BO
ctddump dropna output/clean/dropqc/nrt_bo_bo.parquet output/clean/dropna/nrt_bo_bo.parquet

# CORA BO
ctddump dropna output/clean/dropqc/cora_bo.parquet output/clean/dropna/cora_bo.parquet

3. Filter to the Baltic region

Keep profiles inside the Baltic bounding box (longitude 6 to 30, latitude 53 to 66), then exclude the sub-box (longitude 6 to 15, latitude 60 to 66). The include step writes an intermediate .box.parquet file that the exclude step consumes to produce the final cleaned file.

# NRT BO
ctddump filter --min-lon 6 --max-lon 30 --min-lat 53 --max-lat 66 output/clean/dropna/nrt_bo_bo.parquet output/clean/filter/nrt_bo_bo.box.parquet
ctddump filter --mode exclude --min-lon 6 --max-lon 15 --min-lat 60 --max-lat 66 output/clean/filter/nrt_bo_bo.box.parquet output/clean/filter/nrt_bo_bo.parquet

# CORA BO
ctddump filter --min-lon 6 --max-lon 30 --min-lat 53 --max-lat 66 output/clean/dropna/cora_bo.parquet output/clean/filter/cora_bo.box.parquet
ctddump filter --mode exclude --min-lon 6 --max-lon 15 --min-lat 60 --max-lat 66 output/clean/filter/cora_bo.box.parquet output/clean/filter/cora_bo.parquet

4. Summarise the cleaned data

Summarise each cleaning stage (as TSV), mirroring the data layout under report/clean/.

# after dropqc
ctddump report parquet --level platform output/clean/dropqc/nrt_bo_bo.parquet report/clean/dropqc/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/cora_bo.parquet   report/clean/dropqc/cora_bo.parquet.tsv

# after dropna
ctddump report parquet --level platform output/clean/dropna/nrt_bo_bo.parquet report/clean/dropna/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/cora_bo.parquet   report/clean/dropna/cora_bo.parquet.tsv

# after filter
ctddump report parquet --level platform output/clean/filter/nrt_bo_bo.parquet report/clean/filter/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/filter/cora_bo.parquet   report/clean/filter/cora_bo.parquet.tsv

Data de-duplication

De-duplicate the cleaned Parquet from the previous phase. Two profiles are duplicates when they share the same date and position (longitude/latitude rounded to 3 decimals), ctddump’s defaults, across platforms. markdup flags them (and lists them in a TSV); dedup removes them, keeping the profile with the most observations.

Create the output directories:

mkdir -p output/dedup/markdup output/dedup/dedup report/dedup/markdup report/dedup/dedup

1. Mark duplicate profiles

# NRT BO
ctddump markdup output/clean/filter/nrt_bo_bo.parquet output/dedup/markdup/nrt_bo_bo.parquet output/dedup/markdup/nrt_bo_bo.dups.tsv

# CORA BO
ctddump markdup output/clean/filter/cora_bo.parquet output/dedup/markdup/cora_bo.parquet output/dedup/markdup/cora_bo.dups.tsv

2. Summarise the marked data (duplicate counts)

# NRT BO
ctddump report parquet --level platform output/dedup/markdup/nrt_bo_bo.parquet report/dedup/markdup/nrt_bo_bo.parquet.tsv

# CORA BO
ctddump report parquet --level platform output/dedup/markdup/cora_bo.parquet report/dedup/markdup/cora_bo.parquet.tsv

3. Remove duplicate profiles

# NRT BO
ctddump dedup output/dedup/markdup/nrt_bo_bo.parquet output/dedup/dedup/nrt_bo_bo.parquet

# CORA BO
ctddump dedup output/dedup/markdup/cora_bo.parquet output/dedup/dedup/cora_bo.parquet

4. Summarise the de-duplicated data

# NRT BO
ctddump report parquet --level platform output/dedup/dedup/nrt_bo_bo.parquet report/dedup/dedup/nrt_bo_bo.parquet.tsv

# CORA BO
ctddump report parquet --level platform output/dedup/dedup/cora_bo.parquet report/dedup/dedup/cora_bo.parquet.tsv

The pipeline is automated by scripts/download_data.sh, scripts/convert_data.sh, scripts/clean_data.sh, and scripts/dedup_data.sh. See Helper scripts for their commands and options.

Mediterranean Sea

An end-to-end workflow for the Mediterranean Sea in two phases: data preparation (download, convert, merge, and export the metadata) and data cleaning (drop low-quality profiles and restrict the data to the region).

Data preparation

Prerequisites

Downloading requires a free Copernicus Marine account and the Copernicus Marine Toolbox (documentation), which provides the copernicusmarine command used below.

Run everything from a working directory (e.g. ctddump). Downloads land under source/; ctddump writes data products under output/ and summary reports under report/ (all created as needed). Create source, enter it, and log in once:

mkdir source
cd source
copernicusmarine login

1. Download the data

# NRT: Mediterranean (MO) and Global (GL)
copernicusmarine get -i cmems_obs-ins_med_phybgcwav_mynrt_na_irr --dataset-part "history" --filter "*/CT/*"

# CORA: Mediterranean
copernicusmarine get -i cmems_obs-ins_glo_phy-temp-sal_my_cora_irr --filter "mediterrane/*/*_PR_CT.nc"

# Back to the working root; the steps below use source/, output/, and report/ relative to it.
cd ..

2. Convert NetCDF to Parquet

# NRT MO
ctddump batch convert nrt_mo --threads 10 --output output/convert/mo/mo source

# NRT GL
ctddump batch convert nrt_gl --threads 10 --output output/convert/mo/gl source/INSITU_MED_PHYBGCWAV_DISCRETE_MYNRT_013_035

# CORA MO
ctddump batch convert cora --threads 10 --output output/convert/mo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/mediterrane

3. Merge the Parquet files

First list the profiles whose pressure does not increase in the source order, since the merge sorts by pressure and hides it (see checkpres):

# NRT MO
ctddump checkpres --threads 10 output/convert/mo/mo report/convert/checkpres/nrt_mo_mo

# NRT GL
ctddump checkpres --threads 10 output/convert/mo/gl report/convert/checkpres/nrt_mo_gl

# CORA MO
ctddump checkpres --threads 10 output/convert/mo/cora report/convert/checkpres/cora_mo

Then merge:

# NRT MO
ctddump concat convert --threads 10 output/convert/mo/mo output/convert/nrt_mo_mo.parquet

# NRT GL
ctddump concat convert --threads 10 output/convert/mo/gl output/convert/nrt_mo_gl.parquet

# CORA MO
ctddump concat convert --threads 10 output/convert/mo/cora output/convert/cora_mo.parquet

The merge sorts each profile’s observations by increasing pressure and drops observations without a pressure value; add --no-pres-sort to keep the source order (see Row order and numbering).

4. Export the metadata (headers)

# NRT MO
ctddump batch header nrt --threads 10 --pattern "MO_PR_CT_*.nc" --output output/header/mo/mo source/INSITU_MED_PHYBGCWAV_DISCRETE_MYNRT_013_035

# NRT GL
ctddump batch header nrt --threads 10 --pattern "GL_PR_CT_*.nc" --output output/header/mo/gl source/INSITU_MED_PHYBGCWAV_DISCRETE_MYNRT_013_035

# CORA MO
ctddump batch header cora --threads 10 --output output/header/mo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/mediterrane

5. Merge the header files

# NRT MO
ctddump concat header output/header/mo/mo output/header/nrt_mo_mo.yaml

# NRT GL
ctddump concat header output/header/mo/gl output/header/nrt_mo_gl.yaml

# CORA MO
ctddump concat header output/header/mo/cora output/header/cora_mo.yaml

6. Summarise the results

Write a platform-level summary of each merged Parquet file and a per-file summary of each merged header YAML (as TSV).

mkdir -p report/convert report/header

# NRT MO
ctddump report parquet --level platform output/convert/nrt_mo_mo.parquet report/convert/nrt_mo_mo.parquet.tsv
ctddump report yaml output/header/nrt_mo_mo.yaml report/header/nrt_mo_mo.yaml.tsv

# NRT GL
ctddump report parquet --level platform output/convert/nrt_mo_gl.parquet report/convert/nrt_mo_gl.parquet.tsv
ctddump report yaml output/header/nrt_mo_gl.yaml report/header/nrt_mo_gl.yaml.tsv

# CORA MO
ctddump report parquet --level platform output/convert/cora_mo.parquet report/convert/cora_mo.parquet.tsv
ctddump report yaml output/header/cora_mo.yaml report/header/cora_mo.yaml.tsv

Data cleaning

Clean the merged Parquet from the preparation phase by dropping low-quality profiles and restricting the data to the region. Each step reads the previous step’s output, so the stages chain dropqc → dropna → filter.

Create the output directories:

mkdir -p output/clean/dropqc output/clean/dropna output/clean/filter \
         report/clean/dropqc report/clean/dropna report/clean/filter

1. Drop profiles with bad profile-level QC

Drop profiles whose time_qc or position_qc is a present, non-OK flag; profiles that are OK ("1") or have missing QC are kept.

# NRT MO
ctddump dropqc output/convert/nrt_mo_mo.parquet output/clean/dropqc/nrt_mo_mo.parquet

# NRT GL
ctddump dropqc output/convert/nrt_mo_gl.parquet output/clean/dropqc/nrt_mo_gl.parquet

# CORA MO
ctddump dropqc output/convert/cora_mo.parquet output/clean/dropqc/cora_mo.parquet

2. Drop profiles with no usable data

Drop profiles that are entirely NA in any of temp, psal, or pres.

# NRT MO
ctddump dropna output/clean/dropqc/nrt_mo_mo.parquet output/clean/dropna/nrt_mo_mo.parquet

# NRT GL
ctddump dropna output/clean/dropqc/nrt_mo_gl.parquet output/clean/dropna/nrt_mo_gl.parquet

# CORA MO
ctddump dropna output/clean/dropqc/cora_mo.parquet output/clean/dropna/cora_mo.parquet

3. Filter to the Mediterranean region

Keep profiles inside the Mediterranean bounding box (longitude -5.61 to 35.567, latitude 28.378 to 45.755), then exclude two sub-boxes: (longitude 27 to 36, latitude 41 to 46) and (longitude -5.61 to 0, latitude 42 to 46). Each stage chains through an intermediate .box*.parquet file to produce the final cleaned file.

# NRT MO
ctddump filter --min-lon -5.61 --max-lon 35.567 --min-lat 28.378 --max-lat 45.755 output/clean/dropna/nrt_mo_mo.parquet output/clean/filter/nrt_mo_mo.box1.parquet
ctddump filter --mode exclude --min-lon 27 --max-lon 36 --min-lat 41 --max-lat 46 output/clean/filter/nrt_mo_mo.box1.parquet output/clean/filter/nrt_mo_mo.box2.parquet
ctddump filter --mode exclude --min-lon -5.61 --max-lon 0 --min-lat 42 --max-lat 46 output/clean/filter/nrt_mo_mo.box2.parquet output/clean/filter/nrt_mo_mo.parquet

# NRT GL
ctddump filter --min-lon -5.61 --max-lon 35.567 --min-lat 28.378 --max-lat 45.755 output/clean/dropna/nrt_mo_gl.parquet output/clean/filter/nrt_mo_gl.box1.parquet
ctddump filter --mode exclude --min-lon 27 --max-lon 36 --min-lat 41 --max-lat 46 output/clean/filter/nrt_mo_gl.box1.parquet output/clean/filter/nrt_mo_gl.box2.parquet
ctddump filter --mode exclude --min-lon -5.61 --max-lon 0 --min-lat 42 --max-lat 46 output/clean/filter/nrt_mo_gl.box2.parquet output/clean/filter/nrt_mo_gl.parquet

# CORA MO
ctddump filter --min-lon -5.61 --max-lon 35.567 --min-lat 28.378 --max-lat 45.755 output/clean/dropna/cora_mo.parquet output/clean/filter/cora_mo.box1.parquet
ctddump filter --mode exclude --min-lon 27 --max-lon 36 --min-lat 41 --max-lat 46 output/clean/filter/cora_mo.box1.parquet output/clean/filter/cora_mo.box2.parquet
ctddump filter --mode exclude --min-lon -5.61 --max-lon 0 --min-lat 42 --max-lat 46 output/clean/filter/cora_mo.box2.parquet output/clean/filter/cora_mo.parquet

4. Summarise the cleaned data

Summarise each cleaning stage (as TSV), mirroring the data layout under report/clean/.

# after dropqc
ctddump report parquet --level platform output/clean/dropqc/nrt_mo_mo.parquet report/clean/dropqc/nrt_mo_mo.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/nrt_mo_gl.parquet report/clean/dropqc/nrt_mo_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/cora_mo.parquet   report/clean/dropqc/cora_mo.parquet.tsv

# after dropna
ctddump report parquet --level platform output/clean/dropna/nrt_mo_mo.parquet report/clean/dropna/nrt_mo_mo.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/nrt_mo_gl.parquet report/clean/dropna/nrt_mo_gl.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/cora_mo.parquet   report/clean/dropna/cora_mo.parquet.tsv

# after filter
ctddump report parquet --level platform output/clean/filter/nrt_mo_mo.parquet report/clean/filter/nrt_mo_mo.parquet.tsv
ctddump report parquet --level platform output/clean/filter/nrt_mo_gl.parquet report/clean/filter/nrt_mo_gl.parquet.tsv
ctddump report parquet --level platform output/clean/filter/cora_mo.parquet   report/clean/filter/cora_mo.parquet.tsv

Data de-duplication

De-duplicate the cleaned Parquet from the previous phase. Two profiles are duplicates when they share the same date and position (longitude/latitude rounded to 3 decimals), ctddump’s defaults, across platforms. markdup flags them (and lists them in a TSV); dedup removes them, keeping the profile with the most observations.

Create the output directories:

mkdir -p output/dedup/markdup output/dedup/dedup report/dedup/markdup report/dedup/dedup

1. Mark duplicate profiles

# NRT MO
ctddump markdup output/clean/filter/nrt_mo_mo.parquet output/dedup/markdup/nrt_mo_mo.parquet output/dedup/markdup/nrt_mo_mo.dups.tsv

# NRT GL
ctddump markdup output/clean/filter/nrt_mo_gl.parquet output/dedup/markdup/nrt_mo_gl.parquet output/dedup/markdup/nrt_mo_gl.dups.tsv

# CORA MO
ctddump markdup output/clean/filter/cora_mo.parquet output/dedup/markdup/cora_mo.parquet output/dedup/markdup/cora_mo.dups.tsv

2. Summarise the marked data (duplicate counts)

# NRT MO
ctddump report parquet --level platform output/dedup/markdup/nrt_mo_mo.parquet report/dedup/markdup/nrt_mo_mo.parquet.tsv

# NRT GL
ctddump report parquet --level platform output/dedup/markdup/nrt_mo_gl.parquet report/dedup/markdup/nrt_mo_gl.parquet.tsv

# CORA MO
ctddump report parquet --level platform output/dedup/markdup/cora_mo.parquet report/dedup/markdup/cora_mo.parquet.tsv

3. Remove duplicate profiles

# NRT MO
ctddump dedup output/dedup/markdup/nrt_mo_mo.parquet output/dedup/dedup/nrt_mo_mo.parquet

# NRT GL
ctddump dedup output/dedup/markdup/nrt_mo_gl.parquet output/dedup/dedup/nrt_mo_gl.parquet

# CORA MO
ctddump dedup output/dedup/markdup/cora_mo.parquet output/dedup/dedup/cora_mo.parquet

4. Summarise the de-duplicated data

# NRT MO
ctddump report parquet --level platform output/dedup/dedup/nrt_mo_mo.parquet report/dedup/dedup/nrt_mo_mo.parquet.tsv

# NRT GL
ctddump report parquet --level platform output/dedup/dedup/nrt_mo_gl.parquet report/dedup/dedup/nrt_mo_gl.parquet.tsv

# CORA MO
ctddump report parquet --level platform output/dedup/dedup/cora_mo.parquet report/dedup/dedup/cora_mo.parquet.tsv

The pipeline is automated by scripts/download_data.sh, scripts/convert_data.sh, scripts/clean_data.sh, and scripts/dedup_data.sh. See Helper scripts for their commands and options.