Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ctddump

ctddump is a small, fast command-line tool for converting oceanographic CTD (Conductivity, Temperature, Depth) data from NetCDF into analysis-ready formats:

  • Parquet: the observation data, as a uniform flat table.
  • YAML: the file metadata (dimensions, variables, global attributes).

It is written in Rust, streams data in bounded memory, and processes whole directory trees in parallel.

Data sources

Two families of source files are supported:

SourceDescription
NRTNear Real Time: Arctic Sea (nrt_ar), Baltic Sea (nrt_bo), Mediterranean Sea (nrt_mo), Global (nrt_gl)
CORACopernicus Ocean Reanalysis: current format (cora), legacy format (cora_legacy)
Your ownAny NetCDF laid out like the Copernicus files, described by a mapping file (generic); see Your own NetCDF files

What you can do

CommandPurpose
convertConvert a single NetCDF file to Parquet.
batchConvert a whole directory tree to Parquet or YAML, in parallel.
headerExtract a single file’s metadata to YAML.
checkpresFind profiles whose pressure does not increase in observation order.
concatMerge many Parquet (or YAML) files into one.
reportSummarise a Parquet or YAML file as a text report.
filterFilter a Parquet file’s profiles by a geographic bounding box.
dropnaDrop profiles that are entirely NA in any of temp/psal/pres.
dropqcDrop profiles flagged bad in time_qc or position_qc.
markdupMark duplicate profiles (by date + position) with an is_dup column.
dedupRemove duplicate profiles, keeping the one with the most observations.
compareReport how far each of two Parquet files covers the other’s platforms and profiles.
sqliteExport a Parquet file to a SQLite database (platform, profile, observation tables).
completionsPrint a shell tab-completion script for bash, zsh, fish, elvish, or powershell.

Quick example

# One file to Parquet
ctddump convert nrt_ar input.nc output.parquet

# A whole Arctic directory, then merge into a single Parquet file
ctddump batch convert nrt_ar --output ./parquet ./netcdf
ctddump concat convert ./parquet arctic.parquet

To see what the full pipeline produces, browse the live example report site, built with the helper scripts.

New here? Start with Installation, skim the Commands, then follow an end-to-end regional workflow.