Helper scripts
The scripts/
directory ships seven Bash scripts that automate the full regional pipeline,
the same steps documented one command at a time in the
Regional workflows. They run in seven phases, each
consuming the previous phases’ output:
| Phase | Script | What it does |
|---|---|---|
| 1. Download | download_data.sh | Download the source NetCDF from Copernicus Marine into source/. |
| 2. Convert | convert_data.sh | Convert + merge to Parquet, export + merge headers. |
| 3. Clean | clean_data.sh | Drop bad-QC profiles, drop all-NA profiles, restrict to the region. |
| 4. De-duplicate | dedup_data.sh | Mark duplicate profiles, then remove them. |
| 5. Compare | compare_data.sh | Cross-compare each region’s products (NRT, GL, CORA) for coverage. |
| 6. Summarise | summary_data.sh | Build a per-unit Markdown/HTML summary page from the reports. |
| 7. Publish | summary_site.sh | Render the Markdown summary pages into a static mdBook site. |
Each phase handles the Arctic, Baltic, and Mediterranean regions.
The scripts only orchestrate the ctddump binary, so it must be on your PATH,
except download_data.sh, which instead needs the copernicusmarine toolbox
(and a free Copernicus Marine account) and does not use ctddump at all, and
summary_site.sh, which needs mdBook
(cargo install mdbook).
The Compare phase feeds the Summarise phase: compare_data.sh writes a coverage
summary for each pair of products, and summary_data.sh folds the rows that name
a page’s own product into that page’s Comparison section. It is otherwise
self-contained, so you can also run it on its own to inspect the coverage TSVs
directly.
Running a script
Run the scripts from a working directory (e.g. ctddump). Source NetCDF is read
from source/, data products are written under output/, and summary reports
under report/; all are created as needed. They share one interface:
scripts/<script>.sh [options] [command] [region ...]
- command: the step (or
all) to run; each script defaults to its most common command when omitted (see below). - region: one or more of
arctic,baltic,mediterranean. Omitting them (or passingall) runs every region. - options: may appear anywhere on the line, in either
--out DIRor--out=DIRform.
Before doing any work, a script prints the resolved configuration and asks for confirmation:
$ scripts/clean_data.sh dropqc
Configuration:
command : dropqc
regions : arctic baltic mediterranean
files : 8
out : output
report : report
chunk : default
mode : parallel (per file)
Run with -h/--help to see all options.
Proceed? [y/N]
Answer y to proceed; anything else (including a bare Enter) aborts. Pass
-y/--yes to skip the prompt, required when running non-interactively (a
pipe or CI job), where the script otherwise aborts with a hint rather than hang.
Every script closes its configuration block with the -h/--help reminder shown
above, so the full option list is always one step away.
While running, each step is announced with a timestamp so you can see what is
currently in progress.
Parallelism
The selected work runs in parallel by default, since the units are independent. Each worker tags its log lines with its region so the interleaved output stays readable:
[12:00:01] [arctic] dropqc nrt_ar_ar
[12:00:01] [arctic] dropqc nrt_ar_gl
[12:00:01] [baltic] dropqc nrt_bo_bo
The granularity differs by script:
clean_data.shanddedup_data.shdefault to per file: one worker per merged file (a stem within a region), the finest granularity (8 files across the three regions). Each file runs its whole stage chain (e.g.dropqc → dropna → filter → report) in order. Pass--by-regionfor one worker per region instead (coarser), or--sequentialfor no parallelism.convert_data.shdefaults to per unit: one worker per (region, product), where each region’s three products are its regional NRT, the Global (GL), and CORA (9 units across the three regions). Each unit runs its own convert → merge → header → merge-header chain in order. Pass--by-regionfor one worker per region instead (its three products in order), or--sequentialfor no parallelism.compare_data.shdefaults to per comparison: one worker per (region, comparison), where each region has three comparisons (NRT vs CORA, GL vs CORA, NRT vs GL: 9 across the three regions). Pass--by-regionfor one worker per region instead (its comparisons in order), or--sequentialfor no parallelism.download_data.shparallelises per region;--sequentialdisables it.
If any unit fails, the others still finish and the script exits non-zero after
reporting which unit failed. A run with only one unit is always sequential.
Note that convert/clean/dedup each also use multiple threads within a
unit, so parallel units multiply the load accordingly, convert_data.sh’s
-t/--threads therefore defaults to just 2, since up to 9 units may run at
once. Throttle further with --by-region / --sequential. For
download_data.sh, --sequential also helps if concurrent Copernicus transfers
hit rate limits.
If memory is tight (parallel units each hold a chunk in memory at once), lower
the streaming chunk size with --chunk-rows N on convert/clean/dedup, it
trades a smaller memory footprint for more Parquet row groups without changing
the output. See Configuration for the
full list of tuning variables.
Options
| Option | Scripts | Default | Meaning |
|---|---|---|---|
-t, --threads N | convert | 2 | Worker threads for each ctddump call (kept low because up to 9 units run in parallel by default). |
-s, --src DIR | download, convert | source | Root of the source NetCDF tree (downloaded into / read from). |
-s, --src DIR | site | summary | Directory holding the <stem>.md summary pages to render. |
--src DIR | compare | output/dedup/dedup | Directory holding the input Parquet files to compare. |
-o, --out DIR | convert, clean, dedup, summary, compare | output | Root for the generated / consumed data outputs (summary reads the markdup .dups.tsv from here; compare derives its --src default from it). |
-r, --report DIR | convert, clean, dedup, summary, compare | report | Root for the summary TSV reports (a sibling of output/). |
-d, --dest DIR | summary | summary | Directory the generated summary pages are written to. |
-d, --dest DIR | site | site | Directory the built static site is written to. |
-c, --config FILE | site | built-in | Custom book.toml to use instead of the built-in template. |
-t, --title TEXT | site | ctddump: CTD data summary reports | Book title (built-in template only; ignored with --config). |
-l, --license FILE | site | the LICENSE beside the script | LICENSE copied into the built site. Pass --license "" to skip it. |
--citation FILE | site | the site-CITATION.cff beside the script | CITATION.cff copied into the built site. If it has a top-level doi:, the site README also gets a DOI badge and a Citation section. Pass --citation "" to skip it. |
-f, --format FMT | summary | md | Summary page format: md or html. |
--format FMT | compare | tsv | ctddump compare output format: tsv, text, or json (sets the report file extension). |
--no-platform-key | compare | off | Match profiles on time and position only, ignoring platform_code. |
--chunk-rows N | convert, clean, dedup, compare | ctddump default | Streaming chunk size in rows: lower uses less memory but writes/reads more row groups. Exported as CTDDUMP_CHUNK_ROWS for every ctddump process the script launches. |
--by-region | convert, clean, dedup, compare | off | Parallelise per region instead of per unit/file/comparison. |
--sequential | convert, clean, dedup, compare | off | Process one unit at a time (no parallelism). |
--time | convert, clean, dedup | off | Measure each ctddump step with GNU time and log its wall clock and peak memory (see Timing steps). |
--time-log FILE | convert, clean, dedup | off | Write the --time measurements to FILE (implies --time), so the screen keeps only the normal progress output. |
-y, --yes | all | off | Skip the confirmation prompt. |
-h, --help | all | off | Show the script’s help. |
Timing steps
convert_data.sh, clean_data.sh, and dedup_data.sh accept --time (off by
default). When on, every ctddump step is wrapped in GNU
time and its wall-clock seconds and peak
resident memory are logged as a timed <step>: …s, … MiB peak RSS, …% CPU line
after the step completes:
[12:00:03] dropqc nrt_ar_ar
[12:00:15] timed dropqc nrt_ar_ar: 11.72s, 512 MiB peak RSS, 148% CPU
Notes:
- It requires GNU time (the
timepackage:sudo apt-get install time), not the shelltimebuiltin, which cannot report memory. The scripts resolve/usr/bin/time(orgtime), and check it up front so a missing tool fails fast rather than mid-run. Override the binary with theCTDDUMP_TIME_BINenvironment variable. - Peak RSS is measured per
ctddumpprocess. A multi-step stage such as the multi-boxfilterlogs one line per underlying process (… (include),… (exclude 1), …). - Under the default parallelism, wall-clock times of concurrent steps overlap and
CPU% can exceed 100% (each
ctddumpis itself multi-threaded). For clean, comparable per-step wall times, pair--timewith--sequential; peak RSS is per process and meaningful either way. - The measurements share the log stream (stderr) with the normal progress, so a
plain redirect captures both (
... --time --sequential -y all 2> run.log). To keep them apart, pass--time-log FILE: thetimed …lines go to FILE (which implies--timeand is truncated fresh each run) while the screen keeps only the normal progress. In parallel mode each worker appends its own lines, so the file holds every step’s timing, ordered by completion.
Missing datasets
Every stage tolerates missing inputs, so an unavailable dataset does not fail the
run. A ctddump batch or concat step that matches no files reports this and
writes nothing; clean_data.sh / dedup_data.sh skip any file whose input is
missing (with a note). The concrete case today is the Global (GL) product for
the Baltic Sea, which Copernicus does not yet publish: the nrt_bo_gl outputs
are simply skipped, and appear automatically once the data becomes available.
download_data.sh
Downloads the source NetCDF from Copernicus Marine. Commands: login,
download (default). Needs the copernicusmarine toolbox (not ctddump).
scripts/download_data.sh login # one-time Copernicus login
scripts/download_data.sh # download every region into source/
scripts/download_data.sh download arctic # just the Arctic
Each product’s directory is downloaded under source/ (override with
-s/--src), ready for convert_data.sh.
convert_data.sh
Converts the downloaded NetCDF to Parquet and metadata. Commands: process,
report, all (default). The default runs process and then report, so
the summaries in report/ always match the data just converted; process alone
leaves any earlier reports in place unchanged.
scripts/convert_data.sh # process + report, all regions
scripts/convert_data.sh process arctic # just convert + merge the Arctic
scripts/convert_data.sh report # rewrite the reports from the merged outputs
Reads the source NetCDF from source/ (-s/--src); writes merged Parquet to
output/convert/, merged headers to output/header/, and summaries to
report/convert/ (Parquet data) and report/header/ (header YAML).
The merge runs concat convert with its defaults, so the merged Parquet has
each profile’s observations sorted by increasing pressure, and observations
without a pressure value are dropped. Every later phase works on this order.
See Row order and numbering.
Because the merge hides the source order, the report command also runs
checkpres on the per-file Parquet each merge was
built from (output/convert/<region>/<product>/, kept after the merge) and
writes report/convert/checkpres/<stem>.profiles.tsv, .platforms.tsv and
.summary.tsv: the profiles whose pressure does not increase in the source
order. The summary page shows them in its Pressure order section.
clean_data.sh
Cleans the merged Parquet from phase 1. Commands: dropqc, dropna, filter,
report, all (default). The stages chain
output/convert → clean/dropqc → clean/dropna → clean/filter, with a summary of
each stage in report/clean/{dropqc,dropna,filter}/. The filter step applies
each region’s bounding box(es).
scripts/clean_data.sh # dropqc → dropna → filter → report, all regions
scripts/clean_data.sh -y all baltic # skip the prompt, Baltic only
dedup_data.sh
Removes duplicate profiles from the cleaned data. Commands: markdup, report,
dedup, all (default). It reads output/clean/filter, writes marked data to
output/dedup/markdup/ (plus a duplicates TSV) and de-duplicated data to
output/dedup/dedup/, with summaries under report/dedup/{markdup,dedup}/.
scripts/dedup_data.sh # markdup → report → dedup → report, all regions
compare_data.sh
Cross-compares each region’s de-duplicated products, so the summary pages (and
you) can see how far the products of the same waters overlap. It reads phase 4’s
output from output/dedup/dedup (--src to point elsewhere) and writes a
compare two-way coverage summary per pair to
report/compare/. Three comparisons run per region, selected by command:
nrt-cora: regional NRT (AR/BO/MO) vs CORA.gl-cora: NRT Global (GL) subset vs CORA.nrt-gl: regional NRT vs NRT Global.all(default): all three.
Profiles are matched on ctddump compare’s default key (platform code, the time
as a date, and longitude/latitude to 3 decimals); pass --no-platform-key to
match on time and position alone. Choose the output with --format tsv|text|json
(the report file extension follows). Like the other pipeline scripts it runs the
comparisons in parallel (one worker each), skips a comparison whose input is
missing (so a region’s absent Global product does not fail the run), and honours
--by-region / --sequential / -y.
scripts/compare_data.sh # all three comparisons, every region
scripts/compare_data.sh nrt-cora arctic # just NRT vs CORA for the Arctic
scripts/compare_data.sh --format text -y all # aligned tables, no prompt
The next phase, summary_data.sh, reads these TSVs: each
page’s Comparison section shows the rows whose reference is that page’s own
product. Keep the default --format tsv for that; text and json are for
reading the coverage directly.
summary_data.sh
Builds one report summary page per
(region, product) unit from the TSV reports the earlier phases produced:
Markdown by default, --format html for HTML. It reads reports from report/
(-r/--report) and the markdup .dups.tsv from output/ (-o/--out), and
writes one page per stem to summary/ (-d/--dest). A stem with no report files
(e.g. the not-yet-published Baltic GL product) is skipped, not an error.
Each page includes a Comparison section built from the compare_data.sh
output under report/compare/: the rows whose reference is this page’s own
product, showing how far each other product of the same region covers it. Run
compare_data.sh before this phase to populate it; without those files the
section is simply omitted.
scripts/summary_data.sh # summary/<stem>.md for every unit
scripts/summary_data.sh -y -f html baltic # HTML pages, Baltic only, no prompt
The script gives each page a human-readable title and any product-specific
notes (passed to report summary as --title / --note). Both live in the
Page text section near the top of the script, title_for and notes_for, one
case arm per stem, and that is the place to edit what a page says about a
region or dataset. The section prose on the page is generic and comes from
ctddump itself.
summary_site.sh
Renders the Markdown pages from summary_data.sh into a static local web
site with mdBook. Pages are read from
summary/ (-s/--src) and the built site is written to site/ (-d/--dest);
open site/index.html in a browser, or serve the directory as-is.
Chapters are grouped into one part per region, in the same order the other
scripts use. Each chapter’s name comes from the page’s own top-level heading (the
title summary_data.sh gave it), so titles are defined in exactly one place. A
region whose pages are absent is skipped rather than left as an empty part; if no
pages are found at all, that is an error rather than an empty site.
The book is assembled in a temporary directory, so only the rendered site is left
behind. This phase needs the Markdown pages, so run summary_data.sh with its
default -f md (an HTML run has nothing for mdBook to render).
The built directory is made self-contained and ready to publish: mdBook writes a
.nojekyll (so GitHub Pages serves its FontAwesome/ and other asset folders),
and the script adds a LICENSE (the project’s own by default, -l/--license FILE
to override or --license "" to skip), the site’s CITATION.cff (see below), and a
short README.md describing the site. This is what makes
the output directory suitable to push straight to a publishing repository such as
ctddump-report-example.
mdBook empties the site directory before every build, so anything added to it by
hand, such as a CITATION.cff or an edited README, is gone after the next run.
The site’s citation therefore lives in this repository, as
scripts/site-CITATION.cff, and every build copies it in as CITATION.cff and
regenerates the README’s DOI badge from its doi:. It describes the published
example site (ctddump-report-example, with that site’s Zenodo DOI), so edit it
there when the site’s metadata changes. If you publish a site of your own, pass
your own file with --citation FILE, or --citation "" to leave it out.
A live example of a built site is published at https://aiqc-hub.github.io/ctddump-report-example/ (a static, point-in-time sample of this phase’s output).
scripts/summary_site.sh # site/ from summary/, every region
scripts/summary_site.sh -y arctic # Arctic pages only, no prompt
scripts/summary_site.sh -t "Arctic CTD QC" # override the book title
scripts/summary_site.sh --citation my-CITATION.cff # your own site's citation and DOI badge
By default the script writes a book.toml for you. Pass -c/--config FILE to use
your own instead; it is used verbatim, so it must keep mdBook’s default
src = "src", the script assembles the chapters into a src/ directory beside
it. With --config, --title is ignored (your file sets the title).
scripts/summary_site.sh -c my-book.toml -y all
Full pipeline
Run the seven phases in order (skipping prompts) for every region. Log in once
first if you have not already (scripts/download_data.sh login):
scripts/download_data.sh -y # download every region into source/
scripts/convert_data.sh -y all
scripts/clean_data.sh -y all
scripts/dedup_data.sh -y all
scripts/compare_data.sh -y all # report/compare/<code>_<a>_vs_<b>.tsv
scripts/summary_data.sh -y all # folds the comparison into each page
scripts/summary_site.sh -y all # site/index.html
For the equivalent commands spelled out step by step, see the Regional workflows.