Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Baltic Sea

An end-to-end workflow for the Baltic Sea in two phases: data preparation (download, convert, merge, and export the metadata) and data cleaning (drop low-quality profiles and restrict the data to the region).

This workflow uses the regional NRT (BO), Global (GL), and CORA products. Copernicus does not yet publish the Global (GL) data for the Baltic, so the GL steps currently match no files, ctddump reports this and writes nothing, and the cleaning / de-duplication steps skip the missing nrt_bo_gl outputs, activating automatically once GL becomes available. The manual commands below therefore cover BO and CORA; the helper scripts additionally run the (currently empty) GL steps.

Data preparation

Prerequisites

Downloading requires a free Copernicus Marine account and the Copernicus Marine Toolbox (documentation), which provides the copernicusmarine command used below.

Run everything from a working directory (e.g. ctddump). Downloads land under source/; ctddump writes data products under output/ and summary reports under report/ (all created as needed). Create source, enter it, and log in once:

mkdir source
cd source
copernicusmarine login

1. Download the data

# NRT: Baltic (BO)
copernicusmarine get -i cmems_obs-ins_bal_phybgcwav_mynrt_na_irr --dataset-part "history" --filter "*/CT/*"

# CORA: Baltic
copernicusmarine get -i cmems_obs-ins_glo_phy-temp-sal_my_cora_irr --filter "baltic/*/*_PR_CT.nc"

# Back to the working root; the steps below use source/, output/, and report/ relative to it.
cd ..

2. Convert NetCDF to Parquet

# NRT BO
ctddump batch convert nrt_bo --threads 10 --output output/convert/bo/bo source

# CORA BO
ctddump batch convert cora --threads 10 --output output/convert/bo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/baltic

3. Merge the Parquet files

# NRT BO
ctddump concat convert --threads 10 output/convert/bo/bo output/convert/nrt_bo_bo.parquet

# CORA BO
ctddump concat convert --threads 10 output/convert/bo/cora output/convert/cora_bo.parquet

4. Export the metadata (headers)

# NRT BO
ctddump batch header nrt --threads 10 --pattern "BO_PR_CT_*.nc" --output output/header/bo/bo source/INSITU_BAL_PHYBGCWAV_DISCRETE_MYNRT_013_032

# CORA BO
ctddump batch header cora --threads 10 --output output/header/bo/cora source/INSITU_GLO_PHY_TS_DISCRETE_MY_013_001/cmems_obs-ins_glo_phy-temp-sal_my_cora_irr_202511/baltic

5. Merge the header files

# NRT BO
ctddump concat header output/header/bo/bo output/header/nrt_bo_bo.yaml

# CORA BO
ctddump concat header output/header/bo/cora output/header/cora_bo.yaml

6. Summarise the results

Write a platform-level summary of each merged Parquet file and a per-file summary of each merged header YAML (as TSV).

mkdir -p report/convert report/header

# NRT BO
ctddump report parquet --level platform output/convert/nrt_bo_bo.parquet report/convert/nrt_bo_bo.parquet.tsv
ctddump report yaml output/header/nrt_bo_bo.yaml report/header/nrt_bo_bo.yaml.tsv

# CORA BO
ctddump report parquet --level platform output/convert/cora_bo.parquet report/convert/cora_bo.parquet.tsv
ctddump report yaml output/header/cora_bo.yaml report/header/cora_bo.yaml.tsv

Data cleaning

Clean the merged Parquet from the preparation phase by dropping low-quality profiles and restricting the data to the region. Each step reads the previous step’s output, so the stages chain dropqc → dropna → filter.

Create the output directories:

mkdir -p output/clean/dropqc output/clean/dropna output/clean/filter \
         report/clean/dropqc report/clean/dropna report/clean/filter

1. Drop profiles with bad profile-level QC

Drop profiles whose time_qc or position_qc is a present, non-OK flag; profiles that are OK ("1") or have missing QC are kept.

# NRT BO
ctddump dropqc output/convert/nrt_bo_bo.parquet output/clean/dropqc/nrt_bo_bo.parquet

# CORA BO
ctddump dropqc output/convert/cora_bo.parquet output/clean/dropqc/cora_bo.parquet

2. Drop profiles with no usable data

Drop profiles that are entirely NA in any of temp, psal, or pres.

# NRT BO
ctddump dropna output/clean/dropqc/nrt_bo_bo.parquet output/clean/dropna/nrt_bo_bo.parquet

# CORA BO
ctddump dropna output/clean/dropqc/cora_bo.parquet output/clean/dropna/cora_bo.parquet

3. Filter to the Baltic region

Keep profiles inside the Baltic bounding box (longitude 6 to 30, latitude 53 to 66), then exclude the sub-box (longitude 6 to 15, latitude 60 to 66). The include step writes an intermediate .box.parquet file that the exclude step consumes to produce the final cleaned file.

# NRT BO
ctddump filter --min-lon 6 --max-lon 30 --min-lat 53 --max-lat 66 output/clean/dropna/nrt_bo_bo.parquet output/clean/filter/nrt_bo_bo.box.parquet
ctddump filter --mode exclude --min-lon 6 --max-lon 15 --min-lat 60 --max-lat 66 output/clean/filter/nrt_bo_bo.box.parquet output/clean/filter/nrt_bo_bo.parquet

# CORA BO
ctddump filter --min-lon 6 --max-lon 30 --min-lat 53 --max-lat 66 output/clean/dropna/cora_bo.parquet output/clean/filter/cora_bo.box.parquet
ctddump filter --mode exclude --min-lon 6 --max-lon 15 --min-lat 60 --max-lat 66 output/clean/filter/cora_bo.box.parquet output/clean/filter/cora_bo.parquet

4. Summarise the cleaned data

Summarise each cleaning stage (as TSV), mirroring the data layout under report/clean/.

# after dropqc
ctddump report parquet --level platform output/clean/dropqc/nrt_bo_bo.parquet report/clean/dropqc/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/dropqc/cora_bo.parquet   report/clean/dropqc/cora_bo.parquet.tsv

# after dropna
ctddump report parquet --level platform output/clean/dropna/nrt_bo_bo.parquet report/clean/dropna/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/dropna/cora_bo.parquet   report/clean/dropna/cora_bo.parquet.tsv

# after filter
ctddump report parquet --level platform output/clean/filter/nrt_bo_bo.parquet report/clean/filter/nrt_bo_bo.parquet.tsv
ctddump report parquet --level platform output/clean/filter/cora_bo.parquet   report/clean/filter/cora_bo.parquet.tsv

Data de-duplication

De-duplicate the cleaned Parquet from the previous phase. Two profiles are duplicates when they share the same date and position (longitude/latitude rounded to 3 decimals), ctddump’s defaults, across platforms. markdup flags them (and lists them in a TSV); dedup removes them, keeping the profile with the most observations.

Create the output directories:

mkdir -p output/dedup/markdup output/dedup/dedup report/dedup/markdup report/dedup/dedup

1. Mark duplicate profiles

# NRT BO
ctddump markdup output/clean/filter/nrt_bo_bo.parquet output/dedup/markdup/nrt_bo_bo.parquet output/dedup/markdup/nrt_bo_bo.dups.tsv

# CORA BO
ctddump markdup output/clean/filter/cora_bo.parquet output/dedup/markdup/cora_bo.parquet output/dedup/markdup/cora_bo.dups.tsv

2. Summarise the marked data (duplicate counts)

# NRT BO
ctddump report parquet --level platform output/dedup/markdup/nrt_bo_bo.parquet report/dedup/markdup/nrt_bo_bo.parquet.tsv

# CORA BO
ctddump report parquet --level platform output/dedup/markdup/cora_bo.parquet report/dedup/markdup/cora_bo.parquet.tsv

3. Remove duplicate profiles

# NRT BO
ctddump dedup output/dedup/markdup/nrt_bo_bo.parquet output/dedup/dedup/nrt_bo_bo.parquet

# CORA BO
ctddump dedup output/dedup/markdup/cora_bo.parquet output/dedup/dedup/cora_bo.parquet

4. Summarise the de-duplicated data

# NRT BO
ctddump report parquet --level platform output/dedup/dedup/nrt_bo_bo.parquet report/dedup/dedup/nrt_bo_bo.parquet.tsv

# CORA BO
ctddump report parquet --level platform output/dedup/dedup/cora_bo.parquet report/dedup/dedup/cora_bo.parquet.tsv

The pipeline is automated by scripts/download_data.sh, scripts/convert_data.sh, scripts/clean_data.sh, and scripts/dedup_data.sh. See Helper scripts for their commands and options.