Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

checkpres

Find profiles whose pressure does not increase in observation order, and write three TSV reports: the profiles, their platforms, and totals.

ctddump checkpres [OPTIONS] <src> <out_prefix>

<src> is a directory of Parquet files, read the same way as by concat convert, or a single Parquet file.

When to run it

Run it on the per-file Parquet written by batch convert, before concat convert merges them. The merge sorts every profile by increasing pressure (see Row order and numbering), so afterwards the original order can no longer be checked. Before the merge, rows are still in the source file’s order and observation_no is the level index in the file.

Run it once per merge, on the same directory, so the reports cover exactly what ends up in one merged file:

ctddump checkpres output/convert/ar/ar report/convert/checkpres/nrt_ar_ar
ctddump concat convert output/convert/ar/ar output/convert/nrt_ar_ar.parquet

What is checked

  • A profile is one (filename, platform_code, profile_no). A file may hold many profiles and many platforms; each profile is checked on its own.
  • Within a profile, each observation that has a pressure is compared with the previous observation that had one. Missing pressures are skipped.
  • A decrease is a step to a lower pressure. A repeated pressure is not a decrease.
  • A profile with no usable pressure at all has nothing to check. It is left out of the counts and reported only as n_no_pres, so the counts match the profiles concat convert keeps (it drops observations without pressure).
  • A profile with at least one decrease is reported, as one of two kinds:
KindMeaning
reversedPressure never goes up: the profile is stored from the bottom up. Sorting by pressure turns it the right way round.
non_monotonicPressure goes both up and down: loops, spikes, or levels stored out of order. These are the profiles to look at.

Options

OptionDefaultMeaning
--pattern <GLOB>*.parquetFilename glob used when <src> is a directory
--min-drop <DBAR>0Ignore decreases no larger than this, for sensor noise
--pres-qc <FLAGS>every pressureUse only pressures whose pres_qc is one of these flags, e.g. 1,2
-t, --threads <N>all coresNumber of files checked in parallel; the reports are the same for any value

Outputs

Three tab-separated files, named after <out_prefix>:

  1. <out_prefix>.profiles.tsv: one row per reported profile, sorted by platform and time.

    ColumnMeaning
    filename, platform_code, profile_nothe profile, as in the per-file Parquet
    profile_timestamp, profile_time, longitude, latitudewhen and where; with platform_code, these also find the profile after the merge
    kindreversed or non_monotonic
    n_obs, n_presobservations, and observations with a pressure used by the check
    n_decreasesnumber of decreasing steps
    first_decrease_obsobservation_no where the pressure first drops
    max_decreaselargest drop, in dbar
    pres_first, pres_last, pres_min, pres_maxpressure range, in dbar
    pres_conv1 when some pressure in the profile was derived from depth
  2. <out_prefix>.platforms.tsv: one row per platform with at least one reported profile: platform_code, n_files, n_profiles, n_bad_profiles, pct_bad, n_reversed, n_non_monotonic, first_date, last_date. The dates are those of its first and last reported profile.

  3. <out_prefix>.summary.tsv: one row for the whole source: n_files, n_platforms, n_profiles, n_no_pres, n_bad_profiles, pct_bad, n_reversed, n_non_monotonic, n_bad_platforms, n_bad_files. n_profiles and n_platforms count only what was checked; n_no_pres is the number of profiles without any usable pressure.

The output directory is created if needed. A one-line summary is printed to standard error. Finding problems is not an error: the exit status is 0. When no file matches, or the directory does not exist, the command says so and writes nothing.

Examples

# Check a directory of converted files
ctddump checkpres ./parquet reports/arctic

# Ignore drops of 0.1 dbar or less, and pressures not flagged good
ctddump checkpres --min-drop 0.1 --pres-qc 1,2 ./parquet reports/arctic

# Check one file
ctddump checkpres AR_PR_CT_58KN.parquet reports/58kn

Files read one at a time, in fixed row windows (CTDDUMP_CHUNK_ROWS), so memory follows the size of one file plus the reported profiles.