Workspace organisation for oceanographic data pipelines at NPI#
An example recipe for organising code, data, and documentation across the full lifecycle of a field dataset: from raw instrument output through processing and internal collaboration to long-term public archiving on the NPI Data Centre (NPDC, data.npolar.no).
Intended audience: researchers at NPI (or similar institutes) working with shipboard or moored oceanographic instruments, running Windows with OneDrive and a network disk, publishing to a CF/ACDD-compliant institutional archive.
Largely written by Claude, supervised and final-edited by Tore
The issue#
As soon as it goes beyond the personal hard drive, our data-workflow involves tools and places like OneDrive, NPDATA, data.npolar.no, gitlab, etc. It is not always clear where to put what, how to keep track of all these data locations. Even with a clean structure, juggling these different storage locationscreates real cognitive overhead, where it is hard to remember why things are where they are or find something six months later.
The core principle: separate code, working data, and archive#
The honest answer is that there’s no silver bullet, but the mess mostly comes from one root cause: data has a lifecycle, and each phase genuinely belongs in a different place. The friction you feel is real and unavoidable to some degree. What you can do is make the lifecycle explicit so the locations feel logical rather than arbitrary. Here’s how one can think about it:
Code, working data, and archive copies are three separate things. They have different audiences, different lifetimes, and different access patterns. Mixing them leads to processed files scattered across the code repo, raw data on a network share nobody external can reach, and published files that are indistinguishable from work-in-progress.
Raw collection → Processing → Internal sharing → Pre-publication → Public archive
Network disk Net disk + OneDrive / NPDC restricted NPDC public
(read-only) GitLab code OneDrive share (DOI assigned) (open access)
Phase |
Where |
Who can access |
Gate to next phase |
|---|---|---|---|
Raw data |
Network disk |
NPI internal |
Nothing — lives here permanently |
Processing code |
NPI GitLab (+ GitHub mirror) |
Internal / named external |
Tag release when done |
Working outputs |
OneDrive shared folder |
You + named collaborators |
Copy final file to NPDC |
Pre-publication |
NPDC restricted |
Named collaborators via portal |
Paper accepted |
Published |
NPDC public |
Everyone |
Done — frozen |
Keeping track of everything: DATA_LOCATIONS.md#
The practical thing that helps most: a single plain-text file in the repo called DATA_LOCATIONS.md, updated as each cruise progresses. A plain committed file — the single source of truth for where every cruise’s data lives and what state it is in. Update it as work progresses.
# Data locations
## TT23
- Raw ADCP: \\Npdata\...\TT23\raw\
- Processed .mat: \\Npdata\...\TT23\processed\
- CTD source: C:/thinf/github/troll_transect_ctd_2022_2023/data/final/
- Working output: OneDrive\<project>_working\data\final\<dataset>_2022_2023_v1.nc
- GitLab: gitlab.npolar.no/npiocean/<project> tag: v1.0-TT23
- GitHub: github.com/npi-ocean/<project> tag: v1.0-TT23
- NPDC: doi:10.21334/npolar.XXXXXXX status: PUBLIC since: 2025-11-12
## TT24
- Raw ADCP: \\Npdata\...\TT24\raw\
- Processed .mat: \\Npdata\...\TT24\processed\
- Working output: OneDrive\<project>_working\data\final\<dataset>_2023_2024_v1.nc
- GitLab: gitlab.npolar.no/npiocean/<project> tag: not yet tagged
- NPDC: not yet submitted status: IN PROGRESS
The status field (NOT STARTED / IN PROGRESS / PUBLIC) gives an
at-a-glance view of the whole dataset series.
LADCP Example folder structure#
\\Npdata\...\<cruise>\ ← institutional network disk
raw/ ← original instrument files, read-only
processed/ ← instrument processing output
CTD/
C:/thinf/github/<project>_publish/ ← git repo, NOT on OneDrive (see note)
processing_utils.py ← shared processing functions
configs/ ← per-cruise YAML configs
notebooks/ ← one notebook per cruise
README.md
DATA_LOCATIONS.md ← single source of truth (see below)
local_paths_template.yaml ← committed; users copy to local_paths.yaml
local_paths.yaml ← LOCAL ONLY — never committed
requirements.txt ← updated at publish time
.gitignore
OneDrive/<project>_working/ ← working outputs only, no .git folder
data/
preliminary/ ← intermediate drafts
final/ ← files ready for NPDC
figures/ ← QC plots
OneDrive/<project>_shared/ ← shared with external collaborators
README.md
<dataset>_v1.nc
Why the repo is NOT on OneDrive: OneDrive’s continuous sync conflicts with git’s file locking. It can hold a lock during sync causing git to fail, or sync a partially-written git object and corrupt the
.gitfolder. Keep the repo on a local path and let GitHub/GitLab be the backup for code. OneDrive then only holds the working data folder, which syncs cleanly.
Version control: NPI GitLab primary, GitHub as mirror#
NPI runs its own GitLab at gitlab.npolar.no/npiocean — this is the
canonical home for all processing code. For external collaboration or
public access after publication, selected repos are mirrored to GitHub.
gitlab.npolar.no/npiocean/<project> ← primary: all development
↓ push mirror (on tag or manually)
github.com/npi-ocean/<project> ← mirror: external collaboration
↓ zip at publish time
NPDC dataset record ← archived snapshot
Tag on GitLab first, then push the tag to GitHub:
git tag -a v1.0-TT23 -m "Version used to produce <dataset>_v1.nc"
git push origin v1.0-TT23 # GitLab
git push github v1.0-TT23 # GitHub mirror
What to archive on NPDC#
Self-documenting NetCDF attributes#
The final published file should be self-documenting and nclude provenance in the global attributes:
processing_software: "LDEO LADCP processing toolbox Version IX_15"
processing_software_url: "https://github.com/jgrelet/ladcp"
processing_code: "See processing/ folder in this dataset record"
processing_code_repository: "https://gitlab.npolar.no/npiocean/<project>"
processing_code_version: "v1.0-TT23 (commit abc1234)"
tidal_model: "CATS2008 (Padman et al., 2002; doi:10.3189/172756402781817752)"
history: |
2023-01-20 to 2023-01-31: Data collection.
LDEO LADCP processing toolbox Version IX_15.
Tidal correction: CATS2008.
2025-11-12T10:22:00Z: NetCDF file created.
as well as refrences to other relevant published documents (e.g. cruise report on NVA).
Archive the full processing chain — all layers#
No oceanographer would argue for deleting raw .hex files from a CTD after generating a publishable netcdf file. So where should we keep those? On a hrad drive in your office? On a network drive with an ever so cluttering folder sturcture? Why not in a subfolder (or .zip file) together with the finally published data file? – Ideally toether with an unambigious description of how to transform from one to another. The same logic applies to any instrument: raw files are the primary scientific record. Everything downstream is derived. For LADCP, the analogy is exact:
CTD |
LADCP |
|---|---|
|
|
|
instrument config (embedded in |
SBE processing scripts + |
LDEO |
|
|
Python QC/calibration pipeline |
this Python pipeline |
Published NetCDF |
published NetCDF |
Discarding any layer removes the ability to reprocess with better tools or correct an error found later. Raw files are not large by modern standards (LADCP: 2–10 GB per cruise; CTD hex: < 1 GB), and the long-term storage cost is trivial compared to the ship time that generated them.
The gold standard is to archive full reproducibility, but at least the blueprint (.hex) and a schematic a processing description should be archived together with the final data file.
NPDC dataset structure#
NPDC supports folder structures within dataset records. A complete record could look like this (LADCP example; CTD is analogous):
NPDC dataset: <dataset>_2022_2023_v1
│
├── <dataset>_2022_2023_v1.nc ← published, CF-compliant data
│
├── raw.zip/ ← primary instrument record
│ ├── TT23_001.000 ← RDI raw binary, one per cast
│ └── ...
│
├── instrument_processed.zip/ ← instrument toolbox output
│ ├── mat_files/
│ │ ├── 001_dt.mat ← detided velocity profiles
│ │ └── ...
│ ├── logs/
│ │ ├── 001.log ← full processing log per cast
│ │ └── ...
│ ├── params/
│ │ ├── default.m ← default LDEO parameters
│ │ └── cast_overrides.m ← any per-cast overrides
│ └── INSTRUMENT_PROCESSING.md ← software version, MATLAB version,
│ how to reproduce this step
│
└── processing.zip/ ← Python pipeline (this repo snapshot)
├── PROVENANCE.md
├── processing_utils.py
├── configs/<cruise>.yaml
├── notebooks/<cruise>_make_nc.ipynb
├── local_paths_template.yaml
└── requirements.txt
For code, don’t put whole git repos (with history etc.) into processing.zip — just what’s needed to understand and reproduce the processing for that specific dataset version:
PROVENANCE.md#
Documents the Python pipeline step for netcdf generation from LDEO-processed .mat files. It ties the snapshot back to the live git repo, which might change over time:
# Provenance
Code used to produce <dataset>_2022_2023_v1.nc, archived alongside the data.
Repository: https://gitlab.npolar.no/npiocean/<project>
Mirror: https://github.com/npi-ocean/<project>
Commit: abc1234def5678
Tag: v1.0-TT23
Date: 2025-11-12
To reproduce:
1. Clone the repository at the tagged commit
2. Copy local_paths_template.yaml to local_paths.yaml and fill in local paths
3. pip install -r requirements.txt
4. Run notebooks/<cruise>_make_nc.ipynb
The commit hash is the critical piece — it pins the exact code state regardless of how the repository evolves after publication.
requirements.txt#
Generate at publish time to pin exact package versions:
pip freeze > requirements.txt # or: conda env export > environment.yml
Library behaviour changes across versions — xarray’s time encoding, fill value handling, and NetCDF writing have all changed in ways that can produce subtly different output. Pinning versions means a future user starts from a known-good environment.
INSTRUMENT_PROCESSING.md#
Documents the instrument toolbox step — the part not covered by the Python pipeline for netcdf generation. Keep it short:
# Instrument processing environment
Software: LDEO LADCP processing toolbox Version IX_15
Source: https://github.com/jgrelet/ladcp (commit abc1234)
MATLAB: R2023b, Windows 11
Parameters in params/ are defaults except where noted in individual .log files.
Per-cast overrides are in cast_overrides.m.
To reproduce: install MATLAB R2023b, clone the toolbox at the commit above,
add to path, run process_cast.m with the parameter files in params/.
Portable paths#
Avoid hardcoding absolute paths in config files wherever possible. Use a local local_paths.yaml
(gitignored) with tokens substituted at load time:
# local_paths.yaml — local only, never committed
processed_root: "\\Npdata\...\TrollTransect"
ctd_root: "C:/thinf/github"
output_root: "C:/Users/.../OneDrive/<project>_working/data"
# configs/TT23.yaml — uses tokens, safe to commit
flist_glob: "{processed_root}/TT23/processed/Detide/*_dt.mat"
input_ctd_nc: "{ctd_root}/troll_transect_ctd_2022_2023/data/final/TrollTransectCTD_2022_2023_v1.nc"
output_nc: "{output_root}/final/<dataset>_2022_2023_v1.nc"
Include a committed local_paths_template.yaml so new users know what to fill in.
.gitignore essentials#
*.nc
*.mat
*.000
*.hex
*.cnv
*.ps
data/
local_paths.yaml
__pycache__/
*.pyc
.ipynb_checkpoints/
One strong recommendation: Install nbstripout once per clone to strip notebook outputs before commits:
pip install nbstripout && nbstripout --install
It automatically strips cell outputs from notebooks before every git commit, so you don’t accidentally commit large arrays or file paths that contain usernames/server names into the repo history.
Should I include .cnv files from ctd processing and .mat/ .netcdf files from LADCP processing in the git repos?#
Short answer: technically yes, but you shouldn’t, and for two different reasons depending on file type.
Binary files (.mat, .nc, .000, .hex) — git stores a complete copy of every version of every file forever. Binary files can’t be diffed, so git gives you none of the benefits (you can’t see what changed between versions) but all of the costs (the repo balloons in size permanently; even if you delete the file later, it stays in the history). A 50 MB .mat file committed once is 50 MB in the repo forever.
Text-based processed files (.cnv) — .cnv files are ASCII, so git can diff them, which is slightly better. But they are still derived outputs — generated from .hex + .psa parameters. Committing derived outputs to a repo creates a maintenance problem: the file in the repo can drift out of sync with the code that generates it, and it’s never clear whether the committed version is the canonical one or just someone’s local run.
The general rule is: commit code and configuration, not data or derived outputs, regardless of file size or format.
What to do instead:
For sharing processed files with collaborators before NPDC archiving, use the OneDrive shared folder (Option A from the recipe). For long-term archiving, use the NPDC dataset record with its folder structure — that’s exactly the right place for .cnv, .mat, and .nc files.
The one partial exception: git-lfs
Git Large File Storage (git-lfs) is an extension that stores large files outside the repo history and replaces them with small pointer files. It’s supported by both GitHub and GitLab, including gitlab.npolar.no. It removes the repo-bloat problem for binary files.
Whether it’s worth it depends on the use case:
Regular git |
git-lfs |
|
|---|---|---|
Binary files ( |
✗ bloats repo permanently |
✓ stored externally |
Can diff file contents |
✗ for binary |
✗ still binary |
Collaborators need data alongside code |
manual sharing |
✓ |
Works on NPI GitLab |
— |
✓ supported |
Adds setup complexity |
— |
moderate |
For your workflow specifically I’d still lean against lfs for the data files, because the NPDC archiving step already solves the problem properly — the data has a home with a DOI, access control, and long-term preservation guarantees that a git remote doesn’t provide. Using lfs to store .mat files in GitLab would be a parallel, less well-managed copy of data that already lives on the network disk and will end up on NPDC.
The one case where lfs genuinely helps is small reference files that belong conceptually with the code — for example, a small lookup table, a calibration coefficient file, or a short test dataset used in unit tests. Those are fine in lfs because they are inputs to the code rather than outputs of it.
So the short version to add to the recipe: .cnv, .mat, .nc stay out of git. They go to OneDrive for sharing and NPDC for archiving.
References#
Padman, L., Fricker, H.A., Coleman, R., Howard, S., and Erofeeva, L. (2002). A new tide model for the Antarctic ice shelves and seas. Annals of Glaciology, 34, 247–254. doi:10.3189/172756402781817752
Howard, S.L., Greene, C.A., Padman, L., Erofeeva, S., and Sutterley, T. (2024). CATS2008_v2023: Circum-Antarctic Tidal Simulation 2008, version 2023. U.S. Antarctic Program Data Center. doi:10.15784/601772