# Workspace organisation for oceanographic data pipelines at NPI An example recipe for organising code, data, and documentation across the full lifecycle of a field dataset: from raw instrument output through processing and internal collaboration to long-term public archiving on the NPI Data Centre (NPDC, data.npolar.no). Intended audience: researchers at NPI (or similar institutes) working with shipboard or moored oceanographic instruments, running Windows with OneDrive and a network disk, publishing to a CF/ACDD-compliant institutional archive. Largely written by Claude, supervised and final-edited by Tore --- ## The issue As soon as it goes beyond the personal hard drive, our data-workflow involves tools and places like OneDrive, NPDATA, data.npolar.no, gitlab, etc. It is not always clear where to put what, how to keep track of all these data locations. Even with a clean structure, juggling these different storage locationscreates real cognitive overhead, where it is hard to remember why things are where they are or find something six months later. ## The core principle: separate code, working data, and archive The honest answer is that there's no silver bullet, but the mess mostly comes from one root cause: data has a *lifecycle*, and each phase genuinely belongs in a different place. The friction you feel is real and unavoidable to some degree. What you can do is make the lifecycle explicit so the locations feel logical rather than arbitrary. Here's how one can think about it: **Code, working data, and archive copies are three separate things.** They have different audiences, different lifetimes, and different access patterns. Mixing them leads to processed files scattered across the code repo, raw data on a network share nobody external can reach, and published files that are indistinguishable from work-in-progress. ``` Raw collection → Processing → Internal sharing → Pre-publication → Public archive Network disk Net disk + OneDrive / NPDC restricted NPDC public (read-only) GitLab code OneDrive share (DOI assigned) (open access) ``` | Phase | Where | Who can access | Gate to next phase | |---|---|---|---| | Raw data | Network disk | NPI internal | Nothing — lives here permanently | | Processing code | NPI GitLab (+ GitHub mirror) | Internal / named external | Tag release when done | | Working outputs | OneDrive shared folder | You + named collaborators | Copy final file to NPDC | | Pre-publication | NPDC restricted | Named collaborators via portal | Paper accepted | | Published | NPDC public | Everyone | Done — frozen | --- ## Keeping track of everything: `DATA_LOCATIONS.md` **The practical thing that helps most:** a single plain-text file in the repo called DATA_LOCATIONS.md, updated as each cruise progresses. A plain committed file — the single source of truth for where every cruise's data lives and what state it is in. Update it as work progresses. ```markdown # Data locations ## TT23 - Raw ADCP: \\Npdata\...\TT23\raw\ - Processed .mat: \\Npdata\...\TT23\processed\ - CTD source: C:/thinf/github/troll_transect_ctd_2022_2023/data/final/ - Working output: OneDrive\_working\data\final\_2022_2023_v1.nc - GitLab: gitlab.npolar.no/npiocean/ tag: v1.0-TT23 - GitHub: github.com/npi-ocean/ tag: v1.0-TT23 - NPDC: doi:10.21334/npolar.XXXXXXX status: PUBLIC since: 2025-11-12 ## TT24 - Raw ADCP: \\Npdata\...\TT24\raw\ - Processed .mat: \\Npdata\...\TT24\processed\ - Working output: OneDrive\_working\data\final\_2023_2024_v1.nc - GitLab: gitlab.npolar.no/npiocean/ tag: not yet tagged - NPDC: not yet submitted status: IN PROGRESS ``` The status field (`NOT STARTED` / `IN PROGRESS` / `PUBLIC`) gives an at-a-glance view of the whole dataset series. --- ## LADCP Example folder structure ``` \\Npdata\...\\ ← institutional network disk raw/ ← original instrument files, read-only processed/ ← instrument processing output CTD/ C:/thinf/github/_publish/ ← git repo, NOT on OneDrive (see note) processing_utils.py ← shared processing functions configs/ ← per-cruise YAML configs notebooks/ ← one notebook per cruise README.md DATA_LOCATIONS.md ← single source of truth (see below) local_paths_template.yaml ← committed; users copy to local_paths.yaml local_paths.yaml ← LOCAL ONLY — never committed requirements.txt ← updated at publish time .gitignore OneDrive/_working/ ← working outputs only, no .git folder data/ preliminary/ ← intermediate drafts final/ ← files ready for NPDC figures/ ← QC plots OneDrive/_shared/ ← shared with external collaborators README.md _v1.nc ``` > **Why the repo is NOT on OneDrive:** OneDrive's continuous sync conflicts > with git's file locking. It can hold a lock during sync causing git to fail, > or sync a partially-written git object and corrupt the `.git` folder. Keep > the repo on a local path and let GitHub/GitLab be the backup for code. > OneDrive then only holds the working data folder, which syncs cleanly. --- ## Version control: NPI GitLab primary, GitHub as mirror NPI runs its own GitLab at `gitlab.npolar.no/npiocean` — this is the **canonical home** for all processing code. For external collaboration or public access after publication, selected repos are mirrored to GitHub. ``` gitlab.npolar.no/npiocean/ ← primary: all development ↓ push mirror (on tag or manually) github.com/npi-ocean/ ← mirror: external collaboration ↓ zip at publish time NPDC dataset record ← archived snapshot ``` Tag on GitLab first, then push the tag to GitHub: ```bash git tag -a v1.0-TT23 -m "Version used to produce _v1.nc" git push origin v1.0-TT23 # GitLab git push github v1.0-TT23 # GitHub mirror ``` --- ## What to archive on NPDC ### Self-documenting NetCDF attributes The final published file should be self-documenting and nclude provenance in the global attributes: ```yaml processing_software: "LDEO LADCP processing toolbox Version IX_15" processing_software_url: "https://github.com/jgrelet/ladcp" processing_code: "See processing/ folder in this dataset record" processing_code_repository: "https://gitlab.npolar.no/npiocean/" processing_code_version: "v1.0-TT23 (commit abc1234)" tidal_model: "CATS2008 (Padman et al., 2002; doi:10.3189/172756402781817752)" history: | 2023-01-20 to 2023-01-31: Data collection. LDEO LADCP processing toolbox Version IX_15. Tidal correction: CATS2008. 2025-11-12T10:22:00Z: NetCDF file created. ``` as well as refrences to other relevant published documents (e.g. cruise report on NVA). ### Archive the full processing chain — all layers No oceanographer would argue for deleting raw `.hex` files from a CTD after generating a publishable netcdf file. So where should we keep those? On a hrad drive in your office? On a network drive with an ever so cluttering folder sturcture? Why not in a subfolder (or .zip file) together with the finally published data file? -- Ideally toether with an unambigious description of how to transform from one to another. The same logic applies to any instrument: **raw files are the primary scientific record**. Everything downstream is derived. For LADCP, the analogy is exact: | CTD | LADCP | |---|---| | `.hex` raw binary | `.000` raw binary | | `.XMLCON` sensor config | instrument config (embedded in `.000`) | | SBE processing scripts + `.psa` parameter files | LDEO `.m` parameter files | | `.cnv` processed profiles | `_dt.mat` detided profiles | | Python QC/calibration pipeline | this Python pipeline | | Published NetCDF | published NetCDF | Discarding any layer removes the ability to reprocess with better tools or correct an error found later. Raw files are not large by modern standards (LADCP: 2–10 GB per cruise; CTD hex: < 1 GB), and the long-term storage cost is trivial compared to the ship time that generated them. The gold standard is to archive full reproducibility, but at least the blueprint (.hex) and a schematic a processing description should be archived together with the final data file. ### NPDC dataset structure NPDC supports folder structures within dataset records. A complete record could look like this (LADCP example; CTD is analogous): ``` NPDC dataset: _2022_2023_v1 │ ├── _2022_2023_v1.nc ← published, CF-compliant data │ ├── raw.zip/ ← primary instrument record │ ├── TT23_001.000 ← RDI raw binary, one per cast │ └── ... │ ├── instrument_processed.zip/ ← instrument toolbox output │ ├── mat_files/ │ │ ├── 001_dt.mat ← detided velocity profiles │ │ └── ... │ ├── logs/ │ │ ├── 001.log ← full processing log per cast │ │ └── ... │ ├── params/ │ │ ├── default.m ← default LDEO parameters │ │ └── cast_overrides.m ← any per-cast overrides │ └── INSTRUMENT_PROCESSING.md ← software version, MATLAB version, │ how to reproduce this step │ └── processing.zip/ ← Python pipeline (this repo snapshot) ├── PROVENANCE.md ├── processing_utils.py ├── configs/.yaml ├── notebooks/_make_nc.ipynb ├── local_paths_template.yaml └── requirements.txt ``` For code, don't put whole git repos (with history etc.) into processing.zip — just what's needed to understand and reproduce the processing for that specific dataset version: ### `PROVENANCE.md` Documents the Python pipeline step for netcdf generation from LDEO-processed .mat files. It ties the snapshot back to the live git repo, which might change over time: ```markdown # Provenance Code used to produce _2022_2023_v1.nc, archived alongside the data. Repository: https://gitlab.npolar.no/npiocean/ Mirror: https://github.com/npi-ocean/ Commit: abc1234def5678 Tag: v1.0-TT23 Date: 2025-11-12 To reproduce: 1. Clone the repository at the tagged commit 2. Copy local_paths_template.yaml to local_paths.yaml and fill in local paths 3. pip install -r requirements.txt 4. Run notebooks/_make_nc.ipynb ``` The *commit hash* is the critical piece — it pins the exact code state regardless of how the repository evolves after publication. ### `requirements.txt` Generate at publish time to pin exact package versions: ```bash pip freeze > requirements.txt # or: conda env export > environment.yml ``` Library behaviour changes across versions — xarray's time encoding, fill value handling, and NetCDF writing have all changed in ways that can produce subtly different output. Pinning versions means a future user starts from a known-good environment. ### `INSTRUMENT_PROCESSING.md` Documents the instrument toolbox step — the part not covered by the Python pipeline for netcdf generation. Keep it short: ```markdown # Instrument processing environment Software: LDEO LADCP processing toolbox Version IX_15 Source: https://github.com/jgrelet/ladcp (commit abc1234) MATLAB: R2023b, Windows 11 Parameters in params/ are defaults except where noted in individual .log files. Per-cast overrides are in cast_overrides.m. To reproduce: install MATLAB R2023b, clone the toolbox at the commit above, add to path, run process_cast.m with the parameter files in params/. ``` --- ## Sharing working data before archiving | Option | Mechanism | Best for | |---|---|---| | A | Shared OneDrive folder | Near-final files with 1–5 collaborators during analysis | | B | GitHub mirror (private or public) | Collaborator needs to run the code | | C | Restricted NPDC dataset | Final file; assigns DOI early for manuscript citation | For most cruise data workflows: use **A** during analysis, switch to **C** once the file is finalised and the paper submitted. Use **B** only if the external collaborator is actively contributing to the processing code. **Option C** is the recommended final step regardless: submitting to NPDC as a restricted dataset assigns the DOI immediately, so you can cite it in the manuscript before the paper is accepted, then flip to public on acceptance. --- ## Portable paths Avoid hardcoding absolute paths in config files wherever possible. Use a local `local_paths.yaml` (gitignored) with tokens substituted at load time: ```yaml # local_paths.yaml — local only, never committed processed_root: "\\Npdata\...\TrollTransect" ctd_root: "C:/thinf/github" output_root: "C:/Users/.../OneDrive/_working/data" ``` ```yaml # configs/TT23.yaml — uses tokens, safe to commit flist_glob: "{processed_root}/TT23/processed/Detide/*_dt.mat" input_ctd_nc: "{ctd_root}/troll_transect_ctd_2022_2023/data/final/TrollTransectCTD_2022_2023_v1.nc" output_nc: "{output_root}/final/_2022_2023_v1.nc" ``` Include a committed `local_paths_template.yaml` so new users know what to fill in. --- ## `.gitignore` essentials ```gitignore *.nc *.mat *.000 *.hex *.cnv *.ps data/ local_paths.yaml __pycache__/ *.pyc .ipynb_checkpoints/ ``` **One strong recommendation:** Install **nbstripout** once per clone to strip notebook outputs before commits: ```bash pip install nbstripout && nbstripout --install ``` It automatically strips cell outputs from notebooks before every git commit, so you don't accidentally commit large arrays or file paths that contain usernames/server names into the repo history. --- ## Should I include .cnv files from ctd processing and .mat/ .netcdf files from LADCP processing in the git repos? Short answer: technically yes, but you shouldn't, and for two different reasons depending on file type. **Binary files (`.mat`, `.nc`, `.000`, `.hex`)** — git stores a complete copy of every version of every file forever. Binary files can't be diffed, so git gives you none of the benefits (you can't see what changed between versions) but all of the costs (the repo balloons in size permanently; even if you delete the file later, it stays in the history). A 50 MB `.mat` file committed once is 50 MB in the repo forever. **Text-based processed files (`.cnv`)** — `.cnv` files are ASCII, so git *can* diff them, which is slightly better. But they are still derived outputs — generated from `.hex` + `.psa` parameters. Committing derived outputs to a repo creates a maintenance problem: the file in the repo can drift out of sync with the code that generates it, and it's never clear whether the committed version is the canonical one or just someone's local run. The general rule is: **commit code and configuration, not data or derived outputs**, regardless of file size or format. **What to do instead:** For sharing processed files with collaborators before NPDC archiving, use the OneDrive shared folder (Option A from the recipe). For long-term archiving, use the NPDC dataset record with its folder structure — that's exactly the right place for `.cnv`, `.mat`, and `.nc` files. **The one partial exception: `git-lfs`** Git Large File Storage (git-lfs) is an extension that stores large files outside the repo history and replaces them with small pointer files. It's supported by both GitHub and GitLab, including `gitlab.npolar.no`. It removes the repo-bloat problem for binary files. Whether it's worth it depends on the use case: | | Regular git | git-lfs | |---|---|---| | Binary files (`.mat`, `.nc`) | ✗ bloats repo permanently | ✓ stored externally | | Can diff file contents | ✗ for binary | ✗ still binary | | Collaborators need data alongside code | manual sharing | ✓ `git clone` gets everything | | Works on NPI GitLab | — | ✓ supported | | Adds setup complexity | — | moderate | For your workflow specifically I'd still lean against lfs for the data files, because the NPDC archiving step already solves the problem properly — the data has a home with a DOI, access control, and long-term preservation guarantees that a git remote doesn't provide. Using lfs to store `.mat` files in GitLab would be a parallel, less well-managed copy of data that already lives on the network disk and will end up on NPDC. The one case where lfs genuinely helps is **small reference files that belong conceptually with the code** — for example, a small lookup table, a calibration coefficient file, or a short test dataset used in unit tests. Those are fine in lfs because they are inputs to the code rather than outputs of it. So the short version to add to the recipe: `.cnv`, `.mat`, `.nc` stay out of git. They go to OneDrive for sharing and NPDC for archiving. ## References Padman, L., Fricker, H.A., Coleman, R., Howard, S., and Erofeeva, L. (2002). A new tide model for the Antarctic ice shelves and seas. *Annals of Glaciology*, 34, 247–254. doi:[10.3189/172756402781817752](https://doi.org/10.3189/172756402781817752) Howard, S.L., Greene, C.A., Padman, L., Erofeeva, S., and Sutterley, T. (2024). CATS2008_v2023: Circum-Antarctic Tidal Simulation 2008, version 2023. U.S. Antarctic Program Data Center. doi:[10.15784/601772](https://doi.org/10.15784/601772)