Example - Processing SADCP from the 2022 KPH Arctic Ocean cruise - first attempt#

Note

These are notes from ØF’s attempt at running CODAS processing on SADCP data from the AO22 cruise. The notes include installing necessary software, and some trying and failing.

If you find something useful in here, great - but do not trust this as a processing recipe

Important

During the the AO22 cruise, the KPHs SADCP system still ran on the old VmDAS aquisition setup. Therefore, it was necessary to convert the data to UHDAS format in order to run CODAS post-processing.

In 2025 (?), the KPH switched to the UHDAS aquisition system, which is already set up for post-processing.

17.09

Installing locally#

Decided to try installing this on my system rather than using the VM.

Installed the pycodas environment from the yml file link.
- Took several minutes! Maybe faster using mamba?
Ran the steps to install CODAS - using the conda-based options.
- Everything has seemed to execute and download as intended so far..
For the last step, I made the symlink in ~/mambaforge

Failure: I got as far as trying to install codas3, pycurrents etc. Seemingly sucessful, but these libraries do not become availably in the pycodas environment after installing. Instead of going down a troubleshooting rabbit hold, I’ll just use the VM where everything is already set up.

Installing VM#

Followed the steps, went reasonably smooth to get the VM running.

Setting up the directory structure#

Docs were mildly confusing here, but I’m giving it a go.

Created a shared folder codas_shared and made a symlink on the Guest machine desktop.
Created a folder cruises/ao2022 within sf_codas_shared/[1]
Created directories vmdas_data uhdas_style_data adcp_pyproc within cruises/ao2022.
Copied source files (the folders 38_KHZ_FL and 150_KHZ_FL) to cruises/ao2022/vmdas_data

Preliminary processing (`adcp_database_maker`)#

First, doing th LTA processing (says here that it should be done before ENR processing because it “retrieves all kinds of useful information as well as giving you a first look at the data”).

LTA processing#

Following 2.4.4. LTA processing with adcp_database_maker.py.

Created a directory adcp_pyproc/ao2022_150.
Opened adcp_database_maker, selected the directory with 150KHZ vmdas files, and pressed Convert .LTA files.
Set project directory to adcp_pyproc/ao2022_150.
Set the cruise name to ao2022
Closed and ran dataviewer.py /media/sf_codas_shared/cruises/ao2022/adcp_pyproc/ao2022_150/os150nb_LTA

Everythiong seems to have worked, but we are definitely getting spurious velocities - e.g. mean flow of nearly 1 m/s in approximately the direction of the ships heading:

Some information in the file ao2022_os150nb_LTA_info.txt:

instrument frequency is 150
beam angle is 30
transducer angle (EA) was set as 46.420

Continuing now to ENR (single-pin) processing.

ENR processing#

Following 2.4.4. ENR processing with adcp_database_maker.py.

18.09

Reorganized the file structure to be more like what I think the manual intends:

A single sadcp_proc directory containing all cruises. sadcp_proc/vmdas_data/ has folders for individual cruises, as does adcp_pyproc.
The VMDAS source files are now in sadcp_proc/vmdas_data/ao2022/(150_KHZ_FL, 38_KHZ_FL)

Preliminary processing (`adcp_database_maker`)#

Going straight to the .ENR processing this time having already reviewed the LTA data.

From sadcp_proc, run adcp_database_maker.py
- Selecting sadcp_proc/vmdas_data/ao2022/150_KHZ_FL
- Set cruise_name to ao2022
- Selecting Convert .ENR files.
In the Reform VMDAS form, set the uhdas_style_data directory and run the three steps.

NOTE About the transducer angle in the docs:

“ This requires the “EA” setup command, to specify the orientation of the transducer relative to the keel. High accuracy is not needed—getting within 5 or 10 degrees is adequate”

NOTE A quick google search leads to this link saying “analog gyro (synchro)” - ok, synchro is a ship gyrocompass.

NOTE In the commandline “from ENR”-instructions, here (2.5.4.1.4.), it says that “heading should come from the gyro (N1R)” - so I guess I’ll set this to synchro..

NOTE Nice little snippet in this BODC cruise report by Yvonne Firing

In the Proc starter form, set:

EA: 46.42
Transducer depth: 8
Position: N1R gps
Heading: synchro hdg
Pitch and roll: not enabled (no options)
Heading correction: Enable, N2r hdg

I am pretty sure position and heading are ok (says in the doc to use the gyro for heading..). However, I am not sure about Headin corr - not sre what the N2R input source actually is. Th eother alternative would seem to be N3R rdinc -IN2R hdg don’t know what that means, either.

-> Going ahead with this setup for now

Pressing Make config file and Set up processing directories

ADCP tree form (pops up automatically)

Stay in os150nb pane
Set CODAS database name to akph
Set Starboard location to 1 (value shoudl be 0.658)
Set Forward location to 29 (value should be 28.789)
Set ensemmble length to 300 (5 min)
Change max depth for bottom tracking to 400 m (around the max range of the 150..)
Click the three buttons (Create…, Create.., Create) in sequence
- Last step (create CODAS database) takes a while (minutes)
Exiting the form

There is now an UHDAS-style database in sadcp_proc/uhdas_style_data/ao2022_os150..

Continuing now with post-processing following this demo.

Creating a directory adcp_pyproc/ao2022nb_test
Copying sadcp_proc/os150nb_ENR to sadcp_proc/adcp_pyproc/ao2022nb_test/os150nb_ENR_postproc

(Note: I think I may have done a cofig mistake somewhere; I think sadcp_proc/os150nb_ENR should be in some subfolder- Continuing for now..)

Ran quick_adcp.py --steps2rerun calib:navsteps --auto in the os150nb_ENR_postproc directory.
- This does a (first pass of a?) calibration, producing files with information about water and bottom track stuff in cal/
  - Looking at cal/watertrk/guess_xducerxy.out seems to indicate not entirely unreasonable guesses for xduxer offsets (around 0.5 stbd, around 23 m fwd)
Ran quick_mplplots.py --yearbase 2022 --plots2run all --noshow to reproduce figures
Made a copy: cp -r os150nb_ENR_postproc os150nb_ENR_postproc_orig
Navigate to os150nb_ENR_postproc and enter plot_nav.py nav/akph.gps -> Cruise track looks good. Note that there is a north pole station- let’s see how this does here.
Run figview.py cal/rotate/*png to view the heading correction.
- Seems to be missing entirely! All red. Mayeb I need to change the heading correction input source..

SUMMARY 18.09

Things seem to work - I think I can do this
Data as they are now are not good - looks like some humongous errors whenever steaming (also in open water), looks fairly reasonable on station.
Lacking some stuff in cal that shoudl be available at this point.
The heading (which shoudl be derived from the synchro) seems to be pure noise! -> Aha!..

PLAN:

Delete everything I jus produced.
Go back to adcp_database_maker and try different input sources
- Don’t use synchro..
Try the steps again!
- Check thea heading (in dataviewer.py)
  - Should not be noise!
- Check the heading correction
  - Should be present!
- Check catwt.py commands and so on
  - Shoudl find something!

19.09

Run adcp_database_maker from adcp_pyproc/
- Set cruise name to ao2022
- Convert ENR
Reform VMDAS form
- Select uhdas_style_data directory
- Press all three buttons in succession.
Proc Starter form
- EA 46.42
- Transducer depth: 8
- Position: N1r gps
- Heading: N3R rdinc
- No pitch and roll
- Heading correction: N2R heading
- Click Make Config file and Set up..
  
  NOTE: Should be able to resume from this step without redoing everything..
ADCP tree form
- Set CODAS database name to akph
- Set starboard location to 1 (xducer_dyx)
- Set forward location to 29 (xducer_dy)(should check the sign..)
- Set max bottom depth to 400
- Click all in sequence
  
  NOTE: This creates CODAS database os150nb_ENR in sadcp_proc - not sure why it ended up here..
  - (Executes in a minute or two)
- Closing the form.
- Moving to processing folder: (sadcp_proc)$ mv os150* adcp_pyproc/ao2022_150/
Run dataviewer.py
- At first glance, this looks a lot more sensible.
  - Velocities are reasonable and track biases if any seem small.
  - Heading is also sensible.
  - NOTE: Something happened to the depth range at the North Pole station (was anything reconfigured?)
Run quick_adcp.py --steps2rerun calib:navsteps --auto (from the directory ao2022_150/os150nb_ENR )

Checking cal/watertrack/adcpcal.out:

qADCP watertrack calibration
##
Time range 202.09 to 231.09
Calculation done at Thu Sep 19 06:47:46 2024
delta-u min = -100.00, max = 100.00
delta-v min = -100.00, max = 100.00
clip_amp = 0.04,  clip_ph =  3.0
clip_dt =   60,  clip_var = 0.050
Number of edited points:  46 out of  79
amp   = 1.0157  + -0.0003 (t - 212.8)
phase =  -0.04  + 0.0080 (t - 212.8)
            median     mean      std
amplitude   1.0140   1.0157   0.0143
phase      -0.0855  -0.0391   0.7517
nav - pc    3.0000   2.7826   6.1533
var         0.0010   0.0053   0.0081
min var     0.0010   0.0042   0.0063
delta-u    -0.3050   0.0965   3.0863
delta-v    -2.1400  -0.8674   3.3423

Checking guess_xducerxy.out:

guessing ADCP (dx=starboard, dy=fwd) meters from GPS
positions from akph.agt
calculation done at 2024/09/19 06:47:47
xducer_dx = 0.860407
xducer_dy = 1.948482
signal = 9538.288927

guessing ADCP (dx=starboard, dy=fwd) meters from GPS
positions from akph.agt
calculation done at 2024/09/19 06:58:01
xducer_dx = 0.865263
xducer_dy = 1.944994
signal = 9538.288927

Ref this - this is good news!: The small xducer values means that out initially prescribed values must have been good, and the large signal value is an indication that the calculation is very robust.

Still, we should apply the minor corrections (1-2 m) suggested by the calibration.

Remaking figures and look at plots

quick_mplplots.py --yearbase 2022 --plots2run all --noshow
plot_nav.py nav/akph.gps to see the cruise track (looks good)
figview.py cal/rotate/*png to see the heading correction - looks good!*
- All points are green (we have heading+correction data)
- Values are nonzero but typically within 0.1 degree.

NOTE Should have made a copy _postproc in the processing directory. For now, just going ahead in ao2022_150/os150nb_ENR

24.09

Having gone through the conversion to UHDAS, and run the preliminary calibration, I now proceed to the editing. Note that this may not be the final version, but I want to at least

Get up to speed on how to do this
Get an idea of how the data look
- Provide PD a preliminary datasets
- Collect any questions for JH

Post-processing overview

Post -processing consists of:

Manual editing
- patch_hcorr.py - patching holes in heading correction (should not nbe neccessary here)
- dataviewer.py - editing out bad data, sub-bottom pings, etc
Data calibration
- quick_adcp.py (calibration based on the data - do iteratively after editing)
Generating products (netcdf and other data products)

Having done the initial calibration: Going to editing with dataviewer.py

Following these step-by-step instructions

Starting up dataviewer.py for “manual editing”

dataviewer.py -e (run from adcp_pyproc/ao2022_150/os150nb_ENR)
Threshold editing Quick look at the dataset, tsetting threshold to remove some obviously bad stuff (but haven’t done anything to very carefully assess anything)
- Rejecting Error velocity: 500 mm/s
- Rejecting EddVel Stddev (and PG<80): 300 mm/s
- Rejecting bins with PS less than: 80 Looks like the middle one is redundant gioven the second one..
Hand editing
- Taking out ovbious stuff (single outliers, sb.bottom measurements, etc)
- Taking out stuff that looks a little suspicious in the vicinity of other stuff that is taken out.
- Taking out stuff from going in the ice with a rather broad brush.
  - This includes most of the transect upo to the north pole (I thought they were going in a lead?)
  - Possibly some good data in here, but I find it pretty hard to say..
  - Characterizing ice passage from ship speed/jitter.
- One thing that remains suspicious is a rather latrge vertical velovity in the top bins (seems to be mostly in open water?)
- Don’t know how to exit safely, so just trying closing the dataviewer windows.. (-> That’s fine)
Recalibration

Running quick_adcp.py --steps2rerun calib:navsteps --auto again..

Look at the calibration (catwt.py):

            median     mean      std
amplitude   1.0130   1.0128   0.0111
phase      -0.1740  -0.1817   0.7488

Applying calibration based on this info (since phase is >0.1 deg we include phase as well as amplitude):
- quick_adcp.py --steps2rerun rotate:apply_edit:navsteps:calib --rotate_amplitude 1.013 --rotate_angle -0.18 --auto

Check calibration again (catwt.py):

Number of edited points:  30 out of  35

            median     mean      std
amplitude   1.0000   1.0001   0.0110
phase       0.0050  -0.1002   0.9198

.. ok, great! Seems to have worked

Looking at the figures again (figview.py)
Export to netcdf: adcp_nc.py adcpdb contour/os150nb kph_ao2022_prelim os150nb --ship_name Kronprins Haakon
- Creates the file os150nb_ENR/contour/os150nb.nc

Example - Processing SADCP from the 2022 KPH Arctic Ocean cruise - first attempt

Contents

Example - Processing SADCP from the 2022 KPH Arctic Ocean cruise - first attempt#

Installing locally#

Installing VM#

Setting up the directory structure#

Preliminary processing (adcp_database_maker)#

LTA processing#

ENR processing#

Preliminary processing (adcp_database_maker)#

Preliminary processing (`adcp_database_maker`)#

Preliminary processing (`adcp_database_maker`)#