# *Example* - Processing SADCP from the 2022 KPH Arctic Ocean cruise - *first attempt* :::{admonition} Note :class: warning These are notes from ØF's attempt at running CODAS processing on SADCP data from the AO22 cruise. The notes include installing necessary software, and some trying and failing. If you find something useful in here, great - but *do not trust this as a processing recipe* ::: :::{important} During the the AO22 cruise, the KPHs SADCP system still ran on the old VmDAS aquisition setup. Therefore, it was necessary to convert the data to UHDAS format in order to run CODAS post-processing. In 2025 *(?)*, the KPH switched to the UHDAS aquisition system, which is already set up for post-processing. ::: ____ *17.09* ### Installing locally Decided to try installing this on my system rather than using the VM. - Installed the `pycodas` environment from the yml file [link](https://currents.soest.hawaii.edu/docs/adcp_doc/codas_setup/anaconda_install/index.html). - Took several minutes! Maybe faster using mamba? - Ran the [steps](https://currents.soest.hawaii.edu/docs/adcp_doc/codas_setup/codas_config/index.html) to install CODAS - using the *conda-based* options. - Everything has seemed to execute and download as intended so far.. - For the last step, I made the symlink in `~/mambaforge` > ***Failure:*** I got as far as trying to install codas3, pycurrents etc. Seemingly sucessful, but these libraries do not become availably in the `pycodas` environment after installing. Instead of going down a troubleshooting rabbit hold, I'll just use the VM where everything is already set up. ### Installing VM - Followed the steps, went reasonably smooth to get the VM running. #### Setting up the directory structure Docs were mildly confusing here, but I'm giving it a go. - Created a shared folder `codas_shared` and made a symlink on the Guest machine desktop. - Created a folder `cruises/ao2022` within `sf_codas_shared/`[^tag] - Created directories `vmdas_data uhdas_style_data adcp_pyproc` within `cruises/ao2022`. - Copied source files (the folders `38_KHZ_FL` and `150_KHZ_FL`) to ` cruises/ao2022/vmdas_data` ### Preliminary processing (`adcp_database_maker`) First, doing th LTA processing (says [here](https://currents.soest.hawaii.edu/docs/adcp_doc/codas_doc/adcp_database_maker/adcp_database_maker_ENR.html#enr-dbmaker-index) that it should be done before ENR processing because it *"retrieves all kinds of useful information as well as giving you a first look at the data"*). #### LTA processing Following [2.4.4. LTA processing with adcp_database_maker.py](https://currents.soest.hawaii.edu/docs/adcp_doc/codas_doc/adcp_database_maker/adcp_database_maker_LTA.html). - Created a directory `adcp_pyproc/ao2022_150`. - Opened `adcp_database_maker`, selected the directory with 150KHZ vmdas files, and pressed `Convert .LTA` files. - Set project directory to `adcp_pyproc/ao2022_150`. - Set the cruise name to `ao2022` - Closed and ran `dataviewer.py /media/sf_codas_shared/cruises/ao2022/adcp_pyproc/ao2022_150/os150nb_LTA` Everythiong seems to have *worked*, but we are definitely getting spurious velocities - e.g. mean flow of nearly 1 m/s in approximately the direction of the ships heading: description Some information in the file `ao2022_os150nb_LTA_info.txt`: ``` instrument frequency is 150 beam angle is 30 transducer angle (EA) was set as 46.420 ``` Continuing now to ENR (single-pin) processing. #### ENR processing Following [2.4.4. ENR processing with adcp_database_maker.py](https://currents.soest.hawaii.edu/docs/adcp_doc/codas_doc/adcp_database_maker/adcp_database_maker_ENR.html#enr-dbmaker-index). ____ *18.09* Reorganized the file structure to be more like what I think the manual intends: - A single `sadcp_proc` directory containing all cruises. `sadcp_proc/vmdas_data/` has folders for individual cruises, as does `adcp_pyproc`. - The VMDAS source files are now in `sadcp_proc/vmdas_data/ao2022/(150_KHZ_FL, 38_KHZ_FL)` ### Preliminary processing (`adcp_database_maker`) Going straight to the .ENR processing this time having already reviewed the LTA data. - From `sadcp_proc`, run `adcp_database_maker.py` - Selecting `sadcp_proc/vmdas_data/ao2022/150_KHZ_FL` - Set cruise_name to `ao2022` - Selecting *Convert .ENR files*. - In the *Reform VMDAS form*, set the uhdas_style_data directory and run the three steps. > **NOTE** About the transducer angle in the [docs](https://currents.soest.hawaii.edu/docs/adcp_doc/APPENDIX/best_practices.html): > > *" This requires the “EA” setup command, to specify the orientation of the transducer relative to the keel.* ***High accuracy is not needed***—*getting within 5 or 10 degrees is adequate"* > > **NOTE** A quick google search leads to [this link](https://www.comm-tec.com/prods/mfgs/RDI/Software/Manuals/VMDAS-Manual/VmDas%20Users%20Guide.pdf) saying *"analog gyro (synchro)"* - ok, *synchro* is a ship gyrocompass. > **NOTE** In the commandline "from ENR"-instructions, [here](https://currents.soest.hawaii.edu/docs/adcp_doc/codas_doc/commandline_demos/commandline_from_scratch_ENR.html) (2.5.4.1.4.), it says that *"heading should come from the gyro (N1R)"* - so I guess I'll set this to `synchro`.. > **NOTE** Nice little snippet in this [BODC cruise report by Yvonne Firing](https://www.bodc.ac.uk/resources/inventories/cruise_inventory/reports/jr18002.pdf) In the *Proc starter form*, set: - EA: 46.42 - Transducer depth: 8 - Position: N1R gps - Heading: synchro hdg - Pitch and roll: not enabled (no options) - Heading correction: Enable, N2r hdg > I am pretty sure position and heading are ok (says in the doc to use the gyro for heading..). However, I am not sure about Headin corr - not sre what the N2R input source actually is. Th eother alternative would seem to be *N3R rdinc* -IN2R hdg don't know what that means, either. > > -> *Going ahead with this setup for now* - Pressing *Make config file* and *Set up processing directories* *ADCP tree form* (pops up automatically) - Stay in *os150nb* pane - Set CODAS database name to `akph` - Set Starboard location to `1` (value shoudl be 0.658) - Set Forward location to `29` (value should be 28.789) - Set ensemmble length to 300 (5 min) - Change max depth for bottom tracking to 400 m (around the max range of the 150..) - Click the three buttons (*Create..., Create.., Create*) in sequence - Last step (create CODAS database) takes a while (minutes) - Exiting the form There is now an UHDAS-style database in `sadcp_proc/uhdas_style_data/ao2022_os150`.. Continuing now with *post-processing* following [this demo](https://currents.soest.hawaii.edu/docs/adcp_doc/codas_doc/postprocessing/postprocessing_uhdas.html#postprocessing-uhdas). - Creating a directory `adcp_pyproc/ao2022nb_test` - Copying `sadcp_proc/os150nb_ENR` to `sadcp_proc/adcp_pyproc/ao2022nb_test/os150nb_ENR_postproc` (Note: I think I may have done a cofig mistake somewhere; I think `sadcp_proc/os150nb_ENR` should be in some subfolder- Continuing for now..) - Ran `quick_adcp.py --steps2rerun calib:navsteps --auto` in the `os150nb_ENR_postproc` directory. - This does a (first pass of a?) calibration, producing files with information about water and bottom track stuff in `cal/` - Looking at `cal/watertrk/guess_xducerxy.out seems to indicate not entirely unreasonable guesses for xduxer offsets (around 0.5 stbd, around 23 m fwd) ` - Ran `quick_mplplots.py --yearbase 2022 --plots2run all --noshow` to reproduce figures - Made a copy: `cp -r os150nb_ENR_postproc os150nb_ENR_postproc_orig` - Navigate to `os150nb_ENR_postproc` and enter `plot_nav.py nav/akph.gps` -> Cruise track looks good. Note that there is a north pole station- let's see how this does here. - Run `figview.py cal/rotate/*png` to view the heading correction. - ***Seems to be missing entirely!*** All red. Mayeb I need to change the heading correction input source.. ____ **SUMMARY 18.09** - Things seem to *work* - I think I can do this - Data as they are now are not good - looks like some humongous errors whenever steaming (also in open water), looks fairly reasonable on station. - Lacking some stuff in `cal` that shoudl be available at this point. - The **heading** (which shoudl be derived from the *synchro*) seems to be **pure noise**! -> Aha!.. PLAN: - Delete everything I jus produced. - Go back to `adcp_database_maker` and try different input sources - Don't use *synchro*.. - Try the steps again! - Check thea heading (in dataviewer.py) - Should not be noise! - Check the heading correction - Should be present! - Check `catwt.py` commands and so on - Shoudl find something! ____ *19.09* 1. **Run `adcp_database_maker` from `adcp_pyproc/`** - Set cruise name to `ao2022` - Convert ENR 2. ***Reform VMDAS* form** - Select `uhdas_style_data` directory - Press all three buttons in succession. 3. ***Proc Starter* form** - EA 46.42 - Transducer depth: 8 - Position: N1r gps - Heading: N3R rdinc - No pitch and roll - Heading correction: N2R heading - Click `Make Config file` and `Set up..` > *NOTE*: Should be able to resume from this step without redoing everything.. 4. ***ADCP tree* form** - Set CODAS database name to `akph` - Set starboard location to 1 (`xducer_dyx`) - Set forward location to 29 (`xducer_dy`)(should check the sign..) - Set max bottom depth to 400 - Click all in sequence > *NOTE*: This creates CODAS database `os150nb_ENR` in `sadcp_proc` - not sure why it ended up here.. - (Executes in a minute or two) - Closing the form. - Moving to processing folder: `(sadcp_proc)$ mv os150* adcp_pyproc/ao2022_150/ ` 5. Run `dataviewer.py` - At first glance, this looks a lot more sensible. - Velocities are reasonable and track biases if any seem small. - Heading is also sensible. - ***NOTE***: Something happened to the depth range at the North Pole station (was anything reconfigured?) 6. Run `quick_adcp.py --steps2rerun calib:navsteps --auto` (from the directory `ao2022_150/os150nb_ENR` ) - Checking `cal/watertrack/adcpcal.out`: ``` qADCP watertrack calibration ## Time range 202.09 to 231.09 Calculation done at Thu Sep 19 06:47:46 2024 delta-u min = -100.00, max = 100.00 delta-v min = -100.00, max = 100.00 clip_amp = 0.04, clip_ph = 3.0 clip_dt = 60, clip_var = 0.050 Number of edited points: 46 out of 79 amp = 1.0157 + -0.0003 (t - 212.8) phase = -0.04 + 0.0080 (t - 212.8) median mean std amplitude 1.0140 1.0157 0.0143 phase -0.0855 -0.0391 0.7517 nav - pc 3.0000 2.7826 6.1533 var 0.0010 0.0053 0.0081 min var 0.0010 0.0042 0.0063 delta-u -0.3050 0.0965 3.0863 delta-v -2.1400 -0.8674 3.3423 ``` - Checking `guess_xducerxy.out`: ``` guessing ADCP (dx=starboard, dy=fwd) meters from GPS positions from akph.agt calculation done at 2024/09/19 06:47:47 xducer_dx = 0.860407 xducer_dy = 1.948482 signal = 9538.288927 guessing ADCP (dx=starboard, dy=fwd) meters from GPS positions from akph.agt calculation done at 2024/09/19 06:58:01 xducer_dx = 0.865263 xducer_dy = 1.944994 signal = 9538.288927 ``` Ref [this](https://currents.soest.hawaii.edu/docs/adcp_doc/codas_doc/dataviewer/xducer_dxdy.html) - this is *good news!*: The small `xducer` values means that out initially prescribed values must have been good, and the large `signal` value is an indication that the calculation is very robust. Still, we should apply the minor corrections (1-2 m) suggested by the calibration. 7. Remaking figures and look at plots - `quick_mplplots.py --yearbase 2022 --plots2run all --noshow ` - `plot_nav.py nav/akph.gps` to see the cruise track (looks good) - `figview.py cal/rotate/*png` to see the heading correction - *looks good!** - All points are green (we have heading+correction data) - Values are nonzero but typically within 0.1 degree. > **NOTE** Should have made a copy `_postproc` in the processing directory. For now, just going ahead in `ao2022_150/os150nb_ENR` _____ *24.09* Having gone through the conversion to UHDAS, and run the preliminary calibration, I now proceed to the editing. Note that this may not be the final version, but I want to at least - Get up to speed on how to do this - Get an idea of how the data look - Provide PD a preliminary datasets - Collect any questions for JH **Post-processing overview** Post -processing [consists of](https://currents.soest.hawaii.edu/docs/adcp_doc/glossary/index.html#term-Post-Processing): - Manual editing - `patch_hcorr.py` - patching holes in heading correction (should not nbe neccessary here) - `dataviewer.py` - editing out bad data, sub-bottom pings, etc - Data calibration - `quick_adcp.py` (calibration based on the data - do iteratively after editing) - Generating products (netcdf and other data products) ___ Having done the initial calibration: Going to [editing with `dataviewer.py`](https://currents.soest.hawaii.edu/docs/adcp_doc/codas_doc/dataviewer/dataviewer_edit_mode.html) - Following [these step-by-step instructions](https://currents.soest.hawaii.edu/docs/adcp_doc/codas_doc/postprocessing/postprocessing_uhdas.html#postprocessing-uhdas) **Starting up `dataviewer.py` for "manual editing"** 1. `dataviewer.py -e` (run from `adcp_pyproc/ao2022_150/os150nb_ENR`) 2. **Threshold editing** Quick look at the dataset, tsetting threshold to remove some obviously bad stuff (but haven't done anything to very carefully assess anything) - *Rejecting Error velocity:* 500 mm/s - *Rejecting EddVel Stddev (and PG<80):* 300 mm/s - *Rejecting bins with PS less than:* 80 Looks like the middle one is redundant gioven the second one.. 3. **Hand editing** - Taking out ovbious stuff (single outliers, sb.bottom measurements, etc) - Taking out stuff that looks a little suspicious in the *vicinity* of other stuff that is taken out. - Taking out stuff from going in the ice with a rather broad brush. - This includes most of the transect upo to the north pole (I thought they were going in a lead?) - Possibly some good data in here, but I find it pretty hard to say.. - Characterizing ice passage from ship speed/jitter. - One thing that remains suspicious is a rather latrge vertical velovity in the top bins (seems to be mostly in open water?) - Don't know how to exit safely, so just trying closing the dataviewer windows.. (-> That's fine) 4. **Recalibration** - Running `quick_adcp.py --steps2rerun calib:navsteps --auto ` again.. - Look at the calibration (`catwt.py`): ``` median mean std amplitude 1.0130 1.0128 0.0111 phase -0.1740 -0.1817 0.7488 ``` - Applying calibration based on this info (since phase is >0.1 deg we include phase as well as amplitude): - `quick_adcp.py --steps2rerun rotate:apply_edit:navsteps:calib --rotate_amplitude 1.013 --rotate_angle -0.18 --auto` - Check calibration again (`catwt.py`): ``` Number of edited points: 30 out of 35 median mean std amplitude 1.0000 1.0001 0.0110 phase 0.0050 -0.1002 0.9198 ``` .. ok, great! Seems to have worked - Looking at the figures again (`figview.py`) - Export to netcdf: `adcp_nc.py adcpdb contour/os150nb kph_ao2022_prelim os150nb --ship_name Kronprins Haakon` - Creates the file `os150nb_ENR/contour/os150nb.nc` ____ [^tag]: I don't think this is *quite* the structure that is suggested in the docs, but I'll stick with it for now.