Improvements for the Data Management codes

current suggestions

1. Consider additional formats for convcast
   • GTSPP has new data types. Of special note are HCDT and HCSP.
   • HydroBase climatology (input only)
     • Will need to differentiate HydroBase (data) and HydroBase (climatology)
     • If add this before going to new format, will need to allow user to choose between only keeping topmost contiguous set and just discarding missing values and value with insufficient data support.
   • JJXX
   • POEM91, have some bio-data in that format.

2. Documentation: In addition to describing the programs and file format, need to describe the routines a user would typically use when creating a new program.

3. Revised HOPS format:

           title = sample header
           stations = 659
           max. pts. per sta. = 496
           fields = 5
           max. flds. per sta. = 5
           str_time = 10684.2930, Aug 23 1997 07:01:52
           end_time = 10726.7578, Oct  4 1997 18:11:15
           Jday_offset = 2440000
           lat_min = 34.4530
           lat_max = 40.5000
           lon_min = 14.9710
           lon_max = 22.9910
           format = ascii, record interleaving
           creation_date = Tue Oct 14 12:02:44 1997
           history = filt(M,3,2000); filt(G,6); stable
          VARIABLES
           depth
              long_name:   depth below sea level
              short_name:  depth
              units:       meter
              type:        scalar
              scale:       1.00E-01
              _FillValue:  -10
              FieldWidth:  5
           temp
              long_name:   temperature
              short_name:  temperature
              units:       Celsius
              type:        scalar
              scale:       1.00E-03
              _FillValue:  99999  (-300 would be better, but won't fit)
              FieldWidth:  5
           salt
              long_name:   salinity
              short_name:  salinity
              units:       PSU
              type:        scalar
              scale:       1.00E-03
              _FillValue:  -1000
              FieldWidth:  5
           refsurf
              long_name:   reference surface for dynamic height
              short_name:  reference surface
              units:       meter
              type:        scalar
              scale:       1.00E-01
              _FillValue:  -10
              FieldWidth:  5
           vtot
              long_name:   total velocity
              short_name:  velocity
              units:       centimeter second-1
              type:        2-vector
              components:  zonal, meridional
              scale:       1.00E-02
              _FillValue:  999999
              FieldWidth:  6
          END
   

   Then could use this information to
   • define the output netCDF file for the OA(G) code
   • avoid interpolating through missing values when objectively analyzing.
   • determine what variables are available for objectively analyzing. (use names)
   • fill in classifiers

                    Cast(CTD): depth temp salt
                    Cast(GEN): refsurf
                    Cast(GEN): depth temp salt vtot vtot
           

   The counting of fields depends on context:
   • When counting the number of fields in the fields line of the main header or in the enumeration of the variables descriptors, an n-vector counts as one (1) field
   • When counting the number of fields in the max. flds. per sta. line of the main header or in the count per data block, an n-vector counts as n fields
   Finally, have moved the scaling factors to the file header. Instead of just shortening the cast header, consider replacing the scaling data with error estimates.
   • Initialize error level with scale.
   • Make programs which affect data (filtering) keep track of changes. Use RMS differences to update errors.

4. No longer hardwiring jdayoff. Add logic to cat_hydro to ensure consistent jdayoff. Also rename cat_hydro to cat_hops.

5. Will need new program to remove fields from the new HOPS format. (Disabling convcast's partial ability to do this)

6. Break-up time stamp from decimal julian day to either:
   • julian day, hour, minute, second
   • julian day, decimal hour, second
   • month day, year hour:minute:second
     • would do internal conversions in OA(G) to work with julian dates.

7. Check to make sure that all MATLAB & FORTRAN routines produce consistent conversions between Julian dates & Gregorian dates.
   • The midnight/noon day start is an issue.

8. Break-up positions:
   • degrees; decimal minutes; hemisphere indicator (N,S,E,W)
   • ± degrees; decimal minutes

9. Design other instrument types:
   • AUVs/drifters: (lon,lat,z,t,data)
   • moorings: (one set z, {one t, [data(z)]})

10. When stabilizing profiles, consider trying to maintain TS tendencies:
    • in the middle of a gap of instability, perhaps "interpolating" along the TS curve somehow?
    • with unstable end, use dT, dS to set target values, perhaps using latest 3 or 4 stable points to best fit slope. Perhaps also fit dRHO, then find gamma such that gamma dT and gamma dS give dRHO.

11. Allow some method of specifying climatology rather than a time.

12. Need some method of specifying the expected errors.

13. CJL suggests that the Matlab read/write scripts store the data like contours, not matrices with tons of fill values.