For anyone using USGS data at their sites, please be aware that they made major (and
potentially problematic) changes to their daily data file formats this week, affecting
both recent provisional and published/accepted data. I'm attaching a sample for one of
our stations as an example.

The good:
Long-term published daily summary data and recent provisional summary data can now be
requested using a single query (resulting in a single file) - the 720 day limiter on
daily data is now removed and you can request the full period of record using the daily
data request form. Also, the historic data link and corresponding web app are no longer
available.

The bad and the ugly:
In order to accommodate both accepted and provisional data in the same file, USGS is now
concatenating one or more text flags after *every single data point*, e.g. 23.4_A for
accepted value, 99.6_e_A for estimated accepted value, 33.43_P for provisional value,
etc. You will therefore need to use string handling procedures to remove these flags or
split the data and flags for *every data column*, and since columns may have more than
one flag assigned this could lead to column shifting if care is not taken in splitting
data and flags.

Not surprisingly this change broke the USGS data harvesting service for HydroDB, but I
have now rewritten the file parsing routines to accommodate the wacky new format. I will
re-execute the USGS harvests tomorrow and get things back on track. I will also release
an update to the GCE Data Toolbox this week that incorporates the revised USGS parser and
corresponding metadata templates.

NOAA has also been tweaking their tides online site so it looks like the coastal sites
may be in for more fun this summer.

Happy parsing.

Wade Sheldon (GCE)

(Sent on Jan 04, 2006)