Meta-reader internal interface (aka modification times)

[cpiker]

In many situations it would be handy if the core pyserver could find out how recently a sub-set of data had been modified. To stick with the overall pyserver mentality, the interface for asking this question should be handled by a sub-program. My proposed way to handle this is to add a new processes type:

• meta - Provide requested metadata items given a set of coordinates and options.

This is as opposed to read process type which directly provides data. Though a full set of metadata items need not be defined here, the most immediately useful one is newest_mod_time. This metadata column should contain the most recent modification timestamps for all input files needed by a reader over the given coordinate range, or coordinate range set.

For a more concrete example, a given meta reader might have an interface as follows:

waves_lrs_meta begin end interval metadata_item1 metadata_item2 metadata_item3

So given the call:

waves_lrs_meta 2022-01-01 2022-01-05 86400 newest_mod_time

The following response might be generated (using 2.3/basic-xml stream format in this case):

<stream version="2.3/basic-xml" >
<packet id="1">
  <x pdim="time">
    <array type="isotime" use="min">
  </x>
  </y pdim="newest_mod_time">
    <array type="isotime" use="max">
  </y>
</packet>
<d id="1">2022-01-01 2022-01-15T14:47:32.345</d>
<d id="1">2022-01-02 2022-01-15T14:47:33.664</d>
<d id="1">2022-01-03 2022-01-15T14:47:33.976</d>
<d id="1">2022-01-04 2022-01-15T18:03:18.768</d>
...
</stream>

This new capability could be utilized by the das2-pyserver cache system to determine which cache blocks are out of date, and thus avoid blind refreshes which are inefficient.

There are other uses for metadata information (as noted in the comments below) so this basic capability might affect external interfaces as well.

[jbfaden]

This could be used by the clients to maintain a cache as well. Currently there's no way to automatically cache data on the client side. With HAPI, the client can send "if-modified-since" in the headers and the server can respond with a 304 code (nothing has been updated). I'd love to have this feature in the Das2Server.

[cpiker]

Sounds reasonable to me. I can't think of any good reason not to expose the interface to clients as well.