Collection date times not matching with granules errors

[binni979]

This is a pull request to solve this issue.

Overview:
At the collection level, pyQuARC was not flagging temporal extents that did not match with the granules.

Example
UMM-C Record: C2066317191-LARC

Changes made:
Added "Z" at the end of date_formates which follows standard ISO 8601 formatting rules with the 'Z' indicating UTC time.

Expected behavior:
pyQuARC is showing the error for the Beginning and Ending Date Times if it does not match with granules.

[lavanya3k]

Description of the bug: Including the above code changes, the collection datetime matched the granule datetime. But the datetime format did not have 'T' and 'Z'; this is the recommended datetime format (major bug). In this update, we tested that the collection datetime match with the beginning datetime or ending datetime and the datetime format reads 'YYYY-MM-DDTHH:MM:SSSZ'

** Code updates: ** The changes were made in util.py in the get_date_time (dt_str) function. Along with code changes, we had to update schema_validator.py, test_downloader.py and pytest.ini to ensure no conflicts with pytest runs.

Reproduce the bug:
C1620496172-CDDIS, umm-c (beginning time)
C2404975699-CDDIS, umm-c
C7019526-LARC_ASDC, umm-c (ending time)
C1000000044-CDDIS, umm-c (beginning time)

Expected output:

[FBayat021]

Please address before pushing PR to the Dev/Master branch:
FB: If the time difference is more than 24 hours, the output message should be an "Error" (red) and not a "Warning" (yellow) [if possible]
FB: Please change the output message to the following:

"Note: The collection beginning date time 'COLLECTION DATETIME' is not consistent with the beginning date time in the metadata for the first granule in Earthdata Search 'GRANULE DATETIME'. Recommend updating the beginning date time to match the granule extent."

[slesaad]

@fb0023 i updated this to address your comment - can you please test it out and see all is good? i'll merge this once i get the confirmation

[FBayat021]

@slesaad, the message is not updated in the collection-datetime-bug branch. It also does not change from Warning to Error when the difference is more than 24 hours. For example, for this record C1000000044-CDDIS, the beginning date-time at the collection level is 2000-01-01T00:00:00.000Z; however, the beginning date-time at the granule level is 2019-04-29 00:00:00.000Z. Since the difference is more than 24 hours, it needs to be flagged as an Error.

Ideally, for this example, we need the output message to show this:

Error: The collection beginning date time '2000-01-01T00:00:00.000Z' is not consistent with the beginning date time in the metadata for the first granule in Earthdata Search '2019-04-29 00:00:00.000Z'. Recommend updating the beginning date time to match the granule extent.

[lavanya3k]

My thoughts on why the error and message are not working. Currently, the code change is made in the def validate_datetime_against_granules in the datetime_validator.py. But, in the rules_mapping.json, check_messages.json and checks. json, we are referring to def validate_beginning_datetime_against_granules. Any thoughts on how to handle this? S

[slesaad]

My thoughts on why the error and message are not working. Currently, the code change is made in the def validate_datetime_against_granules in the datetime_validator.py. But, in the rules_mapping.json, check_messages.json and checks. json, we are referring to def validate_beginning_datetime_against_granules. Any thoughts on how to handle this? S

@lavanya3k if you look at the validate_beginning_datetime_against_granules method, you'll see that it's calling validate_datetime_against_granules internally. So that's not the problem

[lavanya3k]

Testing: Tested the collection for datetime difference > 24 hours and < 24 hours.

Example: Used the .native file of C1000000022-CDDIS from Earthdata search. Then, edited the datetime for the beginning and ending datetime.
python pyQuARC/main.py --file C1000000022-CDDIS.native --format umm-c

Expected output

[slesaad]

@lavanya3k can you help replace this with the real collection and granule ids that exist

Filepath: tests/test_downloader.py

def setup_method(self):
        self.concept_ids = {
            "collection": {
                "real": "C1339230297-GES_DISC",
                "dummy": "C123456-LPDAAC_ECS",
            },
            "granule": {
                "real": "G1370895082-GES_DISC",
                "dummy": "G1000000002-CMR_PROV",
            },
            "invalid": "asdfasdf",
        }

[lavanya3k]

Testing in the dev: The above fix works for the collections. Additionally, tested in the test file with a time difference of less than 24 hours, and it works.

Expected outcome: