FileNotFoundError for .zmetadata with MultiZarrToZarr on non-consolidated Zarr sources

[marstonsward]

When using kerchunk.combine.MultiZarrToZarr to create a combined reference dataset from multiple source Zarr stores that do not have consolidated metadata (i.e., no .zmetadata file exists in the source stores), the kerchunk.combine.MultiZarrToZarr() method fails with a FileNotFoundError, indicating it cannot find the .zmetadata file in the first source Zarr store it processes.

Environment:

kerchunk version: 0.2.8
fsspec version: 2025.2.0
zarr version: 3.0.6
xarray version: 2024.11.0
Python version: 3.11.6
Operating System: Linux Jupyter AWS
Filesystem: EFS mount

Expected Behavior: Based on the fix implemented in PR #50 for kerchunk.zarr.single_zarr (addressing Issue #49), it was expected that MultiZarrToZarr would also gracefully handle the absence of .zmetadata in source Zarr stores by falling back to reading individual metadata files (.zgroup, .zarray, .zattrs). The kerchunk.combine.MultiZarrToZarr() method should successfully generate the combined reference dictionary.

Actual Behavior: The mzz.translate() call fails with a FileNotFoundError traceback, indicating it attempted to open the non-existent .zmetadata file within the first source Zarr store. The process does not appear to fall back to reading individual metadata components in this multi-file scenario.

# MultiZarrToZarr
mzz = kerchunk.combine.MultiZarrToZarr(
            input_file_list, # Use the list directly
            concat_dims=[concat_dimension],
            identical_dims=identical_dimensions,
            remote_protocol=None, # Data is local (EFS)
            remote_options=None,  # Data is local (EFS)
            consolidated=False
        )

Output snippet:

kerchunk_create:kerchunk_create:37 - Received 4488 input Zarr stores.
kerchunk_create:kerchunk_create:38 - First few files: 
['/export/sml-data-lake/datastash/47/gfs/gfs_2024_01_01_t00z_f000.zarr', 
'/export/sml-data-lake/datastash/47/gfs/gfs_2024_01_01_t00z_f001.zarr', 
'/export/sml-data-lake/datastash/47/gfs/gfs_2024_01_01_t00z_f002.zarr']
kerchunk_create:kerchunk_create:42 - Generating Kerchunk index for concatenation along 'time'...
kerchunk_create:kerchunk_create:43 - Identical dimensions: ['latitude', 'longitude']
kerchunk_create:kerchunk_create:98 - An error occurred during Kerchunk processing: [Errno 2] No such file or 
directory: '/export/sml-data-lake/datastash/47/gfs/gfs_2024_01_01_t00z_f000.zarr/.zmetadata' 

Reference to PR #50: This issue seems related to the problem solved by PR #50 (#50) for single_zarr, but the fix does not appear to apply or work correctly within the MultiZarrToZarr workflow. Attempts to pass arguments like zarr_kwargs={'consolidated': False} to MultiZarrToZarr resulted in a TypeError as the argument is not supported by its constructor. Minimal Reproducible Example:(Refer to the code provided in the kerchunk_bug_report_code artifact above)

[martindurant]

use_consolidated=False is peppered throughout MultiZarrToZarr (used with zarr.open, zarr.open_group, xr.open_dataset, etc.), so it would be good to see the full traceback, to see if maybe there is a call somewhere in the code that has missed it.

[marstonsward]

@martindurant I will run it again to see if I can get the entire traceback. I thought I had pasted all the tb. Hang tight.

[marstonsward]

@martindurant

I'm afraid that is all the tb the notebook spits out.

2025-04-15 21:05:35.082 | INFO     | kerchunk_create:kerchunk_create:37 - Received 4488 input Zarr stores.
2025-04-15 21:05:35.083 | INFO     | kerchunk_create:kerchunk_create:38 - First few files: ['/export/sml-data-lake/datastash/47/47b16245-7415-4b92-b52f-98e884caef3c/gfs/gfs_2024_01_01_t00z_f000.zarr', '/export/sml-data-lake/datastash/47/47b16245-7415-4b92-b52f-98e884caef3c/gfs/gfs_2024_01_01_t00z_f001.zarr', '/export/sml-data-lake/datastash/47/47b16245-7415-4b92-b52f-98e884caef3c/gfs/gfs_2024_01_01_t00z_f002.zarr']
2025-04-15 21:05:35.083 | INFO     | kerchunk_create:kerchunk_create:42 - Generating Kerchunk index for concatenation along 'time'...
2025-04-15 21:05:35.083 | INFO     | kerchunk_create:kerchunk_create:43 - Identical dimensions: ['latitude', 'longitude']
2025-04-15 21:05:35.084 | INFO     | kerchunk_create:kerchunk_create:50 - Use MultiZarrToZarr with the provided list of files
2025-04-15 21:05:35.084 | INFO     | kerchunk_create:kerchunk_create:59 - Generate the dictionary representation of the virtual dataset
2025-04-15 21:05:35.173 | ERROR    | kerchunk_create:kerchunk_create:97 - An error occurred during Kerchunk processing: [Errno 2] No such file or directory: '/export/sml-data-lake/datastash/47/47b16245-7415-4b92-b52f-98e884caef3c/gfs/gfs_2024_01_01_t00z_f000.zarr/.zmetadata'

Here's the code:

def kerchunk_create(
    input_file_list: list,
    output_index_file: str = './combined_dataset.kerchunk.json',
    concat_dimension: str = 'time',
    identical_dimensions: list = ['latitude', 'longitude', 'level'] # Adjust as needed
    ):
    """
    Creates a Kerchunk index file from a list of Zarr stores,
    optimized for local filesystem access (like EFS). Uses Loguru for logging.

    Args:
        input_file_list: A list of paths to the input Zarr stores.
        output_index_file: Path to save the generated Kerchunk JSON index.
        concat_dimension: Dimension name along which to concatenate.
        identical_dimensions: List of dimension names expected to be identical
                              across all input stores.

    Returns:
        xarray.Dataset: The dataset opened via the generated Kerchunk index.
        Returns None if index creation fails or no input files are provided.
    """

    # --- Step 1: Validate Input File List ---
    if not input_file_list or not all(isinstance(f, str) for f in input_file_list):
        # Using imported loguru logger directly
        logger.error("Input must be a non-empty list of file paths.")
        return None # Or raise ValueError

    logger.info(f"Received {len(input_file_list)} input Zarr stores.")
    logger.info(f"First few files: {input_file_list[:3]}")

    # --- Step 2: Generate the Kerchunk Index ---

    logger.info(f"Generating Kerchunk index for concatenation along '{concat_dimension}'...")
    logger.info(f"Identical dimensions: {identical_dimensions}")

    # Since data is on EFS (mounted locally), protocol is 'file'
    protocol = 'file'
    storage_options = {} # No special options needed for local EFS access

    try:
        logger.info('Use MultiZarrToZarr with the provided list of files')
        mzz = kerchunk.combine.MultiZarrToZarr(
            input_file_list, # Use the list directly
            concat_dims=[concat_dimension],
            identical_dims=identical_dimensions,
            remote_protocol=None, # Data is local (EFS)
            remote_options=None,  # Data is local (EFS)
        )

        logger.info('Generate the dictionary representation of the virtual dataset')
        kerchunk_dict = mzz.translate()
        logger.info("Kerchunk dictionary generated successfully.")

        # --- Step 3: Save the Index to a File ---

        logger.info(f"Saving Kerchunk index to: {output_index_file}")
        output_dir = os.path.dirname(output_index_file)
        if output_dir and not os.path.exists(output_dir):
            os.makedirs(output_dir, exist_ok=True)
            logger.info(f"Created output directory: {output_dir}")

        with open(output_index_file, 'w') as f:
            json.dump(kerchunk_dict, f)
        logger.info("Kerchunk index saved successfully.")


        # --- Step 4: Open the Virtual Dataset using Xarray ---

        logger.info(f"Opening virtual dataset using Xarray from: {output_index_file}")

        ds = xr.open_dataset(
            "reference://",
            engine="zarr",
            backend_kwargs={
                "storage_options": {
                    "fo": output_index_file, # Point to the local JSON index
                    "remote_protocol": None, # Original data is local
                    "remote_options": {}     # Original data is local
                },
            }
        )

        logger.info("Successfully opened dataset via Kerchunk.")
        return ds

    except Exception as e:
        logger.error(f"An error occurred during Kerchunk processing: {e}")
        return None # Return None or handle error appropriately

When I try and pass in consolidated in either the xr.open_dataset() and/or mzz = kerchunk.combine.MultiZarrToZarr() I get the error that argument or kwargs is not recognized.