0.42.0

Multi-Storage Client (MSC)

New Features

The MSC Explorer Web UI is now integrated into MSC, providing a browser-based interface to browse and manage objects across your configured storage backends. See the documentation for more details.

• Bumped OpenTelemetry package version to 1.32.1 for mTLS support.

Bug Fixes

• Avoid HEAD requests when checking source version to reduce unnecessary API calls.
• Resolve sync worker performance regression with bounded backpressure.
• Raise ValueError when base_path is empty in list_objects.
• Resolve multiprocessing metadata loss and related test failures in sync operations.
• Add early validation for cache_line_size exceeding cache size.

Multi-Storage File System (MSFS)

New Features

• Added support for mkdir and rmdir operations.
• Added prefetch of directories when readdir/readdirplus has not already done so, improving directory listing performance.
• MSFS Grafana dashboard now reports READ metrics.

Bug Fixes

• Fixed double locking in directory prefetcher.

0.41.0

Multi-Storage Client (MSC)

Breaking Changes

The default POSIX profile has been renamed from default to __filesystem__. If your code or configuration explicitly sets profile="default", please update it to profile="__filesystem__" (or remove the profile parameter entirely). Users who do not explicitly specify a profile, or who are not using the reserved POSIX profile, are not affected.

New Features

• Added multi-backend configuration support, enabling datasets distributed across multiple storage backends to be accessed through a single unified profile. See configuration reference for usage details.
• Added size-based batching to msc sync operations for improved performance and resource utilization.
• Added automatic ulimit increase for high-concurrency operations to prevent file descriptor exhaustion.
• Added GCS credentials provider support for Rust client with comprehensive refactoring.
• Implemented async metrics collection to significantly reduce OpenTelemetry overhead in telemetry operations.

Bug Fixes

• Fixed sync_from ignoring preserve_source_attributes parameter when using source_files argument.
• Fixed directory creation race conditions on distributed filesystems by implementing proper handling of concurrent mkdir operations.

Multi-Storage File System (MSFS)

New Features

• Added Prometheus/Grafana integration with dashboard support for monitoring file system operations.

Bug Fixes

• Fixed cache ttl_check_interval validation to ensure it is always positive.
• Fixed unlocked cache prune call that could cause race conditions.
• Fixed inode eviction to properly include clean cache lines.
• Fixed missing inode.touch() calls in file system operations.
• Fixed Grafana dashboard job selector for metrics.

0.40.0

New Features

• Add --include and --exclude pattern filtering to msc ls command for selective file listing.
• Enhance SyncResult to separately track files and bytes added/deleted, providing complete visibility into sync operations.
• Add configurable retry settings for Rust client to improve reliability and customization.

Bug Fixes

• Fix automatic retry for HTTP 408 (Request Timeout) errors when accessing Google Cloud Storage via S3-compatible API.
• Add default timeout values for S3 storage providers to prevent indefinite hangs.
• Fix unnecessary directory creation when opening files in read-only mode.

0.39.1

New Features

• Increase default sync concurrency to improve performance, especially on machines with fewer CPU cores.

Bug Fixes

• Update _is_rust_client_enabled() for CompositeStorageClient to return True when all child clients are Rust-enabled.

0.39.0

New Features

• Preserve object storage last-modified time when syncing to a POSIX target.
• Add Prometheus metrics endpoint to MSFS for enhanced observability.

Bug Fixes

• Fix double URL encoding of paths in Rust client that caused issues with special characters in object keys.
• Fix credential refresh deadlock and lock contention in concurrent sync operations.
• Optimize full-file reads to cache the entire file instead of chunking when reading from offset 0 with size ≥ file size.
• Reduce unnecessary list operations in msc sync command by using non-strict path resolution.

0.38.1

Bug Fixes

• Fixed CompositeStorageClient not receiving credentials from ProviderBundleV2 by pre-building child configs with credentials in StorageClientConfigLoader

0.38.0

New Features

This release adds support for composing MSC configurations from multiple files using the include keyword. The feature enables
teams to share common configs (telemetry, cache) while maintaining individual MSC config files. See the
Configuration Reference for more details.

• Add commit_metadata (default True) to StorageClient.sync_from to auto-commit metadata after sync.

Bug Fixes

• Make sync_from fail fast on errors and raise a RuntimeError with collected error details.
• Reduce POSIX sync lock overhead by only creating lock files for large objects (≥64MB).
• Fix metric exporter initialization error caused by a urllib3 version mismatch.
• Improve HuggingFace provider retries and error translation (including rate-limit context when available).

0.37.1

Bug Fixes

• Fix macOS wheel import error by switching from aws-lc-rs to ring as the TLS crypto backend for cross-compiled wheels.

0.37.0

Breaking Changes

• Drop x86 Mac wheel support.
• Rust Client's list_recursive() is converted to an async function.

New Features

• Support Rust client reading AWS style credentials.
• Support native AIStore backend for MSFS.

Bug Fixes

• Retry POST requests from OTEL exporter.

0.36.0

New Features

• Improve Rust client error handling to include HTTP status codes in RustClientError.
• Refactor Rust client to use offset and size instead of start and end for consistency with Python.

Bug Fixes

• Fix object listing behavior when using file or subdirectory paths.
• Make access_time optional when parsing AIStore metadata to prevent errors.