Write, List, Rename & Delete Microbenchmarks #738
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Fixing Block size and consistency options in Extended GCSFS Open (#34) * Separate versioned and non-versioned tests to use different bucket * Update cleanup logic in tests to empty the bucket instead of deleting the bucket * Add consistency and blocksize for ext gcsfs * Merge conflicts resolution * bucket type paramet for fs test for block size and consistency * removed unused mocks variables from some tests * fixing lint errors * fixed small issue with core tests so it can run with exp flag true --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com> * pytest microbenchmarks for seq and random reads both single multi threaded * multiprocess benchmarks * script to run tests * undo settings for bucket names and logs * benchmark script updates * file size and bucket type decorators * file size configuration * removed zonal config * Added README * Readme update * Moving settings and fixture to tests root * Readme update * Readme update * Ignore benchmark pytests in CI * benchmark hook fix * adding skip tests flag * benchmark plugin conditional enablement * Fixing PR Comments, simplifying the configuration by doing auto gen * Fixing PR Comments * default settings --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com>
* Fixing Block size and consistency options in Extended GCSFS Open (#34) * Separate versioned and non-versioned tests to use different bucket * Update cleanup logic in tests to empty the bucket instead of deleting the bucket * Add consistency and blocksize for ext gcsfs * Merge conflicts resolution * bucket type paramet for fs test for block size and consistency * removed unused mocks variables from some tests * fixing lint errors * fixed small issue with core tests so it can run with exp flag true --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com> * pytest microbenchmarks for seq and random reads both single multi threaded * multiprocess benchmarks * script to run tests * undo settings for bucket names and logs * benchmark script updates * file size and bucket type decorators * file size configuration * removed zonal config * Added README * Readme update * Moving settings and fixture to tests root * Readme update * Readme update * Ignore benchmark pytests in CI * benchmark hook fix * adding skip tests flag * benchmark plugin conditional enablement * Fixing PR Comments, simplifying the configuration by doing auto gen * Fixing PR Comments * default settings * added resource monitoring for benchmarks * minor refactoring * moved config to yaml * lint fixes * config yaml update for full read suite * undo zonal file logging changes * simplify single threaded read * bringing back requirements * update readme * csv generation fix when some tests fail * psutil install in cloudbuild * psutil install in cloudbuild * Removing zonal conditional code and updating config --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com>
* Fixing Block size and consistency options in Extended GCSFS Open (#34) * Separate versioned and non-versioned tests to use different bucket * Update cleanup logic in tests to empty the bucket instead of deleting the bucket * Add consistency and blocksize for ext gcsfs * Merge conflicts resolution * bucket type paramet for fs test for block size and consistency * removed unused mocks variables from some tests * fixing lint errors * fixed small issue with core tests so it can run with exp flag true --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com> * pytest microbenchmarks for seq and random reads both single multi threaded * multiprocess benchmarks * script to run tests * undo settings for bucket names and logs * benchmark script updates * file size and bucket type decorators * file size configuration * removed zonal config * Added README * Readme update * Moving settings and fixture to tests root * Readme update * Readme update * Ignore benchmark pytests in CI * benchmark hook fix * adding skip tests flag * benchmark plugin conditional enablement * Fixing PR Comments, simplifying the configuration by doing auto gen * Fixing PR Comments * default settings * added resource monitoring for benchmarks * minor refactoring * moved config to yaml * lint fixes * config yaml update for full read suite * undo zonal file logging changes * simplify single threaded read * bringing back requirements * update readme * csv generation fix when some tests fail * psutil install in cloudbuild * psutil install in cloudbuild * Removing zonal conditional code and updating config * parallel file creation in setup * merge conflicts * merge conflicts - lint fixes * lint issues --------- Co-authored-by: Mahalaxmibejugam <60227368+Mahalaxmibejugam@users.noreply.github.com>
* Add mv method for HNS aware implementation * enable HNS tests in the CI pipeline * Update super mv call to pass all arguments * Add more unit tests and make info call optional * add TODO * refactor tests and address comments * address comments * add debug logs * update cache logic for rename method * add TODO
* Support write mode in ZonalFile and override related methods * Add unit test for init_aaow method in zb_hns_utils * Implement zonal write methods * Remove overwrite parameter since it is not added in AAOW. instead use generation=None to overwrite * Remove redundant init_aaow method from zonal_file * update statement for NotImplementedErrors * set autocommit=false as default for ZonalFile * add logic to route upload methods to zonal implementation * Revert "set autocommit=false as default for ZonalFile" This reverts commit d51c8c3. * add logic to skip test in test_file itself * Added comments for clarity * Added tests for Zonal writes Close GCSFile before aaow/mrd * Do not finalize files on close by default Use simple_flush instead of normal flush add comments for clarity * Update requirements to use python-storage 3.7.0 update ci pipeline to run test_zonal_file also * Update ulimit to 2048 to avoid too many open files error * Update tests to finalize the object before reading * Support append mode in ZonalFile * try removing finalized file if it already exists * Fix lint error * fix failing hns rename tests * Update assertions in mv tests touch method's side effect shouldn't be counted in the assertion of the rename test --------- Co-authored-by: Mahalaxmi <mahalaxmib@google.com>
* Add mv method for HNS aware implementation * enable HNS tests in the CI pipeline * Update super mv call to pass all arguments * Add more unit tests and make info call optional * add TODO * refactor tests and address comments * address comments * add debug logs * update cache logic for rename method * add TODO
Contributor
Author
|
/gcbrun |
1 similar comment
Collaborator
|
/gcbrun |
ankitaluthra1
requested changes
Jan 7, 2026
ankitaluthra1
approved these changes
Jan 9, 2026
Collaborator
|
/gcbrun |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add benchmarks for write, delete, list and rename
Refactored benchmarking code accordingly