Store Data in Hive Partitioning Format

[blackchapel]

[SwapneelM]

Please add a description, documentation and/or bookmarks from Google Docs so we can follow what is being done. Documentation should be issue-specific, please don't link 10-page docs but use bookmarks in Google Docs

[SwapneelM]

Still waiting on a description @blackchapel

[blackchapel]

Still waiting on a description @blackchapel

We tried one approach the, AWS Glue one which cost us a lot of money on GCP, the documentation for which is under works (https://github.com/SimPPL/arbiter-project-tracking/wiki/Hive-Partitioning). Hence this was under facing issue. But now we have another method to tackle this which will use the data on AWS S3 rather than Big Query which is much cheaper to fetch, but I forgot to move it from facing issues to to-do. We will try this approach once and see if it works, otherwise we will get back to you for help. @SwapneelM