Add Polars to Parquet implementations#153
Add Polars to Parquet implementations#153coastalwhite wants to merge 1 commit intoapache:productionfrom
Conversation
| repo_url: https://github.com/pola-rs/polars | ||
| docs_url: null | ||
| versions: | ||
| default: "1.37.0" |
There was a problem hiding this comment.
Should we point to something like latest? I see that in the meantime 1.37.1 has been released in the meantime (which is great)!
There was a problem hiding this comment.
I think that since this data is mostly used to derive this table, it is ok if it is not complete for all versions
There was a problem hiding this comment.
@alamb is correct, maybe we should try to rename this field? "baseline"? (this should be a follow-up)
There was a problem hiding this comment.
@coastalwhite just wanted to verify the X's match your expectation here (2024 I believe specifically had relatively minor changes). If you render the page, mousing over the X will tell you the missing features.
|
amazing! |
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
Thank you @coastalwhite (👋 ) and @orlp. It is great to see all the green and (R) 🚀
I rendered this PR locally and reviewed it for consistency and it looks good to me
I didn't check all the features for accuracy as this page relies on self-reporting and community help to keep it updated.
I'll also post this PR to the parquet mailing list and wait a few more days in case anyone else would like to review it
| status: full | ||
| logical-interval: | ||
| status: read | ||
| logical-json: |
There was a problem hiding this comment.
Just wanted to check for JSON and BSON other engines do support this with the caveat that they just pass through. Does polars not support them at all?
4 participants
Similar to #104, we are heeding the call to add the implementation status here. Polars has its own implementation that was forked
arrow-rsmany, many years ago, and they have grown apart a lot ever since.Although I am no longer working at Polars, I was asked by @orlp to come out of retirement (/s) and see if I can fill this in. As I was the person mostly working on Parquet at Polars for the last 1.5 years, I have a decent idea of the status.
If this needs updating, probably @azimafroozeh, @nameexhaustion and @kdn36 are the people to tag.
Referencing:
ParquetFieldOverwritesin parquet writer for column-wise properties pola-rs/polars#22278 to control encoding, compression, page indicesFloat16dtype pola-rs/polars#25185 for the Float16 workStructpola-rs/polars#24320 about the interval types