Configure Compression Settings

[robertoarnetoli]

I have tried target-s3 and I was able to change prefix and stream_name_path_override but inside the S3 folder I get a compressed ".json.gz". Instead, I need a file called "data.json" without compression.

[crowemi]

Unfortunately this is hardcoded ATM - we need to modify this section to be dynamic. We should do this by adding a compression configuration node (maybe under the format node, or perhaps its own node 🤔)

Thinking big picture, we want to satisfy the following requirements:

1. Enable/disable compression
2. Support multiple compressions types (e.g. {‘NONE’, ‘SNAPPY’, ‘GZIP’, ‘BROTLI’, ‘LZ4’, ‘ZSTD’})

[robertoarnetoli]

Thank you @crowemi for the fast response.
As for the ".json.gz" file is there a way to add a name to the file like "data.json.gz". At the moment is literally ".json.gz" without name

[crowemi]

The problem is here, doesn't look like we're handling any cases where those two config elements aren't set -- if it meets your reqs, you should be able to add a file name by setting the append_date_to_filename in your config here.

[robertoarnetoli]

ok. I was looking for a static filename rather than a timestamp, but thank you anyway.

[rstml]

+1 for this.

Current default option (gzip) isn't he best option for Parquet files:

Based on our tests, gzip decompression is very slow (< 100MB/s), making queries decompression bound. Snappy can decompress at ~ 500MB/s on a single core.

https://issues.apache.org/jira/browse/SPARK-14482

[kronnk]

Hi, i would like to work on this issue. Since there is no contribution guidelines, is there anything i should pay attention to ?
Thanks!