Add example demonstrating how Parquet encryption could be configured with KMS integration #16237
Conversation
|
One thing this example doesn't cover is how users might directly set encryption keys without using a KMS. That could be achieved using the same API and implementing an |
| let mut parquet_options = TableParquetOptions::new(); | ||
| // We specify that we want to use Parquet encryption by setting the identifier of the | ||
| // encryption factory to use. | ||
| parquet_options.encryption.factory_id = ENCRYPTION_FACTORY_ID.to_owned(); |
There was a problem hiding this comment.
I think this needs to be under parquet_options.global... since it affects all columns. And I guess we should figure out if we want a encryption sub-piece for all crypto related options. Maybe call it crypto?
There was a problem hiding this comment.
I initially had it under global but then decided that didn't quite make sense as some of the options might be per-column keys, so moved it to a separate encryption namespace.
And yeah I was using encryption here to include decryption properties too. crypto probably makes more sense, and being shorter to type is nice.
This is a draft PR with a non-working example demonstrating how reading and writing of Parquet files using modular encryption could be supported in DataFusion.
Related to #15216