New lint `clippy::allow_attributes`

[Jefffrey]

Is your feature request related to a problem or challenge?

Lint reference: https://rust-lang.github.io/rust-clippy/master/index.html?search=allow_att#allow_attributes

Checks for usage of the #[allow] attribute and suggests replacing it with the #[expect] attribute (See RFC 2383)

#[expect] attributes suppress the lint emission, but emit a warning, if the expectation is unfulfilled. This can be useful to be notified when the lint is no longer triggered.

We have a quite a few occurrences of #[allow] in our codebase, but should prefer #[expect] instead for the reasons above.

Describe the solution you'd like

Similar to #18503 we should roll this lint our gradually through crates before finally enabling at the workspace level so we don't have one big PR.

Describe alternatives you've considered

Don't enable this lint if we don't see value or if the false positive rate is too high (see below).

Additional context

I tested it for some of our current #[allow]s and it has some false positives, especially for dead_code and deprecated.

Example 1

datafusion/datafusion/core/benches/data_utils/mod.rs

Lines 37 to 40 in c8d26ba

	/// create an in-memory table given the partition len, array len, and batch size,
	/// and the result table will be of array_len in total, and then partitioned, and batched.
	#[allow(dead_code)]
	pub fn create_table_provider(

• If I switch this to #[expect(dead_code)] then clippy will flag it as this lint expectation is unfulfilled. But if I remove the lint then the dead code warning pops up anyway 🤔

Example 2

datafusion/datafusion/execution/src/disk_manager.rs

Lines 117 to 136 in c8d26ba

	/// Configuration for temporary disk access
	#[allow(deprecated)]
	#[deprecated(since = "48.0.0", note = "Use DiskManagerBuilder instead")]
	#[derive(Debug, Clone, Default)]
	pub enum DiskManagerConfig {
	/// Use the provided [DiskManager] instance
	Existing(Arc<DiskManager>),
	/// Create a new [DiskManager] that creates temporary files within
	/// a temporary directory chosen by the OS
	#[default]
	NewOs,
	/// Create a new [DiskManager] that creates temporary files within
	/// the specified directories
	NewSpecified(Vec<PathBuf>),
	/// Disable disk manager, attempts to create temporary files will error
	Disabled,
	}

• Switching this to #[expect(deprecated)] causes similar to above, but also flags NewOS variant as using deprecated variant. Something to do with how Default is derived?

(I say false positive but it seems more like a bug in clippy 🤔)

Maybe there are solutions for the above, I haven't looked into it much yet.

Another thing to keep in mind is how features interact, for example here:

datafusion/datafusion/datasource-parquet/src/reader.rs

Lines 290 to 304 in c8d26ba

	fn get_metadata<'a>(
	&'a mut self,
	#[allow(unused_variables)] options: Option<&'a ArrowReaderOptions>,
	) -> BoxFuture<'a, parquet::errors::Result<Arc<ParquetMetaData>>> {
	let object_meta = self.partitioned_file.object_meta.clone();
	let metadata_cache = Arc::clone(&self.metadata_cache);
	async move {
	#[cfg(feature = "parquet_encryption")]
	let file_decryption_properties = options
	.and_then(|o| o.file_decryption_properties())
	.map(Arc::clone);
	#[cfg(not(feature = "parquet_encryption"))]
	let file_decryption_properties = None;

• This #[allow] is for when parquet_encryption feature is not enabled, but if we change it to #[expect] blindly then I believe it'll cause issues when parquet_encryption is enabled, so we need to be careful

Tasks

• datafusion chore: enforce clippy::allow_attributes for 7 crates #19133
• datafusion-catalog Enable clippy::allow_attributes lint for datafusion-catalog #18973
• datafusion-catalog-listing chore: enforce clippy::allow_attributes for 7 crates #19133
• datafusion-common chore: enforce clippy::allow_attributes for common crates #18988
• datafusion-common-runtime chore: enforce clippy::allow_attributes for common crates #18988
• datafusion-datasource chore: enforce clippy::allow_attributes for datasource crates #19068
• datafusion-datasource-arrow chore: enforce clippy::allow_attributes for datasource crates #19068
• datafusion-datasource-avro chore: enforce clippy::allow_attributes for datasource crates #19068
• datafusion-datasource-csv chore: enforce clippy::allow_attributes for datasource crates #19068
• datafusion-datasource-json chore: enforce clippy::allow_attributes for datasource crates #19068
• datafusion-datasource-parquet chore: enforce clippy::allow_attributes for datasource crates #19068
• datafusion-doc chore: enforce clippy::allow_attributes for 7 crates #19133
• datafusion-execution chore: enforce clippy::allow_attributes for 7 crates #19133
• datafusion-expr chore: enforce clippy::allow_attributes for 7 crates #19133
• datafusion-expr-common chore: enforce clippy::allow_attributes for 7 crates #19133
• datafusion-ffi chore: enforce clippy::allow_attributes for datafusion-ffi crate #19480
• datafusion-functions chore: enforce clippy::allow_attributes for functions-* crates #18986
• datafusion-functions-aggregate chore: enforce clippy::allow_attributes for functions-* crates #18986
• datafusion-functions-aggregate-common chore: enforce clippy::allow_attributes for functions-* crates #18986
• datafusion-functions-nested chore: enforce clippy::allow_attributes for functions-* crates #18986
• datafusion-functions-table chore: enforce clippy::allow_attributes for functions-* crates #18986
• datafusion-functions-window chore: enforce clippy::allow_attributes for functions-* crates #18986
• datafusion-functions-window-common chore: enforce clippy::allow_attributes for functions-* crates #18986
• datafusion-macros chore: enforce clippy::allow_attributes for optimizer and macros #19310
• datafusion-optimizer chore: enforce clippy::allow_attributes for optimizer and macros #19310
• datafusion-physical-expr deny on allow_attributes lint in physical-plan #18983
• datafusion-physical-expr-adapter chore: enforce clippy::allow_attributes for physical crates #19185
• datafusion-physical-expr-common chore: enforce clippy::allow_attributes for physical crates #19185
• datafusion-physical-optimizer chore: enforce clippy::allow_attributes for physical crates #19185
• datafusion-physical-plan deny on allow_attributes lint in physical-plan #18983
• datafusion-proto chore: enforce clippy::allow_attributes for proto, pruning, session #19350
• datafusion-proto-common chore: enforce clippy::allow_attributes for 7 crates #19133
• datafusion-pruning chore: enforce clippy::allow_attributes for proto, pruning, session #19350
• datafusion-session chore: enforce clippy::allow_attributes for proto, pruning, session #19350
• datafusion-spark chore: enforce clippy::allow_attributes for spark,sql,sustrait #19309
• datafusion-sql chore: enforce clippy::allow_attributes for spark,sql,sustrait #19309
• datafusion-substrait chore: enforce clippy::allow_attributes for spark,sql,sustrait #19309
• Enable it globally after all the above tasks are finished Enforce clippy::allow_attributes globally across workspace #19576

Note: we would welcome PRs that can do multiple crates in a single PR (so long as the diff isn't too large). Felt a bit overwhelmed by the volume and speed of PRs that came in for #18503 😅

[Jefffrey]

Thoughts @2010YOUY01 ?

[2010YOUY01]

If there is a good way to deal with the mentioned feature flag issue, I think we should add this lint rule.

[Jefffrey]

If there is a good way to deal with the mentioned feature flag issue, I think we should add this lint rule.

It would look something like this:

    fn get_metadata<'a>(
        &'a mut self,
        #[cfg_attr(not(feature = "parquet_encryption"), expect(unused_variables))]
        options: Option<&'a ArrowReaderOptions>,
    ) -> BoxFuture<'a, parquet::errors::Result<Arc<ParquetMetaData>>> {

Doable, but a little verbose.

[2010YOUY01]

If there is a good way to deal with the mentioned feature flag issue, I think we should add this lint rule.

It would look something like this:

fn get_metadata<'a>(
    &'a mut self,
    #[cfg_attr(not(feature = "parquet_encryption"), expect(unused_variables))]
    options: Option<&'a ArrowReaderOptions>,
) -> BoxFuture<'a, parquet::errors::Result<Arc<ParquetMetaData>>> {

Doable, but a little verbose.

I think this marker describes the arg quite precisely: 'this arg is only used in parquet_encryption' feature.

So I'm +1 for this lint. Maybe we should add comment like the above explanation, to deal with the verbosity.

[Jefffrey]

I've marked it as good first issue, but note to anyone interested, please do not assign this issue to yourself as it is meant to be a tracking issue. Just pick a crate (or crates) that is not yet done/in-progress and leave a comment with your intention to work on those crates (to avoid overlap).

[YuraLitvinov]

Hello, if it's okay I am taking the physical-plan.

[carlosahs]

Hi! If no one else is working on datafusion-functions-nested, I can take that.

[chakkk309]

Hi all, I would like to take the following crates:

• datafusion-catalog
• datafusion-listing
• datafusion-common
• datafusion-common-runtime
• datafusion-datasource
• datafusion-datasource-arrow
• datafusion-datasource-avro
• datafusion-datasource-csv
• datafusion-datasource-json
• datafusion-datasource-parquet
• datafusion-doc
• datafusion-execution
• datafusion-expr
• datafusion-expr-common
• datafusion-ffi

[carlosahs]

@Jefffrey, for datafusion-functions-nested it suffices to add #![deny(clippy::allow_attributes)] as there is no usage of #[allow]. The same applies to datafusion-functions-table. If it's ok, I'll create a PR to enforce clippy::allow_attributes for all datafusion-functions-* crates as it looks like usage of #[allow] is low and to avoid raising multiple 1-liner PRs.

[Jefffrey]

@Jefffrey, for datafusion-functions-nested it suffices to add #![deny(clippy::allow_attributes)] as there is no usage of #[allow]. The same applies to datafusion-functions-table. If it's ok, I'll create a PR to enforce clippy::allow_attributes for all datafusion-functions-* crates as it looks like usage of #[allow] is low and to avoid raising multiple 1-liner PRs.

That sounds good, thanks