Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Substrait + identifier normalization flags issues #14832

Open
alamb opened this issue Feb 22, 2025 · 0 comments
Open

Substrait + identifier normalization flags issues #14832

alamb opened this issue Feb 22, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@alamb
Copy link
Contributor

alamb commented Feb 22, 2025

Describe the bug

This is a report from @lmwnshn and is part of

enable_ident_normalization: I think there may be some extra complications in my situation arising from the use of Substrait (or perhaps Substrait table names are treated as quoted?

In particular, with enable_ident_normalization=false, I cannot register lowercase tablenames and lowercase column names in the parquet files as shown here:

To Reproduce

Please see
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/run_datafusion_ident.py#L37

In particular, with enable_ident_normalization=false, I cannot register lowercase tablenames and lowercase column names in the parquet files as shown here:
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/run_datafusion_ident.py#L12-L19

So I had to hack the parquet files up a bit
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/fix_parquet.py#L11-L23

and switch to uppercased column names + register tables as uppercase
https://github.com/lmwnshn/15799-s25-project1-remnants/blob/main/run_datafusion.py#L12-L19
to get the Substrait plan to execute successfully.

Expected behavior

I expect that the I can register lowercase tablenames and lowercase column names in the parquet files

Additional context

No response

@alamb alamb added the bug Something isn't working label Feb 22, 2025
@alamb alamb changed the title Substrait may not respect identifier normalization flags Substrait + identifier normalization flags issues Feb 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant