Skip to content

Commit

Permalink
enhance row filter for Go Parquet compatibility with version-specific…
Browse files Browse the repository at this point in the history
… type handling
  • Loading branch information
kosiew committed Jan 28, 2025
1 parent 85d0569 commit 88ad42d
Showing 1 changed file with 8 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,14 @@ impl DatafusionArrowPredicate {
let expr = reassign_predicate_columns(candidate.expr, &schema, true)?;
// Apply additional type coercion for version 2.6 compatibility
// This ensures proper handling of type differences between parquet-go and arrow
let expr = expr.clone();
// For Go-generated Parquet files, we need to handle type conversion more carefully
let expr = if version == 2 {
// For version 2.0 files (like those from Go), apply special type handling
println!("==> Row filter: Applying Go Parquet compatibility handling");
reassign_predicate_columns(expr.clone(), &schema, true)?
} else {
expr
};
expr
} else {
println!("==> Row filter: Using standard type handling");
Expand Down

0 comments on commit 88ad42d

Please sign in to comment.