Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Java][Gandiva] Synchronize some methods on Projector. #601

Closed
lriggs opened this issue Feb 12, 2025 · 4 comments · Fixed by #602
Closed

[Java][Gandiva] Synchronize some methods on Projector. #601

lriggs opened this issue Feb 12, 2025 · 4 comments · Fixed by #602
Labels
Type: bug Something isn't working Type: enhancement New feature or request
Milestone

Comments

@lriggs
Copy link
Contributor

lriggs commented Feb 12, 2025

Describe the enhancement requested

Our product, which uses Gandiva, was encountering errors from LLVM during intense query processing stress tests. I traced the issue back to the JIT upgrade (apache/arrow#39098), but it seems it only started occurring then because the new JIT engine requires unique symbol names unlike the old engine.

The error from llvm is: Duplicate definition of symbol 'expr_0_0'.

After updating the llvm version to 18 (apache/arrow#45114) you can see this error occasionally when running the Gandiva ProjectorTest.

Failed to make LLVM module due to Failed to add IR module to LLJIT: Duplicate definition of symbol 'expr_0_0'

	at org.apache.arrow.gandiva.evaluator.JniWrapper.buildProjector(Native Method)
	at org.apache.arrow.gandiva.evaluator.Projector.make(Projector.java:206)
	at org.apache.arrow.gandiva.evaluator.Projector.make(Projector.java:70)
	at org.apache.arrow.gandiva.evaluator.ProjectorTest.lambda$testMakeProjectorParallel$1(ProjectorTest.java:138)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
[WARNING] Tests run: 44, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 6.656 s -- in org.apache.arrow.gandiva.evaluator.ProjectorTest

Adding synchronization the the make and evaluate methods in gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java will fix the issue.

@lriggs lriggs added the Type: enhancement New feature or request label Feb 12, 2025
@lriggs
Copy link
Contributor Author

lriggs commented Feb 12, 2025

diff --git a/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java b/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java
index 951cebc88e..a67d20db6b 100644
--- a/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java
+++ b/java/gandiva/src/main/java/org/apache/arrow/gandiva/evaluator/Projector.java
@@ -187,7 +187,7 @@ public class Projector {
    * @param configurationId Custom configuration created through config builder.
    * @return A native evaluator object that can be used to invoke these projections on a RecordBatch
    */
-  public static Projector make(
+  public static synchronized Projector make(
       Schema schema,
       List<ExpressionTree> exprs,
       SelectionVectorType selectionVectorType,
@@ -313,7 +313,7 @@ public class Projector {
         outColumns);
   }
 
-  private void evaluate(
+  private synchronized void evaluate(
       int numRows,
       List<ArrowBuf> buffers,
       List<ArrowBuffer> buffersLayout,

lriggs added a commit to lriggs/arrow-java that referenced this issue Feb 12, 2025
@lidavidm
Copy link
Member

I believe there's efforts to revert back to the old JIT due to various problems with the new one

@lidavidm
Copy link
Member

Ah I see it looks like we upgraded LLVM instead apache/arrow#45114

@lriggs
Copy link
Contributor Author

lriggs commented Feb 13, 2025

Upgrading llvm fixed the symbol link errors, but after running the disabled tests I found this new issue.

lriggs added a commit to lriggs/arrow-java that referenced this issue Feb 20, 2025
lriggs added a commit to lriggs/arrow-java that referenced this issue Feb 20, 2025
@lidavidm lidavidm added this to the 18.3.0 milestone Feb 21, 2025
@lidavidm lidavidm added the Type: bug Something isn't working label Feb 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: bug Something isn't working Type: enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants