Skip to content

Commit

Permalink
ORC-1568: Use readDiskRanges if orc.use.zerocopy is enabled
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to use legacy `readDiskRanges` method if `orc.use.zerocopy` is enabled.

### Why are the changes needed?

Since `orc.use.zerocopy` is disabled by default, Apache ORC will use `Vectored IO` still after this PR.

The main background of this PR is that we had better keep the original behavior until Apache ORC take advantages of new `Vectored IO` API with ZeroCopy code path. We can improve later at Apache ORC 2.1.x.

Note that `ZeroCopy` reader is known to be used by `Apache Hive LLAP` (`LlapOrcCacheLoader`, `LlapRecordReaderUtil`, `OrcEncodedDataReader`) like the following. However, `Apache Hive 4.0.0` is incompatible with Apache ORC 2.0.0 because it still requires Java 8 minimum support.

- https://github.com/apache/hive/blob/b33b3d3454cc9c65a1879c68679f33f207f21c0e/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/LlapOrcCacheLoader.java#L92
```java
boolean useZeroCopy = (daemonConf != null) && OrcConf.USE_ZEROCOPY.getBoolean(daemonConf);
```

### How was this patch tested?

Pass the CIs.

Closes #1723 from dongjoon-hyun/ORC-1568.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed Jan 3, 2024
1 parent b35bb59 commit c2fad3d
Showing 1 changed file with 6 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,12 @@ public OrcProto.StripeFooter readStripeFooter(StripeInformation stripe) throws I
public BufferChunkList readFileData(BufferChunkList range,
boolean doForceDirect
) throws IOException {
RecordReaderUtils.readDiskRangesVectored(file, range, doForceDirect);
if (zcr == null) {
RecordReaderUtils.readDiskRangesVectored(file, range, doForceDirect);
} else {
RecordReaderUtils.readDiskRanges(file, zcr, range, doForceDirect,
minSeekSize, minSeekSizeTolerance);
}
return range;
}

Expand Down

0 comments on commit c2fad3d

Please sign in to comment.