Add support for resolution alignment during encoding #85
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds two new encoder factories:
SimulcastAlignedVideoEncoderFactory
DefaultAlignedVideoEncoderFactory
These should be generally always preferred to be used instead of the default
SimulcastVideoEncoderFactory
andDefaultVideoEncoderFactory
. The difference is that the new factories allow to specify a resolution alignment requirement (resolutionAdjustment
in the constructor). HW encoders for VP8 (and H264) require a 16x16 aligned resolutions.This code is based on the original source from:
https://github.com/shiguredo/sora-android-sdk/blob/3cc88e806ab2f2327bf3042072e98d6da9df4408/sora-android-sdk/src/main/kotlin/jp/shiguredo/sora/sdk/codec/HardwareVideoEncoderWrapperFactory.kt
Credit and thanks go to Shiguredo, Inc.
This is especially a problem while simulcasting - webrtc computes the resolutions based on the
scaleResolutionDownBy
defined inRtpParameters.Encodings
and these resolutions are not 16x16 and this leads to various hard to debug problems with the encoder. WebRTC allows you to override thegetEncoderInfo()
inHardwareVideoEncoder.java
and define the alignment but in most cases it will not crop the resolution and instead it will just adjust thescaleResolutionDownBy
you defined. This behaviour is defined here. So from 1, 2, 4 (full, half, quarter resolution) it can create 1, 1, 1 to match the alignment and this is not a correct approach. In our tests the encoder would not work correctly and would drop frames in simulcast scenarios (1280x720, 3 simulcast layers - 1, 2, 4. This breaks if you are stream with phone oriented to left-orientation on Pixel devices).Relevant issues:
https://bugs.chromium.org/p/chromium/issues/detail?id=1084702
https://bugs.chromium.org/p/chromium/issues/detail?id=1351371
https://bugs.chromium.org/p/webrtc/issues/detail?id=12328