Audio Support

[TomMD]

It might require more dependencies, but supporting audio seems like an obvious next step. I don't have any specifics here, such as a pre-existing audio frame type already available on Hackage, so this requires quite a bit of fleshing out.

[acowley]

I did play around with SDL2 audio before, and it was relatively straightforward. In the absence of any other feelings, I'd give that a shot again. I haven't yet looked at how ffmpeg bubbles up the audio data, though.

If we go that way, we would provide a callback to SDL that would shuffle the data from ffmpeg into an SDL buffer.

[leftaroundabout]

Why the need for anything like SDL? Couldn't we just have something quite like imageReader, except that it would load chunks of audio data instead of Images?

[acowley]

Yeah, that's the right way to do it in the core library. An example executable showing SDL integration is probably enough to show users a sample complete pipeline.

[DiegoNolan]

I've started working on this but boy is this library difficult to work with. You can see my progress here.

DiegoNolan@f3c3f52

There are some things I want out of this. I want to be able to have an audio reader that gives out the data in chunks. I want to be able to resample audio to combine them and overlay them over videos or extract the audio from the video. I want this as a cleaner abstraction that what FFmpeg provides in the C interface which is quite cumbersome.

I'm not sure how we should represent the audio. FFmpeg has AVSampleFormat. They represent audio in unsigned and signed 8, 16, and 32 bit arrays as well as floats and doubles. It seems like it would be better to have a common format that different audio encodings could use to be used together. The presence of channels complicates things quite a bit since it is essentially multiple vectors of audio.

Just wanted to see your feedback on API design and the progress I've made so far. We could also look at SFML which provides a way to read audio but it doesn't have as much codec support as FFmpeg. It also may provide a newer perspective when compared to SDL.

I also think ideally there should be a pipes interface for all of this to really make it user friendly and usable but this is a lot more work down the line.

[leftaroundabout]

Kudos for all that hard work!

I think an FFmpeg library should keep as close to what FFmpeg spits out as possible while being still safe to use, and not going completely against the grain of the language.

Sure, for end applications we'd eventually want higher-level interfaces – I personally hope to some day get timed-media ready, with no such notion as sample rate etc. (working completely in frequency space). But that sort of stuff has nothing to do with FFmpeg.

[DiegoNolan]

Hmm. I almost have it working but I have a bug. When I extract audio and write it to a mp3 or something there is a bunch of distortion and it doubles the length. Can't seem to figure out where the bug is.

[leftaroundabout]

Blind guess: “bunch of distortion and doubles the length” sounds like stereo-interlacing gone awry. Does the mp3 file say it's mono?

[DiegoNolan]

I also get a seg fault with another video file. If I do ffprode on each of them they are mono. I don't try resampling either. Just going from mp4 with an aac audio stream to a simple aac.

[DiegoNolan]

Well I figured out the seg fault issue but not the other.

[DiegoNolan]

Okay, I was able to extract the audio from a video but I had to remove the resampling part. Also I'm unsure how to represent the frame data. I was only able to do it by passing the decoding frames directly to be re encoding.

https://ffmpeg.org/doxygen/3.1/swresample_8c.html#a39cb25282265493348bb8604120b87b9

swr_convert seems to take an array of arrays of fixed size. AVFrame does something similar. I'm guessing each array represents a channel of audio.