[Model][VLM] Add Qwen2.5-Omni model support (end-to-end full support)

[fyabc]

This draft PR adding support for Qwen2.5-Omni model (end-to-end full support).

This PR is a later version of #15130, it adds support for talker, code2wav, and an OmniLLMEngine class to manage the end-to-end audio generation process.
You can see #15130 for more details about Qwen2.5-Omni model architecture.

NOTE: Since this PR makes significant changes to vLLM, its a draft and will not be merged in the short term.

Requirements

This PR requires huggingface/transformers#36752.

pip install git+https://github.com/huggingface/transformers@f742a644ca32e65758c3adb36225aef1731bd2a8

Note: You need to install transformers from source from that branch

Example Usage

python examples/offline_inference/qwen2_5_omni/end2end.py --model Qwen/Qwen2.5-Omni-7B --prompt audio-in-video-v2 --enforce-eager --do-wave --voice-type m02 --warmup-voice-type m02

This command will print text output and generate .wav output files under current folder.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fyabc.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

I think we can further split this PR, with the first one (after Qwen2.5-Omni thinker only) adding prompt_embeds support to vLLM. For reference, here are some previous/ongoing efforts to add this feature:

• [Core] generate from input embeds #6869
• [Bugfix] add input embedding #11684
• [Core] [Bugfix] Add Input Embeddings #15428

[ywang96]

Thanks for this contribution! As we discussed offline, we'll be carefully reviewing this PR/design and think about how to enable end-to-end support for models like this with vLLM!

[Berrysoft]

Suggested change

	videos.append(fetch_and_read_video(audio_key))
	videos.append(fetch_and_read_video(ele[audio_key]))