Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

一直显示：开始生成...... bug ffmpeg Timeout ， self.stop.is_set() 一直返回是False #36

Open
gg22mm opened this issue Dec 22, 2024 · 1 comment
Open

一直显示：开始生成...... bug ffmpeg Timeout ， self.stop.is_set() 一直返回是False #36

gg22mm opened this issue Dec 22, 2024 · 1 comment

Comments

@gg22mm
Copy link

gg22mm commented Dec 22, 2024 •
edited
Loading

日志

第一部分生成的好好的：

Set seed to 1943014087
并行推理模式已开启
分桶处理模式已开启
Actual Input Reference Text: 喜悦哇塞，今天真是太棒了，悲伤。哎，生活怎么如此艰难？😔。
初始化用时1.09秒
Load voice cost: 1.39s
Flushing pipeline....
-------7、看看 [[None, MultimodalMessage(id=None, elem_id=None, elem_classes=None, name=None, text='您好，请问有什么可以帮到您？您可以在下方的输入框点击麦克风录制音频或直接输入文本与我聊天。', flushing=None, avatar='', files=[])], [{'text': '你好', 'files': []}, {'text': '开始生成..111111111....\n'}]] None True
Warming up THG Module...
[ASR] User input: 你好, cost: 0.01s
--------1、用户输入: 你好
-------2、用户数入格式化: [{'role': 'system', 'content': '你负责为一个语音聊天系统生成对话文本输出，使用短句，确保语气情感丰富、友好，并且响应迅速以保持用户的参与感。'}]
[THG] Start inferencing Video up
-----------3、llm回答... <transformers.generation.streamers.TextIteratorStreamer object at 0x7f5df8b95990>
################################### 888888888888888888888888888 ################################ 您好
---------------  3、llm回答句子 ----------- ['您好']
video in 25 FPS, audio idx in 50FPS
[THG] Processing audio:data/audio/warm_up.wav costs 258.71729850769043ms
14
################################### 888888888888888888888888888 ################################ 
---------------  3、llm回答句子 ----------- ['您好']
################################### 888888888888888888888888888 ################################ ！有什么
---------------  3、llm回答句子 ----------- ['您好！', '有什么']
---------------  44、llm回答句子 ----------- fp_flag: True 当前说的： 您好！ 长度： 3 块: 5
--------------- 4、回答压入队列 Put into queue --------------- : 您好！
[TTS] Get chunk from llm_queue: 您好！
############ 切分文本 ############
Actual Input Target Text:
。您好！
Actual Input Target Text (after sentence segmentation):
['您好！']
############ 提取文本Bert特征 ############
  0%|                                                                                                                                                                                                            | 0/1 [00:00<?, ?it/s]################################### 888888888888888888888888888 ################################ 我能
---------------  3、llm回答句子 ----------- ['有什么我能']
################################### 888888888888888888888888888 ################################ 帮
---------------  3、llm回答句子 ----------- ['有什么我能帮']
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.11it/s]
############ 推理 ############
Processed text from the frontend (per sentence): ['.您好!']
################################### 888888888888888888888888888 ################################ 您的
---------------  3、llm回答句子 ----------- ['有什么我能帮您的']
  0%|                                                                                                                                                                                                         | 0/1500 [00:00<?, ?it/s][Listener] Start yielding results from queue.
################################### 888888888888888888888888888 ################################ 吗
---------------  3、llm回答句子 ----------- ['有什么我能帮您的吗']
  0%|▏                                                                                                                                                                                                | 1/1500 [00:00<06:31,  3.83it/s]################################### 888888888888888888888888888 ################################ 
---------------  3、llm回答句子 ----------- ['有什么我能帮您的吗']
  0%|▎                                                                                                                                                                                                | 2/1500 [00:00<06:11,  4.03it/s]################################### 888888888888888888888888888 ################################ 
---------------  3、llm回答句子 ----------- ['有什么我能帮您的吗']
################################### 888888888888888888888888888 ################################ ？<|im_end|>
--------------- continue continue continue-----------
5、回答压入队列 Put into queue: 有什么我能帮您的吗
6、返回 Response: 您好！有什么我能帮您的吗

  2%|███▋                                                                                                                                                                                            | 29/1500 [00:04<03:05,  7.94it/s]T2S Decoding EOS [218 -> 249]
  2%|███▊                                                                                                                                                                                            | 30/1500 [00:04<03:47,  6.46it/s]
10.733	0.482	4.688	14.568
-------------------------------------------------------------- 32000 [0 0 0 ... 0 0 0]
Save audio ./workspaces/results/Avatar1/20241222_124728/audio/llm_response_audio_0.wav
推理用时19.75秒
Audio 0:Cost 20.839378595352173 secs
[TTS] Get chunk from llm_queue: 有什么我能帮您的吗
############ 切分文本 ############
Actual Input Target Text:
有什么我能帮您的吗
Actual Input Target Text (after sentence segmentation):
['有什么我能帮您的吗。']
############ 提取文本Bert特征 ############
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  3.57it/s]
############ 推理 ############
Processed text from the frontend (per sentence): ['有什么我能帮您的吗.']
  3%|█████▊                                                                                                                                                                                          | 45/1500 [00:02<00:45, 31.79it/s]T2S Decoding EOS [218 -> 264]
  3%|█████▊

第二部就坑了，上面生成的下面读不出来

THG] Video up: Total process time of 14 frames including saving images = 26.159016847610474s
[THG] Get audio from tts_queue: ./workspaces/results/Avatar1/20241222_124728/audio/llm_response_audio_0.wav
[THG] Start inferencing Video 0
video in 25 FPS, audio idx in 50FPS
[THG] Processing audio:./workspaces/results/Avatar1/20241222_124728/audio/llm_response_audio_0.wav costs 111.1297607421875ms
38
ffmpeg Timeout
Time cost: 
Index     Duration     LLM       TTS       THG       ffmpeg    Cost

-------9、看看----------- [[None, MultimodalMessage(id=None, elem_id=None, elem_classes=None, name=None, text='您好，请问有什么可以帮到您？您可以在下方的输入框点击麦克风录制音频或直接输入文本与我聊天。', flushing=None, avatar='', files=[])], [{'text': '你好', 'files': []}, {'text': '开始生成..111111111....\n'}]]
-------10、看看----------- [[None, MultimodalMessage(id=None, elem_id=None, elem_classes=None, name=None, text='您好，请问有什么可以帮到您？您可以在下方的输入框点击麦克风录制音频或直接输入文本与我聊天。', flushing=None, avatar='', files=[])], [{'text': '你好', 'files': []}, {'text': '开始生成..111111111....\n'}]]
[THG] Video 0: Total process time of 38 frames including saving images = 6.291346788406372s
31.573	0.338	2.444	10.111
-------------------------------------------------------------- 32000 [ 0 -1  0 ...  0  0  0]
Save audio ./workspaces/results/Avatar1/20241222_124728/audio/llm_response_audio_1.wav
推理用时12.90秒
Audio 1:Cost 13.762666940689087 secs
[TTS] Get chunk from llm_queue: None
[THG] Get audio from tts_queue: ./workspaces/results/Avatar1/20241222_124728/audio/llm_response_audio_1.wav
[THG] Start inferencing Video 1
video in 25 FPS, audio idx in 50FPS
[THG] Processing audio:./workspaces/results/Avatar1/20241222_124728/audio/llm_response_audio_1.wav costs 204.38051223754883ms
53
[THG] Video 1: Total process time of 53 frames including saving images = 18.83054256439209s
[THG] Get audio from tts_queue: None
Finish pipeline......
^Z
[1]+  Stopped                 python app.py
(dh) λ localhost /paddle/www/txsb/api/VideoChat-cascade_only ^C
(dh) λ localhost /paddle/www/txsb/api/VideoChat-cascade_only ps -aux | grep app.py | awk '{print "kill -9 "$2}' | sh
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
sh: 2: kill: No such process

(dh) λ localhost /paddle/www/txsb/api/VideoChat-cascade_only 
(dh) λ localhost /paddle/www/txsb/api/VideoChat-cascade_only 
(dh) λ localhost /paddle/www/txsb/api/VideoChat-cascade_only 
(dh) λ localhost /paddle/www/txsb/api/VideoChat-cascade_only 
[1]+  Killed                  python app.py
(dh) λ localhost /paddle/www/txsb/api/VideoChat-cascade_only

下面这这一段代码一直读不出队列中的内容

# Merge all videos
            if not self.stop.is_set() and videos_path:
                merged_video_path = merge_videos(videos_path)
                # video mp4 format
                llm_response_txt = user_chatbot[-1][1]["text"]  + f"""<video src="{merged_video_path}"></video>\n""" 
                # print('-------------ddddd-----------',llm_response_txt)
                # First Packet RT
                llm_response_txt = llm_response_txt + f"首包延迟：{round(self.time_cost[-1][0] + self.asr_cost, 2)}s\n"
                user_chatbot[-1][1] = {
                        "text": llm_response_txt,
                        "flushing": False
                    }

            print('-------9、看看-----------',user_chatbot)

--------------- 3、llm回答句子 ----------- ['有什么可以帮助你的吗']
Exception in thread Thread-7 (ffmpeg_worker):
Traceback (most recent call last):
File "/home/data/miniconda3/envs/dh/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/home/data/miniconda3/envs/dh/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/paddle/www/txsb/api/VideoChat-cascade_only/src/pipeline.py", line 371, in ffmpeg_worker
llm_response_audio = self.thg_queue.get(timeout=1)
File "/home/data/miniconda3/envs/dh/lib/python3.10/queue.py", line 179, in get
raise Empty
_queue.Empty

(dh) λ localhost /paddle/www/txsb/api/VideoChat-cascade_only pip list
Package                       Version
----------------------------- -----------
absl-py                       2.1.0
accelerate                    0.28.0
addict                        2.4.0
aiofiles                      23.2.1
aiohappyeyeballs              2.4.4
aiohttp                       3.11.11
aiosignal                     1.3.2
aliyun-python-sdk-core        2.16.0
aliyun-python-sdk-kms         2.16.5
annotated-types               0.7.0
antlr4-python3-runtime        4.9.3
anyio                         4.7.0
astunparse                    1.6.3
async-timeout                 5.0.1
attrs                         24.3.0
audioread                     3.0.1
av                            12.3.0
beautifulsoup4                4.12.3
cachetools                    5.5.0
certifi                       2024.12.14
cffi                          1.17.1
chardet                       5.2.0
charset-normalizer            3.4.0
chumpy                        0.70
click                         8.1.8
cn2an                         0.5.22
colorama                      0.4.6
coloredlogs                   15.0.1
contourpy                     1.3.1
crcmod                        1.7
cryptography                  44.0.0
ctranslate2                   4.5.0
cycler                        0.12.1
Cython                        3.0.11
dashscope                     1.20.14
datasets                      3.2.0
decorator                     4.4.2
diffusers                     0.27.2
dill                          0.3.8
Distance                      0.1.3
distro                        1.9.0
dnspython                     2.7.0
edge-tts                      7.0.0
editdistance                  0.8.1
einops                        0.8.0
email_validator               2.2.0
exceptiongroup                1.2.2
fastapi                       0.115.6
fastapi-cli                   0.0.7
faster-whisper                1.0.3
ffmpeg-python                 0.2.0
ffmpy                         0.5.0
filelock                      3.14.0
flatbuffers                   24.3.25
fonttools                     4.55.3
frozenlist                    1.5.0
fsspec                        2024.9.0
funasr                        1.1.6
future                        1.0.0
g2p-en                        2.1.0
g2pk2                         0.0.3
gast                          0.6.0
gdown                         5.2.0
google-auth                   2.37.0
google-auth-oauthlib          1.0.0
google-pasta                  0.2.0
gradio                        5.9.1
gradio_client                 1.5.2
grpcio                        1.68.1
h11                           0.14.0
h5py                          3.12.1
httpcore                      1.0.7
httptools                     0.6.4
httpx                         0.28.1
huggingface-hub               0.25.2
humanfriendly                 10.0
hydra-core                    1.3.2
idna                          3.10
imageio                       2.35.1
imageio-ffmpeg                0.5.1
importlib_metadata            8.5.0
importlib_resources           6.4.5
inflect                       7.4.0
itsdangerous                  2.2.0
jaconv                        0.4.0
jamo                          0.4.1
jieba                         0.42.1
jieba-fast                    0.53
Jinja2                        3.1.5
jiter                         0.8.2
jmespath                      0.10.0
joblib                        1.4.2
json-tricks                   3.17.3
kagglehub                     0.3.6
kaldiio                       2.18.0
keras                         2.14.0
kiwisolver                    1.4.7
ko-pron                       1.3
LangSegment                   0.3.5
libclang                      18.1.1
librosa                       0.9.2
lightning-utilities           0.11.9
llvmlite                      0.39.1
Markdown                      3.7
markdown-it-py                3.0.0
MarkupSafe                    2.1.5
matplotlib                    3.10.0
mdurl                         0.1.2
ml-dtypes                     0.2.0
mmcv                          2.1.0
mmdet                         3.3.0
mmengine                      0.10.5
mmpose                        1.3.2
model-index                   0.1.11
modelscope                    1.18.0
modelscope_studio             0.5.2
more-itertools                10.5.0
moviepy                       1.0.3
mpmath                        1.3.0
multidict                     6.1.0
multiprocess                  0.70.16
munkres                       1.1.4
networkx                      3.4.2
nltk                          3.9.1
numba                         0.56.4
numpy                         1.23.5
nvidia-cublas-cu12            12.1.3.1
nvidia-cuda-cupti-cu12        12.1.105
nvidia-cuda-nvrtc-cu12        12.1.105
nvidia-cuda-runtime-cu12      12.1.105
nvidia-cudnn-cu12             8.9.2.26
nvidia-cufft-cu12             11.0.2.54
nvidia-curand-cu12            10.3.2.106
nvidia-cusolver-cu12          11.4.5.107
nvidia-cusparse-cu12          12.1.0.106
nvidia-nccl-cu12              2.18.1
nvidia-nvjitlink-cu12         12.4.127
nvidia-nvtx-cu12              12.1.105
oauthlib                      3.2.2
omegaconf                     2.3.0
onnxruntime                   1.20.1
openai                        1.43.0
OpenCC                        1.1.1
opencv-python                 4.9.0.80
opendatalab                   0.0.10
openmim                       0.3.9
openxlab                      0.1.2
opt_einsum                    3.4.0
optimum                       1.23.3
ordered-set                   4.1.0
orjson                        3.10.12
oss2                          2.17.0
packaging                     24.2
pandas                        2.2.3
pillow                        10.4.0
pip                           24.2
platformdirs                  4.3.6
pooch                         1.8.2
proces                        0.1.7
proglog                       0.1.10
propcache                     0.2.1
protobuf                      4.25.5
psutil                        5.9.8
py3langid                     0.3.0
pyarrow                       18.1.0
pyasn1                        0.6.1
pyasn1_modules                0.4.1
pycocotools                   2.0.8
pycparser                     2.22
pycryptodome                  3.21.0
pydantic                      2.10.4
pydantic_core                 2.27.2
pydantic-extra-types          2.10.1
pydantic-settings             2.7.0
pydub                         0.25.1
Pygments                      2.18.0
pyjyutping                    1.0.0
pynndescent                   0.5.13
pyopenjtalk                   0.3.4
pyparsing                     3.2.0
pypinyin                      0.52.0
PySocks                       1.7.1
python-dateutil               2.9.0.post0
python-dotenv                 1.0.1
python-mecab-ko               1.3.7
python-mecab-ko-dic           2.1.1.post2
python-multipart              0.0.20
pytorch-lightning             2.4.0
pytorch-wpe                   0.0.1
pytz                          2023.4
PyYAML                        6.0.2
regex                         2024.11.6
requests                      2.32.3
requests-oauthlib             2.0.0
resampy                       0.4.3
rich                          13.9.4
rich-toolkit                  0.12.0
rotary-embedding-torch        0.7.0
rsa                           4.9
ruff                          0.8.4
safehttpx                     0.1.6
safetensors                   0.4.5
scikit-learn                  1.6.0
scipy                         1.13.1
semantic-version              2.10.0
sentencepiece                 0.2.0
setuptools                    60.2.0
shapely                       2.0.6
shellingham                   1.5.4
six                           1.17.0
sniffio                       1.3.1
soundfile                     0.12.1
soupsieve                     2.6
spaces                        0.30.0
srt                           3.5.3
starlette                     0.41.3
sympy                         1.13.1
tabulate                      0.9.0
tensorboard                   2.14.0
tensorboard-data-server       0.7.2
tensorboardX                  2.6.2.2
tensorflow                    2.14.0
tensorflow-estimator          2.14.0
tensorflow-io-gcs-filesystem  0.37.1
termcolor                     2.5.0
terminaltables                3.1.10
threadpoolctl                 3.5.0
tokenizers                    0.21.0
tomli                         2.2.1
tomlkit                       0.12.0
torch                         2.1.2
torch-complex                 0.4.4
torchaudio                    2.1.2
torchmetrics                  1.6.0
torchvision                   0.16.2
tqdm                          4.67.1
transformers                  4.47.1
transformers-stream-generator 0.0.5
triton                        2.1.0
typeguard                     4.4.1
typer                         0.15.1
typing_extensions             4.12.2
tzdata                        2024.2
ujson                         5.10.0
umap-learn                    0.5.7
urllib3                       2.2.3
uvicorn                       0.34.0
uvloop                        0.21.0
watchfiles                    1.0.3
websocket-client              1.8.0
websockets                    12.0
Werkzeug                      3.1.3
wheel                         0.44.0
wordsegment                   1.3.1
wrapt                         1.14.1
xtcocotools                   1.14.3
xxhash                        3.5.0
yapf                          0.43.0
yarl                          1.18.3
zipp                          3.21.0
@gg22mm gg22mm changed the title 一直显示生在生成bug 一直显示正：开始生成bug Dec 22, 2024
@gg22mm gg22mm changed the title 一直显示正：开始生成bug 一直显示正：开始生成bug ffmpeg Timeout Dec 22, 2024
@gg22mm gg22mm changed the title 一直显示正：开始生成bug ffmpeg Timeout 一直显示：开始生成...... bug ffmpeg Timeout Dec 22, 2024
@gg22mm gg22mm changed the title 一直显示：开始生成...... bug ffmpeg Timeout 一直显示：开始生成...... bug ffmpeg Timeout ， self.stop.is_set() 直接返回是空的 Dec 22, 2024
@gg22mm gg22mm changed the title 一直显示：开始生成...... bug ffmpeg Timeout ， self.stop.is_set() 直接返回是空的 一直显示：开始生成...... bug ffmpeg Timeout ， self.stop.is_set() 一直返回是False Dec 22, 2024
@luan78zaoha
Copy link

我也是一样的问题，一直在循环播放，然后TTS timeout

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gg22mm @luan78zaoha