Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Getting 500 Internal Server Error / Exception in ASGI application #14

Open
andy108369 opened this issue Aug 6, 2024 · 16 comments
Open

Getting 500 Internal Server Error / Exception in ASGI application #14

andy108369 opened this issue Aug 6, 2024 · 16 comments
Labels
bug Something isn't working

Comments

@andy108369
Copy link

Here is a small write-up I've build and use it:

(.venv) root@node2:~# huggingface-cli download meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --local-dir /data/test-llama31-405b-fp8 --token $HF_TOKEN 
Fetching 242 files:   0%|                                                                                            | 0/242 [00:00<?, ?it/s]Downloading 'model-00004-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00004-of-00109.safetensors.376536d7284c6a79a65ee8e3fb3ee3fdf1a13040f72ac31ca7d2e4b13e7d52a3.incomplete'
Downloading 'model-00007-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00007-of-00109.safetensors.1fcdeb3bc4556d7da5bdf242919dc50df950563d8191d3be518fb001ab3e6331.incomplete'
Downloading 'model-00006-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00006-of-00109.safetensors.877173ef17665d02a697423e2b5c1e0fd8b3f50a8ba5c09b413896027c487789.incomplete'
Downloading 'model-00001-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00001-of-00109.safetensors.27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf.incomplete'
Downloading 'model-00003-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00003-of-00109.safetensors.26bb5d1cd5b224dee88f442b1a6acdf196a5d647d5af6834c62aa6a2a01bdd05.incomplete'
Downloading 'model-00005-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00005-of-00109.safetensors.fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931.incomplete'
Downloading 'model-00002-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00002-of-00109.safetensors.efef014770d588f4206f2da94ac251db8958f6a8ce1e38762d28c044bb015198.incomplete'
Downloading 'model-00008-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00008-of-00109.safetensors.410058afd38e1cdbca5a1a9cdae4e25f6d006763c3b224b6101611049458e109.incomplete'
Fetching 242 files:   2%|██                                                                                  | 6/242 [00:01<01:09,  3.41it/s]
Downloading 'model-00009-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00009-of-00109.safetensors.c41de3c512d78c8de6266adffa0db49850ee4692a6d74c2084f6519c450abdd6.incomplete'
Downloading 'model-00010-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00010-of-00109.safetensors.f919b1ef9202fe7b15871fc3f18610d3f799f8bbdd7b4bbfddc3ce7dc652fd21.incomplete'
Downloading 'model-00011-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00011-of-00109.safetensors.b235950238fed5a7260863c9f38efeea90d919467156b9d80d1301429b2c82f3.incomplete'
Traceback (most recent call last):
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://172.16.50.10:8090/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00109.safetensors%3B+filename%3D%22model-00001-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjdkNjc2MGUwMGYzYjhmY2Y2YWI3ODI0YTJkZTUwZjA3ZjU5Y2MwOGIwM2QyNjNiNmMwZDU2MmRhZjg2MjJiZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=fr9up4qmn0-~hwfDOHbqsKbRUKmaY~arE~9CHFuq5soQPLXTV4cwysE2REWplGAadJZCJxlDWUxM6ira3HjnkFt4hqtzSG2Sb3nDJOQ-8lRLD7T2XUIgLtGiwn-zcIaXOOerHg9~rI8R~Q7QpGX078lpTpml0-chIzB8FrHmmMAfVCvnDsGIsT2FYV-hqR2m-YWIlAty7a-WlBykkDpkaQNu3iz-VYztzJviTVQo4IDRd~6P6euGLW5iyTEzQsi1-gYEm6dBfyrazhvHqEzLr56Tm0Y43AewnhYoWC5a~onjcAy502CdYKUcDDCzSjlsaMZ7baHLNu7YOdPwPYCuXA__&Key-Pair-Id=K24J24Z295AEI9

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.venv/bin/huggingface-cli", line 8, in <module>
    sys.exit(main())
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/huggingface_cli.py", line 52, in main
    service.run()
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/download.py", line 146, in run
    print(self._download())  # Print path to downloaded files
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/download.py", line 180, in _download
    return snapshot_download(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 299, in snapshot_download
    thread_map(
  File "/root/.venv/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
  File "/root/.venv/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
  File "/root/.venv/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 273, in _inner_hf_hub_download
    return hf_hub_download(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
    return f(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1220, in hf_hub_download
    return _hf_hub_download_to_local_dir(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1515, in _hf_hub_download_to_local_dir
    _download_to_tmp_and_move(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1915, in _download_to_tmp_and_move
    http_get(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 463, in http_get
    r = _request_wrapper(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 400, in _request_wrapper
    hf_raise_for_status(response)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 371, in hf_raise_for_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: http://172.16.50.10:8090/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00109.safetensors%3B+filename%3D%22model-00001-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjdkNjc2MGUwMGYzYjhmY2Y2YWI3ODI0YTJkZTUwZjA3ZjU5Y2MwOGIwM2QyNjNiNmMwZDU2MmRhZjg2MjJiZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=fr9up4qmn0-~hwfDOHbqsKbRUKmaY~arE~9CHFuq5soQPLXTV4cwysE2REWplGAadJZCJxlDWUxM6ira3HjnkFt4hqtzSG2Sb3nDJOQ-8lRLD7T2XUIgLtGiwn-zcIaXOOerHg9~rI8R~Q7QpGX078lpTpml0-chIzB8FrHmmMAfVCvnDsGIsT2FYV-hqR2m-YWIlAty7a-WlBykkDpkaQNu3iz-VYztzJviTVQo4IDRd~6P6euGLW5iyTEzQsi1-gYEm6dBfyrazhvHqEzLr56Tm0Y43AewnhYoWC5a~onjcAy502CdYKUcDDCzSjlsaMZ7baHLNu7YOdPwPYCuXA__&Key-Pair-Id=K24J24Z295AEI9
(.venv) root@node2:~# 

root@node1:~# curl -I http://localhost:8090/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/model-00005-of-00109.safetensors
HTTP/1.1 500 Internal Server Error
date: Tue, 06 Aug 2024 14:39:15 GMT
server: uvicorn
content-length: 21
content-type: text/plain; charset=utf-8

root@node1:~# Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/model-00005-of-00109.safetensors "HTTP/1.1 403 Forbidden"
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | INFO | stdout | INFO:     127.0.0.1:37654 - "HEAD /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/model-00005-of-00109.safetensors HTTP/1.1" 500 Internal Server Error
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr | Traceback (most recent call last):
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     result = await app(  # type: ignore[func-returns-value]
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     return await self.app(scope, receive, send)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     await super().__call__(scope, receive, send)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     raise exc
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     await self.app(scope, receive, _send)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     raise exc
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     await app(scope, receive, sender)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     await route.handle(scope, receive, send)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     await self.app(scope, receive, send)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     raise exc
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     await app(scope, receive, sender)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     response = await func(request)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     raw_response = await run_endpoint_function(
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     return await dependant.call(**values)
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/olah/olah/server.py", line 402, in lfs_head
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     status_code = await generator.__anext__()
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/olah/olah/proxy/files.py", line 257, in _file_realtime_stream
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     status_code, head_info, content = await _file_full_header(
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |   File "/data/olah/olah/olah/proxy/files.py", line 89, in _file_full_header
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr |     raise Exception(f"Unexpected HTTP status code {response.status_code}")
Aug 06 14:39:16 node1 python[2932916]: 2024-08-06 14:39:16 | ERROR | stderr | Exception: Unexpected HTTP status code 403
@jstzwj
Copy link
Contributor

jstzwj commented Aug 6, 2024

Thank you very much for providing such detailed information and description in your issue.
Based on the logs, I think the issue is due to the sevrer not forwarding the user's token correctly. I have fixed some bugs about hf_tokens in the dev branch. You can try to delete the cache folder and then checkout to dev branch to see if this resolves the issue.
I will release the dev branch in a few days.

@andy108369
Copy link
Author

Thank you for you prompt response @jstzwj !
I'll give it a shot tomorrow morning (GMT+2 TZ here)

@andy108369
Copy link
Author

andy108369 commented Aug 7, 2024 •
edited
Loading

Thank you very much for providing such detailed information and description in your issue. Based on the logs, I think the issue is due to the sevrer not forwarding the user's token correctly. I have fixed some bugs about hf_tokens in the dev branch. You can try to delete the cache folder and then checkout to dev branch to see if this resolves the issue. I will release the dev branch in a few days.

Hey @jstzwj , I'm still getting this error:

latest commit from the dev branch

(.venv) olah@node1:~$ pip freeze |grep olah
-e git+https://github.com/vtuber-plan/olah.git@6c8be801fbefa232c29f8d55df473942cb13612f#egg=olah

Client logs

(.venv) root@node2:~# rm -rf ~/.cache/huggingface /data/test-llama31-405b-fp8

(.venv) root@node2:~# huggingface-cli download meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --local-dir /data/test-llama31-405b-fp8 
Fetching 242 files:   0%|                                                                                                                                                                                                                                               | 0/242 [00:00<?, ?it/s]Downloading 'model-00001-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00001-of-00109.safetensors.27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf.incomplete'
Downloading 'generation_config.json' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/generation_config.json.c85e356463cb94425264a907d7c3fea022326e80.incomplete'
Downloading '.gitattributes' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/.gitattributes.a6344aac8c09253b3b630fb776ae94478aa0275b.incomplete'
Downloading 'config.json' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/config.json.ed166d7b95e1910b9e567693971c6f220cd68915.incomplete'
Downloading 'model-00081-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00081-of-00109.safetensors.e321c3213f419390a815acdf33548d68dc9322379f93596b60b3359eb82b0fc2.incomplete'
Downloading 'model-00042-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00042-of-00109.safetensors.211f7316d4a389f2676d4fd68787d081ef679d41786ddd9d4464c97e4a1a7bda.incomplete'
.gitattributes: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.52k/1.52k [00:00<00:00, 5.57MB/s]
Download complete. Moving file to /data/test-llama31-405b-fp8/.gitattributes                                                                                                                                                                                        | 0.00/1.52k [00:00<?, ?B/s]
Fetching 242 files:   0%|▉                                                                                                                                                                                                                                      | 1/242 [00:02<10:05,  2.51s/it]
generation_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 189/189 [00:00<00:00, 784kB/s]
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22.5k/22.5k [00:00<00:00, 16.1MB/s]
Download complete. Moving file to /data/test-llama31-405b-fp8/generation_config.json
Download complete. Moving file to /data/test-llama31-405b-fp8/config.json
Traceback (most recent call last):
  File "/root/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
  File "/root/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
  File "/root/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 464, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.venv/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/root/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
  File "/root/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/root/.venv/lib/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/root/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
  File "/root/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
  File "/root/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 464, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1751, in _get_metadata_or_catch_error
    metadata = get_hf_file_metadata(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1673, in get_hf_file_metadata
    r = _request_wrapper(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 376, in _request_wrapper
    response = _request_wrapper(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 399, in _request_wrapper
    response = get_session().request(method=method, url=url, **params)
  File "/root/.venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/.venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 66, in send
    return super().send(request, *args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), '(Request ID: d24df324-e319-4864-86de-a016f945a3d0)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.venv/bin/huggingface-cli", line 8, in <module>
    sys.exit(main())
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/huggingface_cli.py", line 52, in main
    service.run()
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/download.py", line 146, in run
    print(self._download())  # Print path to downloaded files
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/download.py", line 180, in _download
    return snapshot_download(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 299, in snapshot_download
    thread_map(
  File "/root/.venv/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
  File "/root/.venv/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
  File "/root/.venv/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 273, in _inner_hf_hub_download
    return hf_hub_download(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
    return f(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1220, in hf_hub_download
    return _hf_hub_download_to_local_dir(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1468, in _hf_hub_download_to_local_dir
    _raise_on_head_call_error(head_call_error, force_download, local_files_only)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1857, in _raise_on_head_call_error
    raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

Client env config

(.venv) root@node2:~# huggingface-cli env

Copy-and-paste the text below in your GitHub issue.

- huggingface_hub version: 0.24.5
- Platform: Linux-6.5.0-45-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /root/.cache/huggingface/token
- Has saved token ?: True
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: N/A
- Jinja2: N/A
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: N/A
- hf_transfer: 0.1.8
- gradio: N/A
- tensorboard: N/A
- numpy: N/A
- pydantic: N/A
- aiohttp: N/A
- ENDPOINT: http://172.16.50.10:8090
- HF_HUB_CACHE: /root/.cache/huggingface/hub
- HF_ASSETS_CACHE: /root/.cache/huggingface/assets
- HF_TOKEN_PATH: /root/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

Olah logs

Aug 07 07:39:55 node1 systemd[1]: Started Olah Server.
Aug 07 07:39:57 node1 python[1454591]: 2024-08-07 07:39:57 | INFO | stdout | Namespace(config='', host='localhost', port=8090, ssl_key=None, ssl_cert=None, repos_path='/data/olah/hf_mirrors', log_path='./logs')
Aug 07 07:39:57 node1 python[1454591]: 2024-08-07 07:39:57 | ERROR | stderr | INFO:     Started server process [1454591]
Aug 07 07:39:57 node1 python[1454591]: 2024-08-07 07:39:57 | ERROR | stderr | INFO:     Waiting for application startup.
Aug 07 07:39:57 node1 python[1454591]: 2024-08-07 07:39:57 | ERROR | stderr | INFO:     Application startup complete.
Aug 07 07:39:57 node1 python[1454591]: 2024-08-07 07:39:57 | ERROR | stderr | INFO:     Uvicorn running on http://localhost:8090 (Press CTRL+C to quit)
Aug 07 07:39:57 node1 python[1454591]: 2024-08-07 07:39:57 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/datasets/Salesforce/wikitext/resolve/main/.gitattributes "HTTP/1.1 200 OK"
Aug 07 07:39:59 node1 python[1454591]: 2024-08-07 07:39:59 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 07 07:39:59 node1 python[1454591]: 2024-08-07 07:39:59 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 07 07:40:00 node1 python[1454591]: 2024-08-07 07:40:00 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 07 07:40:00 node1 python[1454591]: 2024-08-07 07:40:00 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main HTTP/1.0" 200 OK
Aug 07 07:40:00 node1 python[1454591]: 2024-08-07 07:40:00 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:00 node1 python[1454591]: 2024-08-07 07:40:00 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:00 node1 python[1454591]: 2024-08-07 07:40:00 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:00 node1 python[1454591]: 2024-08-07 07:40:00 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:00 node1 python[1454591]: 2024-08-07 07:40:00 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:00 node1 python[1454591]: 2024-08-07 07:40:00 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00001-of-00109.safetensors HTTP/1.0" 302 Found
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/generation_config.json HTTP/1.0" 200 OK
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/.gitattributes HTTP/1.0" 200 OK
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/config.json HTTP/1.0" 200 OK
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | httpx | HTTP Request: HEAD http://172.16.50.10/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00109.safetensors%3B+filename%3D%22model-00001-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjdkNjc2MGUwMGYzYjhmY2Y2YWI3ODI0YTJkZTUwZjA3ZjU5Y2MwOGIwM2QyNjNiNmMwZDU2MmRhZjg2MjJiZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=fr9up4qmn0-~hwfDOHbqsKbRUKmaY~arE~9CHFuq5soQPLXTV4cwysE2REWplGAadJZCJxlDWUxM6ira3HjnkFt4hqtzSG2Sb3nDJOQ-8lRLD7T2XUIgLtGiwn-zcIaXOOerHg9~rI8R~Q7QpGX078lpTpml0-chIzB8FrHmmMAfVCvnDsGIsT2FYV-hqR2m-YWIlAty7a-WlBykkDpkaQNu3iz-VYztzJviTVQo4IDRd~6P6euGLW5iyTEzQsi1-gYEm6dBfyrazhvHqEzLr56Tm0Y43AewnhYoWC5a~onjcAy502CdYKUcDDCzSjlsaMZ7baHLNu7YOdPwPYCuXA__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 404 Not Found"
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00109.safetensors%3B+filename%3D%22model-00001-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjdkNjc2MGUwMGYzYjhmY2Y2YWI3ODI0YTJkZTUwZjA3ZjU5Y2MwOGIwM2QyNjNiNmMwZDU2MmRhZjg2MjJiZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=fr9up4qmn0-~hwfDOHbqsKbRUKmaY~arE~9CHFuq5soQPLXTV4cwysE2REWplGAadJZCJxlDWUxM6ira3HjnkFt4hqtzSG2Sb3nDJOQ-8lRLD7T2XUIgLtGiwn-zcIaXOOerHg9~rI8R~Q7QpGX078lpTpml0-chIzB8FrHmmMAfVCvnDsGIsT2FYV-hqR2m-YWIlAty7a-WlBykkDpkaQNu3iz-VYztzJviTVQo4IDRd~6P6euGLW5iyTEzQsi1-gYEm6dBfyrazhvHqEzLr56Tm0Y43AewnhYoWC5a~onjcAy502CdYKUcDDCzSjlsaMZ7baHLNu7YOdPwPYCuXA__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.0" 500 Internal Server Error
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr | Traceback (most recent call last):
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     result = await app(  # type: ignore[func-returns-value]
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     return await self.app(scope, receive, send)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     await super().__call__(scope, receive, send)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     raise exc
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     await self.app(scope, receive, _send)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     raise exc
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     await app(scope, receive, sender)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     await route.handle(scope, receive, send)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     await self.app(scope, receive, send)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     raise exc
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     await app(scope, receive, sender)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     response = await func(request)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     raw_response = await run_endpoint_function(
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     return await dependant.call(**values)
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/olah/olah/server.py", line 521, in lfs_get
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     status_code = await generator.__anext__()
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/olah/olah/proxy/files.py", line 314, in _file_realtime_stream
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     status_code, head_info, content = await _file_full_header(
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |   File "/data/olah/olah/olah/proxy/files.py", line 133, in _file_full_header
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr |     raise Exception(f"Unexpected HTTP status code {response.status_code}")
Aug 07 07:40:01 node1 python[1454591]: 2024-08-07 07:40:01 | ERROR | stderr | Exception: Unexpected HTTP status code 404
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00042-of-00109.safetensors "HTTP/1.1 302 Found"
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00081-of-00109.safetensors "HTTP/1.1 302 Found"
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00042-of-00109.safetensors HTTP/1.0" 302 Found
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00081-of-00109.safetensors HTTP/1.0" 302 Found
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | httpx | HTTP Request: HEAD http://172.16.50.10/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/e321c3213f419390a815acdf33548d68dc9322379f93596b60b3359eb82b0fc2?response-content-disposition=inline%3B+filename%2A%3DUTF-8%27%27model-00081-of-00109.safetensors%3B+filename%3D%22model-00081-of-00109.safetensors%22%3B&Expires=1723275602&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzI3NTYwMn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZTMyMWMzMjEzZjQxOTM5MGE4MTVhY2RmMzM1NDhkNjhkYzkzMjIzNzlmOTM1OTZiNjBiMzM1OWViODJiMGZjMj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=RMuV-G39XhjJCjztz~KbieXeEsbO07a8vNb5~4ZCIFl4NzaqC5IfO2lYR5j4a5xuDP~aQWv7DaEqQNe9QatinAkdiFEmWaf7yXjiV9Pq7X3orr8U6vYdgkj1sY0n7GaEDkCrEZ-VeiSAdjkCgQico~6BJTvtOxzpJ~qG~dc2QCklHFPw6imtGWGhKWal7C5l8V79xZM1mK6k9E9z4dSwa0rmfwSnVnlNcQGBx-aWAOGzih164kwrKg5jgfVFB~i3jL0p3ORBOvmeCFuC2SZzN3vYHGsrDDM6DviQHYEiUqGHY2ZiDM5kqBQ0y09OJneRCYdKTapOdGabIa7jo~T-iA__&Key-Pair-Id=K24J24Z295AEI9&oriloc=https%3A%2F%2Fcdn-lfs-us-1.huggingface.co%2Frepos%2Fc6%2F4c%2Fc64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591%2Fe321c3213f419390a815acdf33548d68dc9322379f93596b60b3359eb82b0fc2%3Fresponse-content-disposition%3Dinline%253B%2Bfilename%2A%253DUTF-8%2527%2527model-00081-of-00109.safetensors%253B%2Bfilename%253D%2522model-00081-of-00109.safetensors%2522%253B%26Expires%3D1723275602%26Policy%3DeyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzI3NTYwMn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZTMyMWMzMjEzZjQxOTM5MGE4MTVhY2RmMzM1NDhkNjhkYzkzMjIzNzlmOTM1OTZiNjBiMzM1OWViODJiMGZjMj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19%26Signature%3DRMuV-G39XhjJCjztz%257EKbieXeEsbO07a8vNb5%257E4ZCIFl4NzaqC5IfO2lYR5j4a5xuDP%257EaQWv7DaEqQNe9QatinAkdiFEmWaf7yXjiV9Pq7X3orr8U6vYdgkj1sY0n7GaEDkCrEZ-VeiSAdjkCgQico%257E6BJTvtOxzpJ%257EqG%257Edc2QCklHFPw6imtGWGhKWal7C5l8V79xZM1mK6k9E9z4dSwa0rmfwSnVnlNcQGBx-aWAOGzih164kwrKg5jgfVFB%257Ei3jL0p3ORBOvmeCFuC2SZzN3vYHGsrDDM6DviQHYEiUqGHY2ZiDM5kqBQ0y09OJneRCYdKTapOdGabIa7jo%257ET-iA__%26Key-Pair-Id%3DK24J24Z295AEI9 "HTTP/1.1 404 Not Found"
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | httpx | HTTP Request: HEAD http://172.16.50.10/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/211f7316d4a389f2676d4fd68787d081ef679d41786ddd9d4464c97e4a1a7bda?response-content-disposition=inline%3B+filename%2A%3DUTF-8%27%27model-00042-of-00109.safetensors%3B+filename%3D%22model-00042-of-00109.safetensors%22%3B&Expires=1723275602&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzI3NTYwMn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjExZjczMTZkNGEzODlmMjY3NmQ0ZmQ2ODc4N2QwODFlZjY3OWQ0MTc4NmRkZDlkNDQ2NGM5N2U0YTFhN2JkYT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=FkIw99Fb-e-uu2sefvl3Qvp2aPtcicEQIm27RtBm3sMINERXtKlsI~yWQBo6JBo21bMiH~pMwEDjfqcyPBvMdO7PhAK3rRvlO-GN5G9nMdcOYE0JfQxxk5ht7nDlzCz0SJqhUG-ppZQqfFc8epcD1AytUH8bOszdYJC3UvX6UpfIe7FVNZG48ayJzHNyXNPQem9016BYuwQjA3f0I-HepjPZB1gHOsg1M-Y8wpRViP-3yzLoziUDTIqMjBvK0J3ISvSLhdN-8xLk5AJh9lTL6LsswNk7cet1wC5Pe9p9gpey1unFSTa47~Pglh-h2e1JSIEbE6sEvQgjfHA7S7s~Bg__&Key-Pair-Id=K24J24Z295AEI9&oriloc=https%3A%2F%2Fcdn-lfs-us-1.huggingface.co%2Frepos%2Fc6%2F4c%2Fc64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591%2F211f7316d4a389f2676d4fd68787d081ef679d41786ddd9d4464c97e4a1a7bda%3Fresponse-content-disposition%3Dinline%253B%2Bfilename%2A%253DUTF-8%2527%2527model-00042-of-00109.safetensors%253B%2Bfilename%253D%2522model-00042-of-00109.safetensors%2522%253B%26Expires%3D1723275602%26Policy%3DeyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzI3NTYwMn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjExZjczMTZkNGEzODlmMjY3NmQ0ZmQ2ODc4N2QwODFlZjY3OWQ0MTc4NmRkZDlkNDQ2NGM5N2U0YTFhN2JkYT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19%26Signature%3DFkIw99Fb-e-uu2sefvl3Qvp2aPtcicEQIm27RtBm3sMINERXtKlsI%257EyWQBo6JBo21bMiH%257EpMwEDjfqcyPBvMdO7PhAK3rRvlO-GN5G9nMdcOYE0JfQxxk5ht7nDlzCz0SJqhUG-ppZQqfFc8epcD1AytUH8bOszdYJC3UvX6UpfIe7FVNZG48ayJzHNyXNPQem9016BYuwQjA3f0I-HepjPZB1gHOsg1M-Y8wpRViP-3yzLoziUDTIqMjBvK0J3ISvSLhdN-8xLk5AJh9lTL6LsswNk7cet1wC5Pe9p9gpey1unFSTa47%257EPglh-h2e1JSIEbE6sEvQgjfHA7S7s%257EBg__%26Key-Pair-Id%3DK24J24Z295AEI9 "HTTP/1.1 404 Not Found"
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/e321c3213f419390a815acdf33548d68dc9322379f93596b60b3359eb82b0fc2?response-content-disposition=inline%3B+filename%2A%3DUTF-8%27%27model-00081-of-00109.safetensors%3B+filename%3D%22model-00081-of-00109.safetensors%22%3B&Expires=1723275602&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzI3NTYwMn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZTMyMWMzMjEzZjQxOTM5MGE4MTVhY2RmMzM1NDhkNjhkYzkzMjIzNzlmOTM1OTZiNjBiMzM1OWViODJiMGZjMj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=RMuV-G39XhjJCjztz~KbieXeEsbO07a8vNb5~4ZCIFl4NzaqC5IfO2lYR5j4a5xuDP~aQWv7DaEqQNe9QatinAkdiFEmWaf7yXjiV9Pq7X3orr8U6vYdgkj1sY0n7GaEDkCrEZ-VeiSAdjkCgQico~6BJTvtOxzpJ~qG~dc2QCklHFPw6imtGWGhKWal7C5l8V79xZM1mK6k9E9z4dSwa0rmfwSnVnlNcQGBx-aWAOGzih164kwrKg5jgfVFB~i3jL0p3ORBOvmeCFuC2SZzN3vYHGsrDDM6DviQHYEiUqGHY2ZiDM5kqBQ0y09OJneRCYdKTapOdGabIa7jo~T-iA__&Key-Pair-Id=K24J24Z295AEI9&oriloc=https%3A%2F%2Fcdn-lfs-us-1.huggingface.co%2Frepos%2Fc6%2F4c%2Fc64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591%2Fe321c3213f419390a815acdf33548d68dc9322379f93596b60b3359eb82b0fc2%3Fresponse-content-disposition%3Dinline%253B%2Bfilename%2A%253DUTF-8%2527%2527model-00081-of-00109.safetensors%253B%2Bfilename%253D%2522model-00081-of-00109.safetensors%2522%253B%26Expires%3D1723275602%26Policy%3DeyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzI3NTYwMn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZTMyMWMzMjEzZjQxOTM5MGE4MTVhY2RmMzM1NDhkNjhkYzkzMjIzNzlmOTM1OTZiNjBiMzM1OWViODJiMGZjMj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19%26Signature%3DRMuV-G39XhjJCjztz%257EKbieXeEsbO07a8vNb5%257E4ZCIFl4NzaqC5IfO2lYR5j4a5xuDP%257EaQWv7DaEqQNe9QatinAkdiFEmWaf7yXjiV9Pq7X3orr8U6vYdgkj1sY0n7GaEDkCrEZ-VeiSAdjkCgQico%257E6BJTvtOxzpJ%257EqG%257Edc2QCklHFPw6imtGWGhKWal7C5l8V79xZM1mK6k9E9z4dSwa0rmfwSnVnlNcQGBx-aWAOGzih164kwrKg5jgfVFB%257Ei3jL0p3ORBOvmeCFuC2SZzN3vYHGsrDDM6DviQHYEiUqGHY2ZiDM5kqBQ0y09OJneRCYdKTapOdGabIa7jo%257ET-iA__%26Key-Pair-Id%3DK24J24Z295AEI9 HTTP/1.0" 500 Internal Server Error
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr | Traceback (most recent call last):
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     result = await app(  # type: ignore[func-returns-value]
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     return await self.app(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await super().__call__(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     raise exc
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await self.app(scope, receive, _send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     raise exc
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await app(scope, receive, sender)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await route.handle(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await self.app(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     raise exc
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await app(scope, receive, sender)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     response = await func(request)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     raw_response = await run_endpoint_function(
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     return await dependant.call(**values)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/olah/olah/server.py", line 521, in lfs_get
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     status_code = await generator.__anext__()
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/olah/olah/proxy/files.py", line 314, in _file_realtime_stream
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     status_code, head_info, content = await _file_full_header(
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/olah/olah/proxy/files.py", line 133, in _file_full_header
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     raise Exception(f"Unexpected HTTP status code {response.status_code}")
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr | Exception: Unexpected HTTP status code 404
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/211f7316d4a389f2676d4fd68787d081ef679d41786ddd9d4464c97e4a1a7bda?response-content-disposition=inline%3B+filename%2A%3DUTF-8%27%27model-00042-of-00109.safetensors%3B+filename%3D%22model-00042-of-00109.safetensors%22%3B&Expires=1723275602&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzI3NTYwMn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjExZjczMTZkNGEzODlmMjY3NmQ0ZmQ2ODc4N2QwODFlZjY3OWQ0MTc4NmRkZDlkNDQ2NGM5N2U0YTFhN2JkYT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=FkIw99Fb-e-uu2sefvl3Qvp2aPtcicEQIm27RtBm3sMINERXtKlsI~yWQBo6JBo21bMiH~pMwEDjfqcyPBvMdO7PhAK3rRvlO-GN5G9nMdcOYE0JfQxxk5ht7nDlzCz0SJqhUG-ppZQqfFc8epcD1AytUH8bOszdYJC3UvX6UpfIe7FVNZG48ayJzHNyXNPQem9016BYuwQjA3f0I-HepjPZB1gHOsg1M-Y8wpRViP-3yzLoziUDTIqMjBvK0J3ISvSLhdN-8xLk5AJh9lTL6LsswNk7cet1wC5Pe9p9gpey1unFSTa47~Pglh-h2e1JSIEbE6sEvQgjfHA7S7s~Bg__&Key-Pair-Id=K24J24Z295AEI9&oriloc=https%3A%2F%2Fcdn-lfs-us-1.huggingface.co%2Frepos%2Fc6%2F4c%2Fc64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591%2F211f7316d4a389f2676d4fd68787d081ef679d41786ddd9d4464c97e4a1a7bda%3Fresponse-content-disposition%3Dinline%253B%2Bfilename%2A%253DUTF-8%2527%2527model-00042-of-00109.safetensors%253B%2Bfilename%253D%2522model-00042-of-00109.safetensors%2522%253B%26Expires%3D1723275602%26Policy%3DeyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzI3NTYwMn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjExZjczMTZkNGEzODlmMjY3NmQ0ZmQ2ODc4N2QwODFlZjY3OWQ0MTc4NmRkZDlkNDQ2NGM5N2U0YTFhN2JkYT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19%26Signature%3DFkIw99Fb-e-uu2sefvl3Qvp2aPtcicEQIm27RtBm3sMINERXtKlsI%257EyWQBo6JBo21bMiH%257EpMwEDjfqcyPBvMdO7PhAK3rRvlO-GN5G9nMdcOYE0JfQxxk5ht7nDlzCz0SJqhUG-ppZQqfFc8epcD1AytUH8bOszdYJC3UvX6UpfIe7FVNZG48ayJzHNyXNPQem9016BYuwQjA3f0I-HepjPZB1gHOsg1M-Y8wpRViP-3yzLoziUDTIqMjBvK0J3ISvSLhdN-8xLk5AJh9lTL6LsswNk7cet1wC5Pe9p9gpey1unFSTa47%257EPglh-h2e1JSIEbE6sEvQgjfHA7S7s%257EBg__%26Key-Pair-Id%3DK24J24Z295AEI9 HTTP/1.0" 500 Internal Server Error
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr | Traceback (most recent call last):
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     result = await app(  # type: ignore[func-returns-value]
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     return await self.app(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await super().__call__(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     raise exc
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await self.app(scope, receive, _send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     raise exc
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await app(scope, receive, sender)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await route.handle(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await self.app(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     raise exc
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     await app(scope, receive, sender)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     response = await func(request)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     raw_response = await run_endpoint_function(
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     return await dependant.call(**values)
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/olah/olah/server.py", line 521, in lfs_get
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     status_code = await generator.__anext__()
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/olah/olah/proxy/files.py", line 314, in _file_realtime_stream
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     status_code, head_info, content = await _file_full_header(
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |   File "/data/olah/olah/olah/proxy/files.py", line 133, in _file_full_header
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr |     raise Exception(f"Unexpected HTTP status code {response.status_code}")
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | ERROR | stderr | Exception: Unexpected HTTP status code 404
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/.gitattributes HTTP/1.0" 200 OK
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 07 07:40:02 node1 python[1454591]: 2024-08-07 07:40:02 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/generation_config.json HTTP/1.0" 200 OK
Aug 07 07:40:03 node1 python[1454591]: 2024-08-07 07:40:03 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/config.json HTTP/1.0" 200 OK

Olah env

root@node1:~# su - olah
olah@node1:~$ source ~/.venv/bin/activate
(.venv) olah@node1:~$ pip freeze
annotated-types==0.7.0
anyio==4.4.0
cachetools==5.4.0
certifi==2024.7.4
charset-normalizer==3.3.2
click==8.1.7
dnspython==2.6.1
email_validator==2.2.0
exceptiongroup==1.2.2
fastapi==0.112.0
fastapi-cli==0.0.5
fastapi-utils==0.7.0
filelock==3.15.4
fsspec==2024.6.1
gitdb==4.0.11
GitPython==3.1.43
h11==0.14.0
httpcore==1.0.5
httptools==0.6.1
httpx==0.27.0
huggingface-hub==0.24.5
idna==3.7
iniconfig==2.0.0
Jinja2==3.1.4
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mdurl==0.1.2
mpmath==1.3.0
mypy-extensions==1.0.0
networkx==3.3
numpy==2.0.1
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.6.20
nvidia-nvtx-cu12==12.1.105
-e git+https://github.com/vtuber-plan/olah.git@6c8be801fbefa232c29f8d55df473942cb13612f#egg=olah
orjson==3.10.6
packaging==24.1
pluggy==1.5.0
psutil==5.9.8
pydantic==2.8.2
pydantic-settings==2.4.0
pydantic_core==2.20.1
Pygments==2.18.0
pytest==8.2.2
python-dotenv==1.0.1
python-multipart==0.0.9
pytz==2024.1
PyYAML==6.0.1
requests==2.32.3
rich==13.7.1
shellingham==1.5.4
shortuuid==1.0.13
smmap==5.0.1
sniffio==1.3.1
starlette==0.37.2
sympy==1.13.1
tenacity==9.0.0
toml==0.10.2
tomli==2.0.1
torch==2.4.0
tqdm==4.66.5
triton==3.0.0
typer==0.12.3
typing-inspect==0.9.0
typing_extensions==4.12.2
ujson==5.10.0
urllib3==2.2.2
uvicorn==0.30.5
uvloop==0.19.0
watchfiles==0.22.0
websockets==12.0
(.venv) olah@node1:~$

@jstzwj
Copy link
Contributor

jstzwj commented Aug 8, 2024

I noticed a difference between the logs above and mine: my request was sent to 127.0.0.1, while yours was sent to 172.16.50.12. This difference might have triggered a bug at line 294 in olah/proxy/files.py.
The script detects the URL as mirror_netloc (i.e., 127.0.0.1) and replaces it with Hugging Face's LFS CDN. However, it does not perform this replacement for 172.16.50.12, hence the 404 error when accessing /repos/c6/4c/c64c854f39008.
I fixed this and pushed the changes to the dev branch.

@andy108369
Copy link
Author

andy108369 commented Aug 8, 2024 •
edited
Loading

I noticed a difference between the logs above and mine: my request was sent to 127.0.0.1, while yours was sent to 172.16.50.12. This difference might have triggered a bug at line 294 in olah/proxy/files.py. The script detects the URL as mirror_netloc (i.e., 127.0.0.1) and replaces it with Hugging Face's LFS CDN. However, it does not perform this replacement for 172.16.50.12, hence the 404 error when accessing /repos/c6/4c/c64c854f39008. I fixed this and pushed the changes to the dev branch.

Thanks @jstzwj !
It appears that the issue is now a bit different:

(.venv) olah@node1:~/olah$ git log -1
commit 70d8aecd5c6673246380a89d94f31d4fed1956be (HEAD -> dev, origin/dev)
Author: jstzwj <1103870790@qq.com>
Date:   Thu Aug 8 13:18:48 2024 +0800

    query param bug fix

Client

root@node2:~# source ~/.venv/bin/activate
(.venv) root@node2:~# 
(.venv) root@node2:~# export HF_ENDPOINT=http://172.16.50.10:8090
(.venv) root@node2:~# read -s HF_TOKEN
(.venv) root@node2:~# export HF_TOKEN
(.venv) root@node2:~# echo $HF_

(.venv) root@node2:~# echo $HF_TOKEN 
REDACTED (token is valid though)

(.venv) root@node2:~# python -c "from huggingface_hub import HfApi; print(HfApi().endpoint)"
http://172.16.50.10:8090
(.venv) root@node2:~# huggingface-cli download --resume-download meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --local-dir /data/test-llama31-405b-fp8
/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Fetching 242 files:   0%|                                                                                                                                                                                                                                               | 0/242 [00:00<?, ?it/s]Downloading 'model-00002-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00002-of-00109.safetensors.efef014770d588f4206f2da94ac251db8958f6a8ce1e38762d28c044bb015198.incomplete'
Downloading 'model-00005-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00005-of-00109.safetensors.fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931.incomplete'
Downloading 'model-00007-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00007-of-00109.safetensors.1fcdeb3bc4556d7da5bdf242919dc50df950563d8191d3be518fb001ab3e6331.incomplete'
Downloading 'model-00008-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00008-of-00109.safetensors.410058afd38e1cdbca5a1a9cdae4e25f6d006763c3b224b6101611049458e109.incomplete'
Downloading 'model-00001-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00001-of-00109.safetensors.27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf.incomplete'
Downloading 'model-00006-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00006-of-00109.safetensors.877173ef17665d02a697423e2b5c1e0fd8b3f50a8ba5c09b413896027c487789.incomplete'
Downloading 'model-00004-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00004-of-00109.safetensors.376536d7284c6a79a65ee8e3fb3ee3fdf1a13040f72ac31ca7d2e4b13e7d52a3.incomplete'
Downloading 'model-00003-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00003-of-00109.safetensors.26bb5d1cd5b224dee88f442b1a6acdf196a5d647d5af6834c62aa6a2a01bdd05.incomplete'
Fetching 242 files:   2%|█████▋                                                                                                                                                                                                                                 | 6/242 [00:02<01:20,  2.93it/s]
Downloading 'model-00011-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00011-of-00109.safetensors.b235950238fed5a7260863c9f38efeea90d919467156b9d80d1301429b2c82f3.incomplete'
Downloading 'model-00014-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00014-of-00109.safetensors.5136b94feb1cfdf45bab99b07c555f1a68e2e559d17713f9eaccf6e132be2ddb.incomplete'
Downloading 'model-00012-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00012-of-00109.safetensors.05e418379ebffbabdd2497a4a6c7f7a0d466f606b69a4b300437e34b66974253.incomplete'
Downloading 'model-00009-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00009-of-00109.safetensors.c41de3c512d78c8de6266adffa0db49850ee4692a6d74c2084f6519c450abdd6.incomplete'
Downloading 'model-00010-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00010-of-00109.safetensors.f919b1ef9202fe7b15871fc3f18610d3f799f8bbdd7b4bbfddc3ce7dc652fd21.incomplete'
Downloading 'model-00013-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00013-of-00109.safetensors.f58ae5a0e97541fa9612723d673c63d3afe7574da0e68291fc3596307dcd2a3d.incomplete'
Traceback (most recent call last):
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://172.16.50.10:8090/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00109.safetensors%3B+filename%3D%22model-00001-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjdkNjc2MGUwMGYzYjhmY2Y2YWI3ODI0YTJkZTUwZjA3ZjU5Y2MwOGIwM2QyNjNiNmMwZDU2MmRhZjg2MjJiZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=fr9up4qmn0-~hwfDOHbqsKbRUKmaY~arE~9CHFuq5soQPLXTV4cwysE2REWplGAadJZCJxlDWUxM6ira3HjnkFt4hqtzSG2Sb3nDJOQ-8lRLD7T2XUIgLtGiwn-zcIaXOOerHg9~rI8R~Q7QpGX078lpTpml0-chIzB8FrHmmMAfVCvnDsGIsT2FYV-hqR2m-YWIlAty7a-WlBykkDpkaQNu3iz-VYztzJviTVQo4IDRd~6P6euGLW5iyTEzQsi1-gYEm6dBfyrazhvHqEzLr56Tm0Y43AewnhYoWC5a~onjcAy502CdYKUcDDCzSjlsaMZ7baHLNu7YOdPwPYCuXA__&Key-Pair-Id=K24J24Z295AEI9

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 748, in _error_catcher
    yield
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 894, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(0 bytes read, 106 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 820, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 1060, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 949, in read
    data = self._raw_read(amt)
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 872, in _raw_read
    with self._error_catcher():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 772, in _error_catcher
    raise ProtocolError(arg, e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read, 106 more expected)', IncompleteRead(0 bytes read, 106 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.venv/bin/huggingface-cli", line 8, in <module>
    sys.exit(main())
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/huggingface_cli.py", line 52, in main
    service.run()
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/download.py", line 146, in run
    print(self._download())  # Print path to downloaded files
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/download.py", line 180, in _download
    return snapshot_download(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 299, in snapshot_download
    thread_map(
  File "/root/.venv/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
  File "/root/.venv/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
  File "/root/.venv/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 273, in _inner_hf_hub_download
    return hf_hub_download(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
    return f(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1220, in hf_hub_download
    return _hf_hub_download_to_local_dir(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1515, in _hf_hub_download_to_local_dir
    _download_to_tmp_and_move(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1915, in _download_to_tmp_and_move
    http_get(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 463, in http_get
    r = _request_wrapper(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 400, in _request_wrapper
    hf_raise_for_status(response)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 367, in hf_raise_for_status
    raise HfHubHTTPError(message, response=response) from e
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 70, in __init__
    server_data = response.json()
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 974, in json
    return complexjson.loads(self.text, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 926, in text
    if not self.content:
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 902, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 822, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read, 106 more expected)', IncompleteRead(0 bytes read, 106 more expected))
(.venv) root@node2:~# echo $?
1
(.venv) root@node2:~#

nginx server

nginx server config is here

root@node1:~# tail -f /var/log/nginx/*.log
...
...

==> /var/log/nginx/access.log <==
172.16.50.12 - - [08/Aug/2024:11:12:05 +0000] "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/f919b1ef9202fe7b15871fc3f18610d3f799f8bbdd7b4bbfddc3ce7dc652fd21?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00010-of-00109.safetensors%3B+filename%3D%22model-00010-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZjkxOWIxZWY5MjAyZmU3YjE1ODcxZmMzZjE4NjEwZDNmNzk5ZjhiYmRkN2I0YmJmZGRjM2NlN2RjNjUyZmQyMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=LdVchzwlU5W~SJzOtsnzoMFSAzBLOxtIUTvXRRS4hOVuBY0fqce-yjvrgPXBb3kcnn1tlr5J4ngjxMugq~YScTOmtmKhRWrk2nF1ZH1Z4F93ztuTCsa7A82XE~YMT7i5CCrlqWTq4Cv6SO6dsx7GIJMlNNdb9kYkSMxthZ8XRkUZz-KSN2uyxeKPt1uglL24ziqpuJI-DtD9fsdy9rL3OQm5G0BIMwWnz-YYEOS10ugjGIepOvZ-0WV2LifiUJoRBMv5~ar0Y2qX5OeoMQXz-XSiQG6EdanG5iz9S6xC4TXNM0RLs8AYHI8p-Z0knR5nPPizwZf-tLUKP9vWML-oFw__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 0 "-" "huggingface-cli/None; hf_hub/0.24.5; python/3.10.12"

==> /var/log/nginx/error.log <==
2024/08/08 11:12:05 [error] 1844019#1844019: *27 upstream prematurely closed connection while reading upstream, client: 172.16.50.12, server: , request: "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/c41de3c512d78c8de6266adffa0db49850ee4692a6d74c2084f6519c450abdd6?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00009-of-00109.safetensors%3B+filename%3D%22model-00009-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvYzQxZGUzYzUxMmQ3OGM4ZGU2MjY2YWRmZmEwZGI0OTg1MGVlNDY5MmE2ZDc0YzIwODRmNjUxOWM0NTBhYmRkNj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=oEWlKqZWagDqtpwoNzOlUJvl3TXKrlf1dlyJAm0tXWRGV7iY2UqbkneuiGK5RhCUqOuuitSpdPiYkjhl1duFOhxqcT7UmNr2JfithcKueq7QC4MUgvuKahx3A5iTSq~Pk1Clfj~cWA1Ev-69ziVRJv9AdU7IfwkvDIgacYYA0fxTWg2cOviHMkdkDllJWLzf4tM-LRvzOZeVXQv3W3Ak-Vh~2Yqf-sU6y88mTEinNW1SJW1LFt~tLNOi4vyynHYzd-F~WsuHM~NNqPBjtHRr6rnyGyJXtGtN7P7BAWfz6mhzWg-3STVV6~T6vf44dCa1vB9QVBe7Ot7gLrcntnTFVA__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1", upstream: "http://127.0.0.1:8090/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/c41de3c512d78c8de6266adffa0db49850ee4692a6d74c2084f6519c450abdd6?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00009-of-00109.safetensors%3B+filename%3D%22model-00009-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvYzQxZGUzYzUxMmQ3OGM4ZGU2MjY2YWRmZmEwZGI0OTg1MGVlNDY5MmE2ZDc0YzIwODRmNjUxOWM0NTBhYmRkNj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=oEWlKqZWagDqtpwoNzOl

==> /var/log/nginx/access.log <==
172.16.50.12 - - [08/Aug/2024:11:12:05 +0000] "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/c41de3c512d78c8de6266adffa0db49850ee4692a6d74c2084f6519c450abdd6?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00009-of-00109.safetensors%3B+filename%3D%22model-00009-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvYzQxZGUzYzUxMmQ3OGM4ZGU2MjY2YWRmZmEwZGI0OTg1MGVlNDY5MmE2ZDc0YzIwODRmNjUxOWM0NTBhYmRkNj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=oEWlKqZWagDqtpwoNzOlUJvl3TXKrlf1dlyJAm0tXWRGV7iY2UqbkneuiGK5RhCUqOuuitSpdPiYkjhl1duFOhxqcT7UmNr2JfithcKueq7QC4MUgvuKahx3A5iTSq~Pk1Clfj~cWA1Ev-69ziVRJv9AdU7IfwkvDIgacYYA0fxTWg2cOviHMkdkDllJWLzf4tM-LRvzOZeVXQv3W3Ak-Vh~2Yqf-sU6y88mTEinNW1SJW1LFt~tLNOi4vyynHYzd-F~WsuHM~NNqPBjtHRr6rnyGyJXtGtN7P7BAWfz6mhzWg-3STVV6~T6vf44dCa1vB9QVBe7Ot7gLrcntnTFVA__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 0 "-" "huggingface-cli/None; hf_hub/0.24.5; python/3.10.12"

==> /var/log/nginx/error.log <==
2024/08/08 11:12:05 [error] 1844019#1844019: *33 upstream prematurely closed connection while reading upstream, client: 172.16.50.12, server: , request: "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/f58ae5a0e97541fa9612723d673c63d3afe7574da0e68291fc3596307dcd2a3d?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00013-of-00109.safetensors%3B+filename%3D%22model-00013-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZjU4YWU1YTBlOTc1NDFmYTk2MTI3MjNkNjczYzYzZDNhZmU3NTc0ZGEwZTY4MjkxZmMzNTk2MzA3ZGNkMmEzZD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=ovu94T1hk5LO68IUMBJl52a87oqdSKKv7tdxZbw0DjuNBjWe8V1kGj2qsi9WBPiOHB7iUZgmpsrT5k1uIzgZhkDu0M~Kg2dWpj9m3M8liHUCKrkqsA~9jkPxFjJYxI9XDxMOdNgQe2d0rmlRKkM940OZSQz1Rij0o6WFQHIUlC2uLaFXu7r1tIDEaDoFJtxzuRih54TDE0FhGZJ8v-VhYjCQCleaQlZ9NpPoYJyuXTESP9NblWgphL-1lNuiu3jW7Sk7tJlnE0Ly6x5zGofOIHMzjaw6CmMy2WeoumWKAmZhELHFzZQOWMZ37~L17Ng5ghOhvpLeSdeqAsy6MFNE2Q__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1", upstream: "http://127.0.0.1:8090/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/f58ae5a0e97541fa9612723d673c63d3afe7574da0e68291fc3596307dcd2a3d?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00013-of-00109.safetensors%3B+filename%3D%22model-00013-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZjU4YWU1YTBlOTc1NDFmYTk2MTI3MjNkNjczYzYzZDNhZmU3NTc0ZGEwZTY4MjkxZmMzNTk2MzA3ZGNkMmEzZD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=ovu94T1hk5LO68IUMBJl

==> /var/log/nginx/access.log <==
172.16.50.12 - - [08/Aug/2024:11:12:05 +0000] "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/f58ae5a0e97541fa9612723d673c63d3afe7574da0e68291fc3596307dcd2a3d?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00013-of-00109.safetensors%3B+filename%3D%22model-00013-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZjU4YWU1YTBlOTc1NDFmYTk2MTI3MjNkNjczYzYzZDNhZmU3NTc0ZGEwZTY4MjkxZmMzNTk2MzA3ZGNkMmEzZD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=ovu94T1hk5LO68IUMBJl52a87oqdSKKv7tdxZbw0DjuNBjWe8V1kGj2qsi9WBPiOHB7iUZgmpsrT5k1uIzgZhkDu0M~Kg2dWpj9m3M8liHUCKrkqsA~9jkPxFjJYxI9XDxMOdNgQe2d0rmlRKkM940OZSQz1Rij0o6WFQHIUlC2uLaFXu7r1tIDEaDoFJtxzuRih54TDE0FhGZJ8v-VhYjCQCleaQlZ9NpPoYJyuXTESP9NblWgphL-1lNuiu3jW7Sk7tJlnE0Ly6x5zGofOIHMzjaw6CmMy2WeoumWKAmZhELHFzZQOWMZ37~L17Ng5ghOhvpLeSdeqAsy6MFNE2Q__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 0 "-" "huggingface-cli/None; hf_hub/0.24.5; python/3.10.12"

olah server

root@node1:~# curl -I http://127.0.0.1:8090/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931
HTTP/1.1 403 Forbidden
date: Thu, 08 Aug 2024 11:12:23 GMT
server: uvicorn
server: CloudFront
date: Thu, 08 Aug 2024 11:12:24 GMT
content-type: text/xml
content-length: 146
connection: keep-alive
x-cache: Error from cloudfront
via: 1.1 7af4698bea7e964f89ad8d7ae22213a2.cloudfront.net (CloudFront)
x-amz-cf-pop: SFO53-P5
x-amz-cf-id: cqCkutCJB0e8Kg1rfEKPUJbUHJJKwirPDmE5jLOFjfFMn8JgJImbmA==
cache-control: public, max-age=604800, immutable, s-maxage=604800
vary: Origin

root@node1:~# export HF_TOKEN=REDACTED
root@node1:~# curl -H "Authorization: Bearer ${HF_TOKEN}" -I http://127.0.0.1:8090/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931
HTTP/1.1 403 Forbidden
date: Thu, 08 Aug 2024 11:13:10 GMT
server: uvicorn
server: CloudFront
date: Thu, 08 Aug 2024 11:13:10 GMT
content-type: text/xml
content-length: 146
connection: keep-alive
x-cache: Error from cloudfront
via: 1.1 c21a0d27ceec21e266c9f962d0349438.cloudfront.net (CloudFront)
x-amz-cf-pop: SEA73-P2
x-amz-cf-id: hGE_XqyVhSWht6ZWTZhdA0mYqs49AiCsv_luXSb3i3nplOH9AHUh3Q==
cache-control: public, max-age=604800, immutable, s-maxage=604800
vary: Origin

root@node1:~# journalctl -u olah-server --since '5 minutes ago' --no-pager -l
...
Aug 08 11:12:04 node1 python[1825074]: 2024-08-08 11:12:04 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:04 node1 python[1825074]: 2024-08-08 11:12:04 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:04 node1 python[1825074]: 2024-08-08 11:12:04 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:04 node1 python[1825074]: 2024-08-08 11:12:04 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:04 node1 python[1825074]: 2024-08-08 11:12:04 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:04 node1 python[1825074]: 2024-08-08 11:12:04 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:04 node1 python[1825074]: 2024-08-08 11:12:04 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:04 node1 python[1825074]: 2024-08-08 11:12:04 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:04 node1 python[1825074]: 2024-08-08 11:12:04 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:04 node1 python[1825074]: 2024-08-08 11:12:04 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00011-of-00109.safetensors HTTP/1.0" 302 Found
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00014-of-00109.safetensors HTTP/1.0" 302 Found
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00012-of-00109.safetensors HTTP/1.0" 302 Found
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00009-of-00109.safetensors HTTP/1.0" 302 Found
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00013-of-00109.safetensors HTTP/1.0" 302 Found
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00010-of-00109.safetensors HTTP/1.0" 302 Found
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/b235950238fed5a7260863c9f38efeea90d919467156b9d80d1301429b2c82f3?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00011-of-00109.safetensors%3B+filename%3D%22model-00011-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvYjIzNTk1MDIzOGZlZDVhNzI2MDg2M2M5ZjM4ZWZlZWE5MGQ5MTk0NjcxNTZiOWQ4MGQxMzAxNDI5YjJjODJmMz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=EqOZeBOe4PK86cFz9x5yQZVMrcQ1B9oifX~jkS3OOLn8chc0lMaLMxyxSJnmSpI0rak9UamUX3Xoo-n1XLREimh~SWq1Au5KZBaCvHLO2TgtqOrYas8fADF7in8SKao1UdWXm7zDDxtqKwnGJyUxU3ps1GsyKCXoikCEmM7KdgqFNaN9L9r3XVGCefOazR38oBfrnV9E0FWfk5sB8CQHEmU3c0Js4ADTeQ5j-pwwC0vFd4aivZVmj0aRYUZZ09BmL~-usdmfdzMp29HFWO8jLPhJ5ksDmqRIfCtM5igZfyW7Lya6G2LMDiQaAJMcwc2mnJc4DfojsVsR~n25VvYruw__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/b235950238fed5a7260863c9f38efeea90d919467156b9d80d1301429b2c82f3?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00011-of-00109.safetensors%3B+filename%3D%22model-00011-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvYjIzNTk1MDIzOGZlZDVhNzI2MDg2M2M5ZjM4ZWZlZWE5MGQ5MTk0NjcxNTZiOWQ4MGQxMzAxNDI5YjJjODJmMz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=EqOZeBOe4PK86cFz9x5yQZVMrcQ1B9oifX~jkS3OOLn8chc0lMaLMxyxSJnmSpI0rak9UamUX3Xoo-n1XLREimh~SWq1Au5KZBaCvHLO2TgtqOrYas8fADF7in8SKao1UdWXm7zDDxtqKwnGJyUxU3ps1GsyKCXoikCEmM7KdgqFNaN9L9r3XVGCefOazR38oBfrnV9E0FWfk5sB8CQHEmU3c0Js4ADTeQ5j-pwwC0vFd4aivZVmj0aRYUZZ09BmL~-usdmfdzMp29HFWO8jLPhJ5ksDmqRIfCtM5igZfyW7Lya6G2LMDiQaAJMcwc2mnJc4DfojsVsR~n25VvYruw__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.0" 403 Forbidden
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     message = await receive()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await fut
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e75926d450
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     +------------------------------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/05e418379ebffbabdd2497a4a6c7f7a0d466f606b69a4b300437e34b66974253?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00012-of-00109.safetensors%3B+filename%3D%22model-00012-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMDVlNDE4Mzc5ZWJmZmJhYmRkMjQ5N2E0YTZjN2Y3YTBkNDY2ZjYwNmI2OWE0YjMwMDQzN2UzNGI2Njk3NDI1Mz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=IjrdXh1F76oFsN1Iua3wOVKyZsDhPMq4bxzpgjT~ILYgoG6AT79Iv5~Pc2YsUIIMhKAcNXjfHsA2-dQYgQFroFuZ~eRQw9P~jifLNS9dmboW7mkRzES~vNtT9q~jWCM0dVnNHugs~A3L4wLho2lO~hJF5mmfmUSrVtzwiQgCpfpkHGvX6l3oliZj~Z5zI443--8dpE6vMNTsMMCNbrCd36vbBt72xHEK8Xot6mj8RN6JpNJrfomoxf2jj-sp4hVjWkLWhyPVo6Vb5iHSq9qWEiOlc8U9Sa272JXYv5n39yjj7jkz0~gZxFr9iXgiSoenLRTO-ZpfsajRbIJFeDTv9A__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/05e418379ebffbabdd2497a4a6c7f7a0d466f606b69a4b300437e34b66974253?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00012-of-00109.safetensors%3B+filename%3D%22model-00012-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMDVlNDE4Mzc5ZWJmZmJhYmRkMjQ5N2E0YTZjN2Y3YTBkNDY2ZjYwNmI2OWE0YjMwMDQzN2UzNGI2Njk3NDI1Mz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=IjrdXh1F76oFsN1Iua3wOVKyZsDhPMq4bxzpgjT~ILYgoG6AT79Iv5~Pc2YsUIIMhKAcNXjfHsA2-dQYgQFroFuZ~eRQw9P~jifLNS9dmboW7mkRzES~vNtT9q~jWCM0dVnNHugs~A3L4wLho2lO~hJF5mmfmUSrVtzwiQgCpfpkHGvX6l3oliZj~Z5zI443--8dpE6vMNTsMMCNbrCd36vbBt72xHEK8Xot6mj8RN6JpNJrfomoxf2jj-sp4hVjWkLWhyPVo6Vb5iHSq9qWEiOlc8U9Sa272JXYv5n39yjj7jkz0~gZxFr9iXgiSoenLRTO-ZpfsajRbIJFeDTv9A__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.0" 403 Forbidden
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/5136b94feb1cfdf45bab99b07c555f1a68e2e559d17713f9eaccf6e132be2ddb?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00014-of-00109.safetensors%3B+filename%3D%22model-00014-of-00109.safetensors%22%3B&Expires=1723210410&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQxMH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvNTEzNmI5NGZlYjFjZmRmNDViYWI5OWIwN2M1NTVmMWE2OGUyZTU1OWQxNzcxM2Y5ZWFjY2Y2ZTEzMmJlMmRkYj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=ViSuT8vUkmw0MFyV3o0WcCmHlyoNkqr1sTw5wW4yurtGn46qUuaKiV2w1v4UIXUbF9gMyHE15qz9jlt7DJx2f1j0ehlvghO2Ul8zhnk0V52rn2kHR9EqarGake6LoWAwUigd1jiXg76uAAmgEf~Y-sdGbIau9sVn-j~9a1tNX8YAhivDBAN3i6BJJuSU1~PTrRqe1l5HRQBrWtWD~~QJ71cuV1SaXjFLb78rq-VINoAsMZDfLSj5giGTNlEXpuYbtw6VKfl~oAvfurD2sL7RdC90-5ES9xh4tgLwhZj408L2ZmFEdU4l3J-S0DTPiH9SJqapr8QScbM0rVmMxlznwg__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/f919b1ef9202fe7b15871fc3f18610d3f799f8bbdd7b4bbfddc3ce7dc652fd21?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00010-of-00109.safetensors%3B+filename%3D%22model-00010-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZjkxOWIxZWY5MjAyZmU3YjE1ODcxZmMzZjE4NjEwZDNmNzk5ZjhiYmRkN2I0YmJmZGRjM2NlN2RjNjUyZmQyMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=LdVchzwlU5W~SJzOtsnzoMFSAzBLOxtIUTvXRRS4hOVuBY0fqce-yjvrgPXBb3kcnn1tlr5J4ngjxMugq~YScTOmtmKhRWrk2nF1ZH1Z4F93ztuTCsa7A82XE~YMT7i5CCrlqWTq4Cv6SO6dsx7GIJMlNNdb9kYkSMxthZ8XRkUZz-KSN2uyxeKPt1uglL24ziqpuJI-DtD9fsdy9rL3OQm5G0BIMwWnz-YYEOS10ugjGIepOvZ-0WV2LifiUJoRBMv5~ar0Y2qX5OeoMQXz-XSiQG6EdanG5iz9S6xC4TXNM0RLs8AYHI8p-Z0knR5nPPizwZf-tLUKP9vWML-oFw__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/c41de3c512d78c8de6266adffa0db49850ee4692a6d74c2084f6519c450abdd6?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00009-of-00109.safetensors%3B+filename%3D%22model-00009-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvYzQxZGUzYzUxMmQ3OGM4ZGU2MjY2YWRmZmEwZGI0OTg1MGVlNDY5MmE2ZDc0YzIwODRmNjUxOWM0NTBhYmRkNj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=oEWlKqZWagDqtpwoNzOlUJvl3TXKrlf1dlyJAm0tXWRGV7iY2UqbkneuiGK5RhCUqOuuitSpdPiYkjhl1duFOhxqcT7UmNr2JfithcKueq7QC4MUgvuKahx3A5iTSq~Pk1Clfj~cWA1Ev-69ziVRJv9AdU7IfwkvDIgacYYA0fxTWg2cOviHMkdkDllJWLzf4tM-LRvzOZeVXQv3W3Ak-Vh~2Yqf-sU6y88mTEinNW1SJW1LFt~tLNOi4vyynHYzd-F~WsuHM~NNqPBjtHRr6rnyGyJXtGtN7P7BAWfz6mhzWg-3STVV6~T6vf44dCa1vB9QVBe7Ot7gLrcntnTFVA__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/f58ae5a0e97541fa9612723d673c63d3afe7574da0e68291fc3596307dcd2a3d?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00013-of-00109.safetensors%3B+filename%3D%22model-00013-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZjU4YWU1YTBlOTc1NDFmYTk2MTI3MjNkNjczYzYzZDNhZmU3NTc0ZGEwZTY4MjkxZmMzNTk2MzA3ZGNkMmEzZD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=ovu94T1hk5LO68IUMBJl52a87oqdSKKv7tdxZbw0DjuNBjWe8V1kGj2qsi9WBPiOHB7iUZgmpsrT5k1uIzgZhkDu0M~Kg2dWpj9m3M8liHUCKrkqsA~9jkPxFjJYxI9XDxMOdNgQe2d0rmlRKkM940OZSQz1Rij0o6WFQHIUlC2uLaFXu7r1tIDEaDoFJtxzuRih54TDE0FhGZJ8v-VhYjCQCleaQlZ9NpPoYJyuXTESP9NblWgphL-1lNuiu3jW7Sk7tJlnE0Ly6x5zGofOIHMzjaw6CmMy2WeoumWKAmZhELHFzZQOWMZ37~L17Ng5ghOhvpLeSdeqAsy6MFNE2Q__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     message = await receive()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await fut
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e7591a40d0
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     +------------------------------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/5136b94feb1cfdf45bab99b07c555f1a68e2e559d17713f9eaccf6e132be2ddb?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00014-of-00109.safetensors%3B+filename%3D%22model-00014-of-00109.safetensors%22%3B&Expires=1723210410&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQxMH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvNTEzNmI5NGZlYjFjZmRmNDViYWI5OWIwN2M1NTVmMWE2OGUyZTU1OWQxNzcxM2Y5ZWFjY2Y2ZTEzMmJlMmRkYj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=ViSuT8vUkmw0MFyV3o0WcCmHlyoNkqr1sTw5wW4yurtGn46qUuaKiV2w1v4UIXUbF9gMyHE15qz9jlt7DJx2f1j0ehlvghO2Ul8zhnk0V52rn2kHR9EqarGake6LoWAwUigd1jiXg76uAAmgEf~Y-sdGbIau9sVn-j~9a1tNX8YAhivDBAN3i6BJJuSU1~PTrRqe1l5HRQBrWtWD~~QJ71cuV1SaXjFLb78rq-VINoAsMZDfLSj5giGTNlEXpuYbtw6VKfl~oAvfurD2sL7RdC90-5ES9xh4tgLwhZj408L2ZmFEdU4l3J-S0DTPiH9SJqapr8QScbM0rVmMxlznwg__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.0" 403 Forbidden
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/f919b1ef9202fe7b15871fc3f18610d3f799f8bbdd7b4bbfddc3ce7dc652fd21?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00010-of-00109.safetensors%3B+filename%3D%22model-00010-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZjkxOWIxZWY5MjAyZmU3YjE1ODcxZmMzZjE4NjEwZDNmNzk5ZjhiYmRkN2I0YmJmZGRjM2NlN2RjNjUyZmQyMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=LdVchzwlU5W~SJzOtsnzoMFSAzBLOxtIUTvXRRS4hOVuBY0fqce-yjvrgPXBb3kcnn1tlr5J4ngjxMugq~YScTOmtmKhRWrk2nF1ZH1Z4F93ztuTCsa7A82XE~YMT7i5CCrlqWTq4Cv6SO6dsx7GIJMlNNdb9kYkSMxthZ8XRkUZz-KSN2uyxeKPt1uglL24ziqpuJI-DtD9fsdy9rL3OQm5G0BIMwWnz-YYEOS10ugjGIepOvZ-0WV2LifiUJoRBMv5~ar0Y2qX5OeoMQXz-XSiQG6EdanG5iz9S6xC4TXNM0RLs8AYHI8p-Z0knR5nPPizwZf-tLUKP9vWML-oFw__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.0" 403 Forbidden
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/c41de3c512d78c8de6266adffa0db49850ee4692a6d74c2084f6519c450abdd6?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00009-of-00109.safetensors%3B+filename%3D%22model-00009-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvYzQxZGUzYzUxMmQ3OGM4ZGU2MjY2YWRmZmEwZGI0OTg1MGVlNDY5MmE2ZDc0YzIwODRmNjUxOWM0NTBhYmRkNj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=oEWlKqZWagDqtpwoNzOlUJvl3TXKrlf1dlyJAm0tXWRGV7iY2UqbkneuiGK5RhCUqOuuitSpdPiYkjhl1duFOhxqcT7UmNr2JfithcKueq7QC4MUgvuKahx3A5iTSq~Pk1Clfj~cWA1Ev-69ziVRJv9AdU7IfwkvDIgacYYA0fxTWg2cOviHMkdkDllJWLzf4tM-LRvzOZeVXQv3W3Ak-Vh~2Yqf-sU6y88mTEinNW1SJW1LFt~tLNOi4vyynHYzd-F~WsuHM~NNqPBjtHRr6rnyGyJXtGtN7P7BAWfz6mhzWg-3STVV6~T6vf44dCa1vB9QVBe7Ot7gLrcntnTFVA__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.0" 403 Forbidden
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | INFO | stdout | INFO:     172.16.50.12:0 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/f58ae5a0e97541fa9612723d673c63d3afe7574da0e68291fc3596307dcd2a3d?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00013-of-00109.safetensors%3B+filename%3D%22model-00013-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZjU4YWU1YTBlOTc1NDFmYTk2MTI3MjNkNjczYzYzZDNhZmU3NTc0ZGEwZTY4MjkxZmMzNTk2MzA3ZGNkMmEzZD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=ovu94T1hk5LO68IUMBJl52a87oqdSKKv7tdxZbw0DjuNBjWe8V1kGj2qsi9WBPiOHB7iUZgmpsrT5k1uIzgZhkDu0M~Kg2dWpj9m3M8liHUCKrkqsA~9jkPxFjJYxI9XDxMOdNgQe2d0rmlRKkM940OZSQz1Rij0o6WFQHIUlC2uLaFXu7r1tIDEaDoFJtxzuRih54TDE0FhGZJ8v-VhYjCQCleaQlZ9NpPoYJyuXTESP9NblWgphL-1lNuiu3jW7Sk7tJlnE0Ly6x5zGofOIHMzjaw6CmMy2WeoumWKAmZhELHFzZQOWMZ37~L17Ng5ghOhvpLeSdeqAsy6MFNE2Q__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.0" 403 Forbidden
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     message = await receive()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await fut
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e7592274f0
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     +------------------------------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     message = await receive()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await fut
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e75929cfd0
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     +------------------------------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     message = await receive()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await fut
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e75920e680
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     +------------------------------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     message = await receive()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     await fut
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e7591859f0
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise exc
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await func()
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     await send(message)
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:12:05 node1 python[1825074]: 2024-08-08 11:12:05 | ERROR | stderr |     +------------------------------------
Aug 08 11:12:24 node1 python[1825074]: 2024-08-08 11:12:24 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931 "HTTP/1.1 403 Forbidden"
Aug 08 11:12:24 node1 python[1825074]: 2024-08-08 11:12:24 | INFO | stdout | INFO:     127.0.0.1:51142 - "HEAD /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931 HTTP/1.1" 403 Forbidden
Aug 08 11:13:10 node1 python[1825074]: 2024-08-08 11:13:10 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931 "HTTP/1.1 403 Forbidden"
Aug 08 11:13:10 node1 python[1825074]: 2024-08-08 11:13:10 | INFO | stdout | INFO:     127.0.0.1:42842 - "HEAD /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931 HTTP/1.1" 403 Forbidden

@andy108369
Copy link
Author

andy108369 commented Aug 8, 2024 •
edited
Loading

And when directly accessing olah server, without nginx in between

(.venv) root@node1:~# export HF_ENDPOINT=http://127.0.0.1:8090
(.venv) root@node1:~# read -s HF_TOKEN
(.venv) root@node1:~# export HF_TOKEN
(.venv) root@node1:~# python -c "from huggingface_hub import HfApi; print(HfApi().endpoint)"
http://127.0.0.1:8090
(.venv) root@node1:~# huggingface-cli download --resume-download meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --local-dir /data/test-llama31-405b-fp8
/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Fetching 242 files:   0%|                                                                                                                                                                                                                                               | 0/242 [00:00<?, ?it/s]Downloading 'USE_POLICY.md' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/USE_POLICY.md.81ebb55902285e8dd5804ccf423d17ffb2a622ee.incomplete'
Downloading 'generation_config.json' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/generation_config.json.c85e356463cb94425264a907d7c3fea022326e80.incomplete'
Downloading '.gitattributes' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/.gitattributes.a6344aac8c09253b3b630fb776ae94478aa0275b.incomplete'
Downloading 'config.json' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/config.json.ed166d7b95e1910b9e567693971c6f220cd68915.incomplete'
Downloading 'LICENSE' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/LICENSE.a7c3ca16cee30425ed6ad841a809590f2bcbf290.incomplete'
Downloading 'model-00001-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00001-of-00109.safetensors.27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf.incomplete'
Downloading 'README.md' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/README.md.c69aa1991ae64d06636c513373b0978ea16bc9f1.incomplete'
Downloading 'model-00002-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00002-of-00109.safetensors.efef014770d588f4206f2da94ac251db8958f6a8ce1e38762d28c044bb015198.incomplete'
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 22.5k/22.5k [00:00<00:00, 44.8MB/s]
Download complete. Moving file to /data/test-llama31-405b-fp8/config.json                                                                                                                                                                                           | 0.00/22.5k [00:00<?, ?B/s]
generation_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 189/189 [00:00<00:00, 683kB/s]
Download complete. Moving file to /data/test-llama31-405b-fp8/generation_config.json                                                                                                                                                                                  | 0.00/189 [00:00<?, ?B/s]

Downloading 'model-00003-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00003-of-00109.safetensors.26bb5d1cd5b224dee88f442b1a6acdf196a5d647d5af6834c62aa6a2a01bdd05.incomplete'                                                                       Downloading 'model-00004-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00004-of-00109.safetensors.376536d7284c6a79a65ee8e3fb3ee3fdf1a13040f72ac31ca7d2e4b13e7d52a3.incomplete'                                             | 0.00/4.69k [00:00<?, ?B/s]
USE_POLICY.md: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.69k/4.69k [00:00<00:00, 336kB/s]
Download complete. Moving file to /data/test-llama31-405b-fp8/USE_POLICY.md
README.md: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40.3k/40.3k [00:00<00:00, 4.25MB/s]
Download complete. Moving file to /data/test-llama31-405b-fp8/README.md                                                                                                                                                                                             | 0.00/1.52k [00:00<?, ?B/s]
.gitattributes: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.52k/1.52k [00:00<00:00, 340kB/s]
Download complete. Moving file to /data/test-llama31-405b-fp8/.gitattributes
LICENSE: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.63k/7.63k [00:00<00:00, 19.6MB/s]
Download complete. Moving file to /data/test-llama31-405b-fp8/LICENSE                                                                                                                                                                                               | 0.00/7.63k [00:00<?, ?B/s]
Fetching 242 files:   2%|█████▋                                                                                                                                                                                                                                 | 6/242 [00:03<02:26,  1.61it/s]
Downloading 'model-00009-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00009-of-00109.safetensors.c41de3c512d78c8de6266adffa0db49850ee4692a6d74c2084f6519c450abdd6.incomplete'
Downloading 'model-00007-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00007-of-00109.safetensors.1fcdeb3bc4556d7da5bdf242919dc50df950563d8191d3be518fb001ab3e6331.incomplete'
Downloading 'model-00010-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00010-of-00109.safetensors.f919b1ef9202fe7b15871fc3f18610d3f799f8bbdd7b4bbfddc3ce7dc652fd21.incomplete'
Downloading 'model-00005-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00005-of-00109.safetensors.fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931.incomplete'
Downloading 'model-00008-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00008-of-00109.safetensors.410058afd38e1cdbca5a1a9cdae4e25f6d006763c3b224b6101611049458e109.incomplete'
Downloading 'model-00006-of-00109.safetensors' to '/data/test-llama31-405b-fp8/.cache/huggingface/download/model-00006-of-00109.safetensors.877173ef17665d02a697423e2b5c1e0fd8b3f50a8ba5c09b413896027c487789.incomplete'
Traceback (most recent call last):
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://localhost:8090/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00109.safetensors%3B+filename%3D%22model-00001-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjdkNjc2MGUwMGYzYjhmY2Y2YWI3ODI0YTJkZTUwZjA3ZjU5Y2MwOGIwM2QyNjNiNmMwZDU2MmRhZjg2MjJiZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=fr9up4qmn0-~hwfDOHbqsKbRUKmaY~arE~9CHFuq5soQPLXTV4cwysE2REWplGAadJZCJxlDWUxM6ira3HjnkFt4hqtzSG2Sb3nDJOQ-8lRLD7T2XUIgLtGiwn-zcIaXOOerHg9~rI8R~Q7QpGX078lpTpml0-chIzB8FrHmmMAfVCvnDsGIsT2FYV-hqR2m-YWIlAty7a-WlBykkDpkaQNu3iz-VYztzJviTVQo4IDRd~6P6euGLW5iyTEzQsi1-gYEm6dBfyrazhvHqEzLr56Tm0Y43AewnhYoWC5a~onjcAy502CdYKUcDDCzSjlsaMZ7baHLNu7YOdPwPYCuXA__&Key-Pair-Id=K24J24Z295AEI9

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 748, in _error_catcher
    yield
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 894, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(0 bytes read, 106 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 820, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 1060, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 949, in read
    data = self._raw_read(amt)
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 872, in _raw_read
    with self._error_catcher():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/root/.venv/lib/python3.10/site-packages/urllib3/response.py", line 772, in _error_catcher
    raise ProtocolError(arg, e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read, 106 more expected)', IncompleteRead(0 bytes read, 106 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.venv/bin/huggingface-cli", line 8, in <module>
    sys.exit(main())
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/huggingface_cli.py", line 52, in main
    service.run()
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/download.py", line 146, in run
    print(self._download())  # Print path to downloaded files
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/commands/download.py", line 180, in _download
    return snapshot_download(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 299, in snapshot_download
    thread_map(
  File "/root/.venv/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
  File "/root/.venv/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
  File "/root/.venv/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 273, in _inner_hf_hub_download
    return hf_hub_download(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
    return f(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1220, in hf_hub_download
    return _hf_hub_download_to_local_dir(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1515, in _hf_hub_download_to_local_dir
    _download_to_tmp_and_move(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1915, in _download_to_tmp_and_move
    http_get(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 463, in http_get
    r = _request_wrapper(
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 400, in _request_wrapper
    hf_raise_for_status(response)
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 367, in hf_raise_for_status
    raise HfHubHTTPError(message, response=response) from e
  File "/root/.venv/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 70, in __init__
    server_data = response.json()
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 974, in json
    return complexjson.loads(self.text, **kwargs)
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 926, in text
    if not self.content:
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 902, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/root/.venv/lib/python3.10/site-packages/requests/models.py", line 822, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read, 106 more expected)', IncompleteRead(0 bytes read, 106 more expected))
(.venv) root@node1:~#

olah server

(.venv) root@node1:~# journalctl -u olah-server --since '3 minutes ago' --no-pager -l
Aug 08 11:18:49 node1 python[1825074]: 2024-08-08 11:18:49 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 08 11:18:49 node1 python[1825074]: 2024-08-08 11:18:49 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 08 11:18:49 node1 python[1825074]: 2024-08-08 11:18:49 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 08 11:18:49 node1 python[1825074]: 2024-08-08 11:18:49 | INFO | stdout | INFO:     127.0.0.1:56504 - "GET /api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main HTTP/1.1" 200 OK
Aug 08 11:18:50 node1 python[1825074]: 2024-08-08 11:18:50 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:50 node1 python[1825074]: 2024-08-08 11:18:50 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:50 node1 python[1825074]: 2024-08-08 11:18:50 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:50 node1 python[1825074]: 2024-08-08 11:18:50 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:50 node1 python[1825074]: 2024-08-08 11:18:50 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:50 node1 python[1825074]: 2024-08-08 11:18:50 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:50 node1 python[1825074]: 2024-08-08 11:18:50 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:50 node1 python[1825074]: 2024-08-08 11:18:50 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:50 node1 python[1825074]: 2024-08-08 11:18:50 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | stdout | INFO:     127.0.0.1:56520 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/USE_POLICY.md HTTP/1.1" 200 OK
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | stdout | INFO:     127.0.0.1:56540 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/generation_config.json HTTP/1.1" 200 OK
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | stdout | INFO:     127.0.0.1:56528 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/config.json HTTP/1.1" 200 OK
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | stdout | INFO:     127.0.0.1:56588 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/.gitattributes HTTP/1.1" 200 OK
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | stdout | INFO:     127.0.0.1:56578 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00001-of-00109.safetensors HTTP/1.1" 302 Found
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | stdout | INFO:     127.0.0.1:56568 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/LICENSE HTTP/1.1" 200 OK
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | stdout | INFO:     127.0.0.1:56538 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/README.md HTTP/1.1" 200 OK
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | stdout | INFO:     127.0.0.1:56554 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00002-of-00109.safetensors HTTP/1.1" 302 Found
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/USE_POLICY.md "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/config.json "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/generation_config.json "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/.gitattributes "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/LICENSE "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/efef014770d588f4206f2da94ac251db8958f6a8ce1e38762d28c044bb015198?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00002-of-00109.safetensors%3B+filename%3D%22model-00002-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZWZlZjAxNDc3MGQ1ODhmNDIwNmYyZGE5NGFjMjUxZGI4OTU4ZjZhOGNlMWUzODc2MmQyOGMwNDRiYjAxNTE5OD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=ADDXo1pVuG2LrImPpqCiE60h~CX3iCkRyuFQAkY5lfglXF~-SjXSg2L2xKGlLkMLMgYq4ysIruczmVFgKRf-wTsVJ~4KunqLx2ALlfA5GkExHlKEcbfsDeW8PbYL8AiutlSFeLVfHk4mXKo-acKrSuZifdGKSHhEt8oILU49SNklZ-dY4GqXn~wvc~FEssdOBVuU7qqptzHGGa-Caf~Si3xS1SlPJcs0gWoXwWR83N-dkimexQbfGVbxPuuVd6oCCxBsdqjpMIR1j6AyPsPftetLosrNBiTHOkkePdwlLM5eykoRxln7MQ3gCkxTUIsb6b20zwnzPpxR54pFCDRYuw__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00109.safetensors%3B+filename%3D%22model-00001-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjdkNjc2MGUwMGYzYjhmY2Y2YWI3ODI0YTJkZTUwZjA3ZjU5Y2MwOGIwM2QyNjNiNmMwZDU2MmRhZjg2MjJiZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=fr9up4qmn0-~hwfDOHbqsKbRUKmaY~arE~9CHFuq5soQPLXTV4cwysE2REWplGAadJZCJxlDWUxM6ira3HjnkFt4hqtzSG2Sb3nDJOQ-8lRLD7T2XUIgLtGiwn-zcIaXOOerHg9~rI8R~Q7QpGX078lpTpml0-chIzB8FrHmmMAfVCvnDsGIsT2FYV-hqR2m-YWIlAty7a-WlBykkDpkaQNu3iz-VYztzJviTVQo4IDRd~6P6euGLW5iyTEzQsi1-gYEm6dBfyrazhvHqEzLr56Tm0Y43AewnhYoWC5a~onjcAy502CdYKUcDDCzSjlsaMZ7baHLNu7YOdPwPYCuXA__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/README.md "HTTP/1.1 200 OK"
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | stdout | INFO:     127.0.0.1:40748 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/efef014770d588f4206f2da94ac251db8958f6a8ce1e38762d28c044bb015198?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00002-of-00109.safetensors%3B+filename%3D%22model-00002-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZWZlZjAxNDc3MGQ1ODhmNDIwNmYyZGE5NGFjMjUxZGI4OTU4ZjZhOGNlMWUzODc2MmQyOGMwNDRiYjAxNTE5OD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=ADDXo1pVuG2LrImPpqCiE60h~CX3iCkRyuFQAkY5lfglXF~-SjXSg2L2xKGlLkMLMgYq4ysIruczmVFgKRf-wTsVJ~4KunqLx2ALlfA5GkExHlKEcbfsDeW8PbYL8AiutlSFeLVfHk4mXKo-acKrSuZifdGKSHhEt8oILU49SNklZ-dY4GqXn~wvc~FEssdOBVuU7qqptzHGGa-Caf~Si3xS1SlPJcs0gWoXwWR83N-dkimexQbfGVbxPuuVd6oCCxBsdqjpMIR1j6AyPsPftetLosrNBiTHOkkePdwlLM5eykoRxln7MQ3gCkxTUIsb6b20zwnzPpxR54pFCDRYuw__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 Forbidden
Aug 08 11:18:51 node1 python[1825074]: 2024-08-08 11:18:51 | INFO | stdout | INFO:     127.0.0.1:40744 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/27d6760e00f3b8fcf6ab7824a2de50f07f59cc08b03d263b6c0d562daf8622bf?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00109.safetensors%3B+filename%3D%22model-00001-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjdkNjc2MGUwMGYzYjhmY2Y2YWI3ODI0YTJkZTUwZjA3ZjU5Y2MwOGIwM2QyNjNiNmMwZDU2MmRhZjg2MjJiZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=fr9up4qmn0-~hwfDOHbqsKbRUKmaY~arE~9CHFuq5soQPLXTV4cwysE2REWplGAadJZCJxlDWUxM6ira3HjnkFt4hqtzSG2Sb3nDJOQ-8lRLD7T2XUIgLtGiwn-zcIaXOOerHg9~rI8R~Q7QpGX078lpTpml0-chIzB8FrHmmMAfVCvnDsGIsT2FYV-hqR2m-YWIlAty7a-WlBykkDpkaQNu3iz-VYztzJviTVQo4IDRd~6P6euGLW5iyTEzQsi1-gYEm6dBfyrazhvHqEzLr56Tm0Y43AewnhYoWC5a~onjcAy502CdYKUcDDCzSjlsaMZ7baHLNu7YOdPwPYCuXA__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 Forbidden
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     await func()
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     message = await receive()
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     await fut
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e7592600a0
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     raise exc
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     raise exc
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     raise exc
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     await func()
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     +------------------------------------
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     await func()
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     message = await receive()
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     await fut
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e75b3a4ac0
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     raise exc
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     raise exc
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     raise exc
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     await func()
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | ERROR | stderr |     +------------------------------------
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | stdout | INFO:     127.0.0.1:56528 - "GET /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/config.json HTTP/1.1" 200 OK
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:52 node1 python[1825074]: 2024-08-08 11:18:52 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | stdout | INFO:     127.0.0.1:56538 - "GET /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/README.md HTTP/1.1" 200 OK
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | stdout | INFO:     127.0.0.1:56588 - "GET /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/.gitattributes HTTP/1.1" 200 OK
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | stdout | INFO:     127.0.0.1:56540 - "GET /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/generation_config.json HTTP/1.1" 200 OK
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | stdout | INFO:     127.0.0.1:56520 - "GET /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/USE_POLICY.md HTTP/1.1" 200 OK
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | stdout | INFO:     127.0.0.1:56554 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00004-of-00109.safetensors HTTP/1.1" 302 Found
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | stdout | INFO:     127.0.0.1:56578 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00003-of-00109.safetensors HTTP/1.1" 302 Found
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | stdout | INFO:     127.0.0.1:56568 - "GET /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/LICENSE HTTP/1.1" 200 OK
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/26bb5d1cd5b224dee88f442b1a6acdf196a5d647d5af6834c62aa6a2a01bdd05?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00003-of-00109.safetensors%3B+filename%3D%22model-00003-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjZiYjVkMWNkNWIyMjRkZWU4OGY0NDJiMWE2YWNkZjE5NmE1ZDY0N2Q1YWY2ODM0YzYyYWE2YTJhMDFiZGQwNT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=cVwJvcVKRglS-l8R4x0fOGzbKh4KxtALoy28bILtwZ0wjPgMlb9wtMOLv08dHqmFikxF05gGMpke9PaogPbSw-CozS6WAt~rDOqiBHQGZgZSNY~denc~ia5912e1VklQl0~qnxxhy7zLnysI7v-r4uOZwLj5QQ9tzqZmgdycJdh2PD88thfy3nbaeWiFnlOdsn~Eg7R7vaWkAhbYyItTGK-AL3fv3ljNJzwTFv25n3DNwbSiDnVNoqdc-jOWLWM0J-nslsQDoJmalrllI-LY0h3IAyvxsAmNtnx3pEubUexQZnSyIV8d2hTTbSWQtVOuIjU3ecna34O983EbxIjZiA__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/376536d7284c6a79a65ee8e3fb3ee3fdf1a13040f72ac31ca7d2e4b13e7d52a3?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00004-of-00109.safetensors%3B+filename%3D%22model-00004-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMzc2NTM2ZDcyODRjNmE3OWE2NWVlOGUzZmIzZWUzZmRmMWExMzA0MGY3MmFjMzFjYTdkMmU0YjEzZTdkNTJhMz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=XoyQObuOUeprk8VIWFqSw6VB0srKdnOZSVadjvCGvdtiwh8mJy39vmTYIGWE~oU42aZDoo7BvjzbcG~T6m6aeUgLL8nTd4Ic1kxXpx9nCR5hYgkriitKygfWzfHtAMwEoJiO71oLo9TycX6~O54HrmHGxxe3n0VirYW18UTHYeck2pLSE2gFSOdZFX6XycZ6zoxA9Uc7eGWRwVCaodvPFBpUbG5US2Q~bzYuvkwujkmNxQW-k3OtKndUaCfCTWU8PVotC5Q5Kd7z01aKdNDvTC5N~em3A0hwGqFG-UJnJzpiY4XYeEUvDcZOLTya4UthL0Jp75eM4mrGBWKOfrnrgg__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | stdout | INFO:     127.0.0.1:40764 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/26bb5d1cd5b224dee88f442b1a6acdf196a5d647d5af6834c62aa6a2a01bdd05?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00003-of-00109.safetensors%3B+filename%3D%22model-00003-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMjZiYjVkMWNkNWIyMjRkZWU4OGY0NDJiMWE2YWNkZjE5NmE1ZDY0N2Q1YWY2ODM0YzYyYWE2YTJhMDFiZGQwNT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=cVwJvcVKRglS-l8R4x0fOGzbKh4KxtALoy28bILtwZ0wjPgMlb9wtMOLv08dHqmFikxF05gGMpke9PaogPbSw-CozS6WAt~rDOqiBHQGZgZSNY~denc~ia5912e1VklQl0~qnxxhy7zLnysI7v-r4uOZwLj5QQ9tzqZmgdycJdh2PD88thfy3nbaeWiFnlOdsn~Eg7R7vaWkAhbYyItTGK-AL3fv3ljNJzwTFv25n3DNwbSiDnVNoqdc-jOWLWM0J-nslsQDoJmalrllI-LY0h3IAyvxsAmNtnx3pEubUexQZnSyIV8d2hTTbSWQtVOuIjU3ecna34O983EbxIjZiA__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 Forbidden
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:53 node1 python[1825074]: 2024-08-08 11:18:53 | INFO | stdout | INFO:     127.0.0.1:40778 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/376536d7284c6a79a65ee8e3fb3ee3fdf1a13040f72ac31ca7d2e4b13e7d52a3?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00004-of-00109.safetensors%3B+filename%3D%22model-00004-of-00109.safetensors%22%3B&Expires=1723210405&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMzc2NTM2ZDcyODRjNmE3OWE2NWVlOGUzZmIzZWUzZmRmMWExMzA0MGY3MmFjMzFjYTdkMmU0YjEzZTdkNTJhMz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=XoyQObuOUeprk8VIWFqSw6VB0srKdnOZSVadjvCGvdtiwh8mJy39vmTYIGWE~oU42aZDoo7BvjzbcG~T6m6aeUgLL8nTd4Ic1kxXpx9nCR5hYgkriitKygfWzfHtAMwEoJiO71oLo9TycX6~O54HrmHGxxe3n0VirYW18UTHYeck2pLSE2gFSOdZFX6XycZ6zoxA9Uc7eGWRwVCaodvPFBpUbG5US2Q~bzYuvkwujkmNxQW-k3OtKndUaCfCTWU8PVotC5Q5Kd7z01aKdNDvTC5N~em3A0hwGqFG-UJnJzpiY4XYeEUvDcZOLTya4UthL0Jp75eM4mrGBWKOfrnrgg__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 Forbidden
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     await func()
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     message = await receive()
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     await fut
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e75926c490
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     raise exc
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     raise exc
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     raise exc
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     await func()
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     +------------------------------------
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     await func()
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     message = await receive()
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     await fut
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e7591f6b90
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     raise exc
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     raise exc
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     raise exc
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     await func()
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | ERROR | stderr |     +------------------------------------
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | stdout | INFO:     127.0.0.1:56588 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00009-of-00109.safetensors HTTP/1.1" 302 Found
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | stdout | INFO:     127.0.0.1:56538 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00007-of-00109.safetensors HTTP/1.1" 302 Found
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/7c365a34d0fed5b4afb118cf253395f47f6efa72 "HTTP/1.1 200 OK"
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | stdout | INFO:     127.0.0.1:56540 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00006-of-00109.safetensors HTTP/1.1" 302 Found
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | stdout | INFO:     127.0.0.1:56568 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00010-of-00109.safetensors HTTP/1.1" 302 Found
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | stdout | INFO:     127.0.0.1:56520 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00008-of-00109.safetensors HTTP/1.1" 302 Found
Aug 08 11:18:54 node1 python[1825074]: 2024-08-08 11:18:54 | INFO | stdout | INFO:     127.0.0.1:56528 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/model-00005-of-00109.safetensors HTTP/1.1" 302 Found
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/c41de3c512d78c8de6266adffa0db49850ee4692a6d74c2084f6519c450abdd6?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00009-of-00109.safetensors%3B+filename%3D%22model-00009-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvYzQxZGUzYzUxMmQ3OGM4ZGU2MjY2YWRmZmEwZGI0OTg1MGVlNDY5MmE2ZDc0YzIwODRmNjUxOWM0NTBhYmRkNj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=oEWlKqZWagDqtpwoNzOlUJvl3TXKrlf1dlyJAm0tXWRGV7iY2UqbkneuiGK5RhCUqOuuitSpdPiYkjhl1duFOhxqcT7UmNr2JfithcKueq7QC4MUgvuKahx3A5iTSq~Pk1Clfj~cWA1Ev-69ziVRJv9AdU7IfwkvDIgacYYA0fxTWg2cOviHMkdkDllJWLzf4tM-LRvzOZeVXQv3W3Ak-Vh~2Yqf-sU6y88mTEinNW1SJW1LFt~tLNOi4vyynHYzd-F~WsuHM~NNqPBjtHRr6rnyGyJXtGtN7P7BAWfz6mhzWg-3STVV6~T6vf44dCa1vB9QVBe7Ot7gLrcntnTFVA__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | stdout | INFO:     127.0.0.1:40786 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/c41de3c512d78c8de6266adffa0db49850ee4692a6d74c2084f6519c450abdd6?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00009-of-00109.safetensors%3B+filename%3D%22model-00009-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvYzQxZGUzYzUxMmQ3OGM4ZGU2MjY2YWRmZmEwZGI0OTg1MGVlNDY5MmE2ZDc0YzIwODRmNjUxOWM0NTBhYmRkNj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=oEWlKqZWagDqtpwoNzOlUJvl3TXKrlf1dlyJAm0tXWRGV7iY2UqbkneuiGK5RhCUqOuuitSpdPiYkjhl1duFOhxqcT7UmNr2JfithcKueq7QC4MUgvuKahx3A5iTSq~Pk1Clfj~cWA1Ev-69ziVRJv9AdU7IfwkvDIgacYYA0fxTWg2cOviHMkdkDllJWLzf4tM-LRvzOZeVXQv3W3Ak-Vh~2Yqf-sU6y88mTEinNW1SJW1LFt~tLNOi4vyynHYzd-F~WsuHM~NNqPBjtHRr6rnyGyJXtGtN7P7BAWfz6mhzWg-3STVV6~T6vf44dCa1vB9QVBe7Ot7gLrcntnTFVA__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 Forbidden
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     message = await receive()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await fut
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e7591460e0
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     +------------------------------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/f919b1ef9202fe7b15871fc3f18610d3f799f8bbdd7b4bbfddc3ce7dc652fd21?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00010-of-00109.safetensors%3B+filename%3D%22model-00010-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZjkxOWIxZWY5MjAyZmU3YjE1ODcxZmMzZjE4NjEwZDNmNzk5ZjhiYmRkN2I0YmJmZGRjM2NlN2RjNjUyZmQyMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=LdVchzwlU5W~SJzOtsnzoMFSAzBLOxtIUTvXRRS4hOVuBY0fqce-yjvrgPXBb3kcnn1tlr5J4ngjxMugq~YScTOmtmKhRWrk2nF1ZH1Z4F93ztuTCsa7A82XE~YMT7i5CCrlqWTq4Cv6SO6dsx7GIJMlNNdb9kYkSMxthZ8XRkUZz-KSN2uyxeKPt1uglL24ziqpuJI-DtD9fsdy9rL3OQm5G0BIMwWnz-YYEOS10ugjGIepOvZ-0WV2LifiUJoRBMv5~ar0Y2qX5OeoMQXz-XSiQG6EdanG5iz9S6xC4TXNM0RLs8AYHI8p-Z0knR5nPPizwZf-tLUKP9vWML-oFw__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | stdout | INFO:     127.0.0.1:40798 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/f919b1ef9202fe7b15871fc3f18610d3f799f8bbdd7b4bbfddc3ce7dc652fd21?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00010-of-00109.safetensors%3B+filename%3D%22model-00010-of-00109.safetensors%22%3B&Expires=1723210408&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwOH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZjkxOWIxZWY5MjAyZmU3YjE1ODcxZmMzZjE4NjEwZDNmNzk5ZjhiYmRkN2I0YmJmZGRjM2NlN2RjNjUyZmQyMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=LdVchzwlU5W~SJzOtsnzoMFSAzBLOxtIUTvXRRS4hOVuBY0fqce-yjvrgPXBb3kcnn1tlr5J4ngjxMugq~YScTOmtmKhRWrk2nF1ZH1Z4F93ztuTCsa7A82XE~YMT7i5CCrlqWTq4Cv6SO6dsx7GIJMlNNdb9kYkSMxthZ8XRkUZz-KSN2uyxeKPt1uglL24ziqpuJI-DtD9fsdy9rL3OQm5G0BIMwWnz-YYEOS10ugjGIepOvZ-0WV2LifiUJoRBMv5~ar0Y2qX5OeoMQXz-XSiQG6EdanG5iz9S6xC4TXNM0RLs8AYHI8p-Z0knR5nPPizwZf-tLUKP9vWML-oFw__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 Forbidden
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00005-of-00109.safetensors%3B+filename%3D%22model-00005-of-00109.safetensors%22%3B&Expires=1723210407&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwN319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZmJhNzBjODJhODAwZGEzMGRmMWYwOGEyZmQ4MDgyOTY3OWQyOTBkYWMxNDBhODIwM2M5YTRjODA3NmVhYjkzMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=M779EGlTJZrB8vwcRYhrDogKDvR9NGLStKkNJq3Zi0SfkwWqCfZ9HWnAsv3~ODnukcsPaZGk6PLSRv0SsTKnoDyekWWsiJ6UeNBPPJyhMZFLX9O-tmhNYhChMPg24IHwXY770TyFM6QCTn25lt-INXeBJACUHJdQTUxbuvu8EE-XJo7e4d6-ZG3~zjmVWSff2T1TQWaTt~UyAl8jRtyoc26K8Okny-EmdDvv5bSZPNikcyVZQ9-xj9k10BnWrGcFXbLr163CIeceUulpY1wvOJ0veuUJwvYEK8~jE4elaURfp4ddt5ex8b58Z2QtDXtx3~MD40cqlvxrKLrEYpeasw__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/410058afd38e1cdbca5a1a9cdae4e25f6d006763c3b224b6101611049458e109?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00008-of-00109.safetensors%3B+filename%3D%22model-00008-of-00109.safetensors%22%3B&Expires=1723210407&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwN319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvNDEwMDU4YWZkMzhlMWNkYmNhNWExYTljZGFlNGUyNWY2ZDAwNjc2M2MzYjIyNGI2MTAxNjExMDQ5NDU4ZTEwOT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=eBNGxEffrnrpv1Odvdxrdk7nJTQnyCXrWBGU2kFLyWowi6B-CXCWjX9s6TTcfNOXkHNpdh5ZnJRx9ggp~ePcgDt0M42MhofGYSdtvr4YcguATdfJDsupnzp2Kq6Sx5ovK~e0K-ZFHSqGVm8MlZK4K4QpQWw8pUi1U5IvBePpiXSWdn~iRQXZ6KlHu7j5XwOSq0wZehnDajJXW6P2IZ-SoliPdmYSQlQ77vzf1klG5Ly2YT6v7U~bOpCmIrNG-Uc7VqKfmuM1-VKs0U8gUWrz1dg0PbYVOb~FdMk9AjuLcRR9eUjK8fgKKVl~vcl1RKvfbaX4NYtRssGz3~8LbkYH5w__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/1fcdeb3bc4556d7da5bdf242919dc50df950563d8191d3be518fb001ab3e6331?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00007-of-00109.safetensors%3B+filename%3D%22model-00007-of-00109.safetensors%22%3B&Expires=1723208568&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIwODU2OH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMWZjZGViM2JjNDU1NmQ3ZGE1YmRmMjQyOTE5ZGM1MGRmOTUwNTYzZDgxOTFkM2JlNTE4ZmIwMDFhYjNlNjMzMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=JfEBKJOblMxIQ6QsdD-LVSQ0DxmMeMGH028bDnLzVBsUn3Vb9Ws7PoRTF2~5thgHoGoLLjnnDtgRPST9ny1EF3l9iamunQ6NK9UeLEa5LY3vFNRPrAE-QKmBAtNps2vxwqDbF5XqiEzIxJ03jIC-T73xfxxlepz34sFhKpIo3xjBh7Qi23hddEMcttOTlGWa4Lh5kIi5L4FqizznpCVqIhAHt5RZulVYC6OFio3mOwN2mycHiGId66JEkZwxd324wbn4Ihaw8XLzhkz0ac4qFns~VeOkhvE5cZUZbQWNmrp~RfNCe0x5TQ7X9qH1KdRRB2rPogdJmksyap-~B4Lz7A__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | httpx | HTTP Request: HEAD https://cdn-lfs.huggingface.co/repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/877173ef17665d02a697423e2b5c1e0fd8b3f50a8ba5c09b413896027c487789?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00006-of-00109.safetensors%3B+filename%3D%22model-00006-of-00109.safetensors%22%3B&Expires=1723210407&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwN319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvODc3MTczZWYxNzY2NWQwMmE2OTc0MjNlMmI1YzFlMGZkOGIzZjUwYThiYTVjMDliNDEzODk2MDI3YzQ4Nzc4OT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=AUVV86QUBmfTFfkh7rQ9v8A-VDXR-9qmHU9l8VCvMFJQ6tapLOsNiBevAOavV136~oaxI6UpsFgViy6bx~EJ5JlG2490OAwjDM9x8yRhVWZnNrUGyK5oJe7i4duXs8rnddwPWb4Wks8e8LGD1ZQ0ceflVer4GCuM4UHXa6~eTQoFkscK0KPCM-MP9YGboQ5va1V5c0JM4JmSvoHeBKBd5qS7HFqgf4tAPdPJW0Zl5zkPYi682Lk9xTWddtaU8QCGhpO8KGlbnYuRZh0ok56HyFYzCfzEKZSl-hepBDf1pA2TGDC1n5wNGzM9nwpvCafsuSl4BQZh8N2iCygI-7iR7g__&Key-Pair-Id=K24J24Z295AEI9 "HTTP/1.1 403 Forbidden"
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | stdout | INFO:     127.0.0.1:40802 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/fba70c82a800da30df1f08a2fd80829679d290dac140a8203c9a4c8076eab931?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00005-of-00109.safetensors%3B+filename%3D%22model-00005-of-00109.safetensors%22%3B&Expires=1723210407&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwN319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvZmJhNzBjODJhODAwZGEzMGRmMWYwOGEyZmQ4MDgyOTY3OWQyOTBkYWMxNDBhODIwM2M5YTRjODA3NmVhYjkzMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=M779EGlTJZrB8vwcRYhrDogKDvR9NGLStKkNJq3Zi0SfkwWqCfZ9HWnAsv3~ODnukcsPaZGk6PLSRv0SsTKnoDyekWWsiJ6UeNBPPJyhMZFLX9O-tmhNYhChMPg24IHwXY770TyFM6QCTn25lt-INXeBJACUHJdQTUxbuvu8EE-XJo7e4d6-ZG3~zjmVWSff2T1TQWaTt~UyAl8jRtyoc26K8Okny-EmdDvv5bSZPNikcyVZQ9-xj9k10BnWrGcFXbLr163CIeceUulpY1wvOJ0veuUJwvYEK8~jE4elaURfp4ddt5ex8b58Z2QtDXtx3~MD40cqlvxrKLrEYpeasw__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 Forbidden
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | stdout | INFO:     127.0.0.1:40804 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/410058afd38e1cdbca5a1a9cdae4e25f6d006763c3b224b6101611049458e109?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00008-of-00109.safetensors%3B+filename%3D%22model-00008-of-00109.safetensors%22%3B&Expires=1723210407&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwN319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvNDEwMDU4YWZkMzhlMWNkYmNhNWExYTljZGFlNGUyNWY2ZDAwNjc2M2MzYjIyNGI2MTAxNjExMDQ5NDU4ZTEwOT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=eBNGxEffrnrpv1Odvdxrdk7nJTQnyCXrWBGU2kFLyWowi6B-CXCWjX9s6TTcfNOXkHNpdh5ZnJRx9ggp~ePcgDt0M42MhofGYSdtvr4YcguATdfJDsupnzp2Kq6Sx5ovK~e0K-ZFHSqGVm8MlZK4K4QpQWw8pUi1U5IvBePpiXSWdn~iRQXZ6KlHu7j5XwOSq0wZehnDajJXW6P2IZ-SoliPdmYSQlQ77vzf1klG5Ly2YT6v7U~bOpCmIrNG-Uc7VqKfmuM1-VKs0U8gUWrz1dg0PbYVOb~FdMk9AjuLcRR9eUjK8fgKKVl~vcl1RKvfbaX4NYtRssGz3~8LbkYH5w__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 Forbidden
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | stdout | INFO:     127.0.0.1:40788 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/1fcdeb3bc4556d7da5bdf242919dc50df950563d8191d3be518fb001ab3e6331?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00007-of-00109.safetensors%3B+filename%3D%22model-00007-of-00109.safetensors%22%3B&Expires=1723208568&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIwODU2OH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvMWZjZGViM2JjNDU1NmQ3ZGE1YmRmMjQyOTE5ZGM1MGRmOTUwNTYzZDgxOTFkM2JlNTE4ZmIwMDFhYjNlNjMzMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=JfEBKJOblMxIQ6QsdD-LVSQ0DxmMeMGH028bDnLzVBsUn3Vb9Ws7PoRTF2~5thgHoGoLLjnnDtgRPST9ny1EF3l9iamunQ6NK9UeLEa5LY3vFNRPrAE-QKmBAtNps2vxwqDbF5XqiEzIxJ03jIC-T73xfxxlepz34sFhKpIo3xjBh7Qi23hddEMcttOTlGWa4Lh5kIi5L4FqizznpCVqIhAHt5RZulVYC6OFio3mOwN2mycHiGId66JEkZwxd324wbn4Ihaw8XLzhkz0ac4qFns~VeOkhvE5cZUZbQWNmrp~RfNCe0x5TQ7X9qH1KdRRB2rPogdJmksyap-~B4Lz7A__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 Forbidden
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     message = await receive()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await fut
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e7591a45e0
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     +------------------------------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | INFO | stdout | INFO:     127.0.0.1:40812 - "GET /repos/c6/4c/c64c854f39008d2148640af0dd443a6a3334f993ea71b7a970e2ac0dbdb25591/877173ef17665d02a697423e2b5c1e0fd8b3f50a8ba5c09b413896027c487789?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00006-of-00109.safetensors%3B+filename%3D%22model-00006-of-00109.safetensors%22%3B&Expires=1723210407&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMzIxMDQwN319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zL2M2LzRjL2M2NGM4NTRmMzkwMDhkMjE0ODY0MGFmMGRkNDQzYTZhMzMzNGY5OTNlYTcxYjdhOTcwZTJhYzBkYmRiMjU1OTEvODc3MTczZWYxNzY2NWQwMmE2OTc0MjNlMmI1YzFlMGZkOGIzZjUwYThiYTVjMDliNDEzODk2MDI3YzQ4Nzc4OT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=AUVV86QUBmfTFfkh7rQ9v8A-VDXR-9qmHU9l8VCvMFJQ6tapLOsNiBevAOavV136~oaxI6UpsFgViy6bx~EJ5JlG2490OAwjDM9x8yRhVWZnNrUGyK5oJe7i4duXs8rnddwPWb4Wks8e8LGD1ZQ0ceflVer4GCuM4UHXa6~eTQoFkscK0KPCM-MP9YGboQ5va1V5c0JM4JmSvoHeBKBd5qS7HFqgf4tAPdPJW0Zl5zkPYi682Lk9xTWddtaU8QCGhpO8KGlbnYuRZh0ok56HyFYzCfzEKZSl-hepBDf1pA2TGDC1n5wNGzM9nwpvCafsuSl4BQZh8N2iCygI-7iR7g__&Key-Pair-Id=K24J24Z295AEI9 HTTP/1.1" 403 Forbidden
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     message = await receive()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await fut
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e75929dde0
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     +------------------------------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     message = await receive()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await fut
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e759224d90
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     +------------------------------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     message = await receive()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await fut
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e7591a74c0
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     +------------------------------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 265, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await wrap(partial(self.listen_for_disconnect, receive))
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     message = await receive()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await self.message_event.wait()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     await fut
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | asyncio.exceptions.CancelledError: Cancelled by cancel scope 76e7591f4bb0
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr | During handling of the above exception, another exception occurred:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   + Exception Group Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     result = await app(  # type: ignore[func-returns-value]
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     return await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await super().__call__(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, _send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.middleware_stack(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await route.handle(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await self.app(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise exc
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await app(scope, receive, sender)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 75, in app
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     await response(scope, receive, send)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 258, in __call__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     async with anyio.create_task_group() as task_group:
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |   File "/data/olah/.venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   |     raise BaseExceptionGroup(
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   | exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |   +-+---------------- 1 ----------------
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | Traceback (most recent call last):
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 261, in wrap
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await func()
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/responses.py", line 255, in stream_response
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send({"type": "http.response.body", "body": b"", "more_body": False})
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 50, in sender
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 161, in _send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     await send(message)
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 534, in send
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     |     raise RuntimeError("Response content shorter than Content-Length")
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     | RuntimeError: Response content shorter than Content-Length
Aug 08 11:18:55 node1 python[1825074]: 2024-08-08 11:18:55 | ERROR | stderr |     +------------------------------------
Aug 08 11:20:10 node1 python[1825074]: 2024-08-08 11:20:10 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/datasets/Salesforce/wikitext/resolve/main/.gitattributes "HTTP/1.1 200 OK"
(.venv) root@node1:~#

@jstzwj
Copy link
Contributor

jstzwj commented Aug 9, 2024

I don't know the reason for sure. It happens occasionally when I pull very large repository.
My guess is that 1) it could be a network issue. I split a whole file into many chunks and request different ranges from huggingface. Too many requests caused a disconnect.
2) Too large cache block setting causing a timeout.
I will try to address the these points to see if it can reduce the likelihood of this issue occurring.

@andy108369
Copy link
Author

I don't know the reason for sure. It happens occasionally when I pull very large repository. My guess is that 1) it could be a network issue. I split a whole file into many chunks and request different ranges from huggingface. Too many requests caused a disconnect. 2) Too large cache block setting causing a timeout. I will try to address the these points to see if it can reduce the likelihood of this issue occurring.

Thank you @jstzwj , I'll be looking forward to test the new version.
It shouldn't be a networking issue since I'm always able to download this model directly.
And it fails pretty much immediately. I've retried it multiple times.

You can also just try exact same model https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 -- it is gated but you can request the access to it.
If you want, I can give you my HF_TOKEN to test in case if you can't get the access approved for any reason.

@jstzwj
Copy link
Contributor

jstzwj commented Aug 10, 2024

I just attempted to optimize and fix some bugs in the caching. The new commit resolved the RuntimeError: Response content shorter than Content-Length error in my development environment.
I am not sure if this commit will work for you.
Now, I am testing Meta-Llama-3.1-405B-Instruct-FP8, which is time-consuming. I need to clone multiple times (with and without caching) to compare if the binaries are consistent.

@andy108369
Copy link
Author

andy108369 commented Aug 14, 2024 •
edited
Loading

Hey @jstzwj , I figured that I have been getting 403 errors due to something stale under my --repos-path (/data/olah/hf_mirrors).
After rm -rf /data/olah/hf_mirrors/* -- it worked flawlessly!
Have tested 33e7cf1 version.

I think you might want to make some improvement to the olah so it doesn't require a user to manually clear the --repos-path as it may lead to confusing HTTP 403 errors.

Thank you!

Update
I am still downloading the dataset, it already downloaded 49Gi out of 462Gi. Will post in case of issues.

@jstzwj
Copy link
Contributor

jstzwj commented Aug 14, 2024

When downloading files from huggingface's LFS, a 302 redirect occurs from huggingface.co to the cdn-lfs.huggingface.co. The ./hf_mirrors/heads directory stores records of HTTP redirects for the mirror site. The redirect location generated by older versions Olah is incorrect, resulting in HTTP 403 errors.

@andy108369
Copy link
Author

When downloading files from huggingface's LFS, a 302 redirect occurs from huggingface.co to the cdn-lfs.huggingface.co. The ./hf_mirrors/heads directory stores records of HTTP redirects for the mirror site. The redirect location generated by older versions Olah is incorrect, resulting in HTTP 403 errors.

It would be great if olah could automatically clear the data in hf_mirrors/heads. I only realized to clear this directory (and actually the entire hf_mirrors directory) by a stroke of luck. :-)

@andy108369
Copy link
Author

Hey @jstzwj , we are constantly getting a 500 error as we are trying to use vllm serve ... to download the gated meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 model:

vllm command is vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tensor-parallel-size 8 --gpu-memory-utilization 1 --max-model-len=32768 --enable-chunked-prefill=False ; after the exported HF_TOKEN of course for this gated model.

olah errors

Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/7c365a34d0fed5b4afb118cf253395f47f6efa72/added_tokens.json "HTTP/1.1 404 Not Found"
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | INFO | stdout | INFO:     38.128.232.161:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/main/added_tokens.json HTTP/1.0" 500 Internal Server Error
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr | ERROR:    Exception in ASGI application
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr | Traceback (most recent call last):
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     result = await app(  # type: ignore[func-returns-value]
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     return await self.app(scope, receive, send)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     await super().__call__(scope, receive, send)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     raise exc
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     await self.app(scope, receive, _send)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     raise exc
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     await app(scope, receive, sender)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     await route.handle(scope, receive, send)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     await self.app(scope, receive, send)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     raise exc
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     await app(scope, receive, sender)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     response = await func(request)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     raw_response = await run_endpoint_function(
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/.venv/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     return await dependant.call(**values)
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/olah/olah/server.py", line 322, in file_head2
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     return await file_head_common(
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/olah/olah/server.py", line 287, in file_head_common
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     status_code = await generator.__anext__()
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/olah/olah/proxy/files.py", line 462, in _file_realtime_stream
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     status_code, head_info, content = await _file_full_header(
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |   File "/data/olah/olah/olah/proxy/files.py", line 198, in _file_full_header
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr |     raise Exception(
Aug 15 12:34:12 node1 python[2995464]: 2024-08-15 12:34:12 | ERROR | stderr | Exception: Unexpected HTTP status code 404

root@node1:~# curl -H "Authorization: Bearer ${HF_TOKEN}" -LI http://node1.h100.mon.obl.akash.pub:8090/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/main/added_tokens.json
HTTP/1.1 500 Internal Server Error
Server: nginx/1.18.0 (Ubuntu)
Date: Thu, 15 Aug 2024 12:34:12 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 21
Connection: keep-alive

oot@node1:~# curl -H "Authorization: Bearer ${HF_TOKEN}" -LI http://node1.h100.mon.obl.akash.pub:8090/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/main/config.json
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Thu, 15 Aug 2024 12:34:20 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 22530
Connection: keep-alive
x-powered-by: huggingface-moon
cross-origin-opener-policy: same-origin
referrer-policy: strict-origin-when-cross-origin
x-request-id: Root=1-66bca7e3-1bb680aa2a52cee41f24a295;59d26fd4-abac-45b4-b89b-adb3ba28d474
access-control-allow-origin: https://huggingface.co/
vary: Origin
access-control-expose-headers: X-Repo-Commit,X-Request-Id,X-Error-Code,X-Error-Message,X-Total-Count,ETag,Link,Accept-Ranges,Content-Range
x-repo-commit: 7c365a34d0fed5b4afb118cf253395f47f6efa72
accept-ranges: bytes
content-disposition: inline; filename*=UTF-8''config.json; filename="config.json";
content-security-policy: default-src none; sandbox
etag: "ed166d7b95e1910b9e567693971c6f220cd68915"
x-cache: Miss from cloudfront
via: 1.1 905aa3bc80ce385e5945d99189fc1eac.cloudfront.net (CloudFront)
x-amz-cf-pop: YUL62-P1
x-amz-cf-id: czkFSggQcrWXoZJUB4haieMrt2T-jrmmKO58-YW75cSvGbIiT7zWTg==

root@node1:~# journalctl -u olah-server.service -f --no-pager -l -n 10
Aug 15 12:43:38 node1 python[3015924]: 2024-08-15 12:43:38 | INFO | stdout | INFO:     38.128.233.166:0 - "GET /api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 HTTP/1.0" 404 Not Found
Aug 15 12:44:45 node1 python[3015924]: 2024-08-15 12:44:45 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 15 12:44:45 node1 python[3015924]: 2024-08-15 12:44:45 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 15 12:44:45 node1 python[3015924]: 2024-08-15 12:44:45 | INFO | stdout | INFO:     38.128.232.161:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/main/added_tokens.json HTTP/1.0" 404 Not Found
Aug 15 12:44:49 node1 python[3015924]: 2024-08-15 12:44:49 | INFO | httpx | HTTP Request: HEAD https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 15 12:44:49 node1 python[3015924]: 2024-08-15 12:44:49 | INFO | httpx | HTTP Request: GET https://huggingface.co/api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/revision/main "HTTP/1.1 200 OK"
Aug 15 12:44:49 node1 python[3015924]: 2024-08-15 12:44:49 | INFO | stdout | INFO:     38.128.232.161:0 - "HEAD /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/main/config.json HTTP/1.0" 200 OK
Aug 15 12:44:58 node1 python[3015924]: 2024-08-15 12:44:58 | INFO | stdout | INFO:     38.128.232.161:0 - "HEAD /api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 HTTP/1.0" 404 Not Found
Aug 15 12:45:28 node1 python[3015924]: 2024-08-15 12:45:28 | INFO | stdout | INFO:     38.128.232.161:0 - "HEAD /api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 HTTP/1.0" 404 Not Found
Aug 15 12:46:07 node1 python[3015924]: 2024-08-15 12:46:07 | INFO | stdout | INFO:     38.128.232.161:0 - "HEAD /api/models/meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 HTTP/1.0" 404 Not Found

@jstzwj
Copy link
Contributor

jstzwj commented Aug 15, 2024

I realized that I have not implemented the method "/api/{repo_type}/{org_repo}/tree/{commit}", which led to some issues. However, I am unsure why vllm requests /meta-llama/Meta-Llama-3.1-405B-Instruct-FP8/resolve/main/added_tokens.json. I think there is no such file in this repository.
Therefore, in the latest commit, I have retained the 404 status and forwarded it to the client.

@jstzwj
Copy link
Contributor

jstzwj commented Aug 15, 2024

I just pushed a new commit to address this issue, which includes changes to the format of caching. Therefore, when using the latest commit (5b74eba), please delete the hf_mirrors/api folder.

@andy108369
Copy link
Author

I just pushed a new commit to address this issue, which includes changes to the format of caching. Therefore, when using the latest commit (5b74eba), please delete the hf_mirrors/api folder.

great job @jstzwj !
it seem to be working, I haven't finished waiting for the vllm to fully download it, but it's been up for at least ~6 minutes downloading the model without any issues.
I'll run more tests next week and will report here.

Thank you for your prompt responses and lightning fast development! 🦾

@jstzwj jstzwj added the bug Something isn't working label Nov 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jstzwj @andy108369