Connectivity Error

[rosiecakes5]

Description

Chat stopped responding, followed by the following error pop up:

Reproduction

This bug occurred while using the app; there was no identified trigger

Expected behavior

The chat would continue to respond

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct and Contributing Guidelines

[philippzagar]

Thanks for raising the issue @rosiecakes5! 🚀

This seems like a pretty regular network layer timeout to me, meaning the transport threw an error. Tbh, not that unlikely with all the different inference requests while summarizing individual FHIR resources.
I'll investigate further to see if we have any actual error here. We probably should implement a retry mechanism in case the network request fails / times out.

[PSchmiedmayer]

I am wondering if this is due to the face that we have a lot of parallel function calls; could be that some dispatching of them is failing due to that parallel nature? And agree, maybe a small retry (one time) on a few HTTP codes that indicate that would make sense; could be configurable in the LLM Schema definition and could have a default value.

[philippzagar]

@PSchmiedmayer Totally agreed, as already mentioned above, the large amount of individual parallel function calls might lead to that. The LLMOpenAIPlatform however offers an easy way to constrain the maximal amount of parallel LLM inferences, so we might be able to mitigate that by reducing that number, of course at the cost of overall response time from the model.

In general, a small retry policy in the case of a transport error (such as timeouts) would be a great thing in SpeziLLM. Not sure if I would actually extend that to HTTP response codes, but there might be some use cases where that makes sense.
Here's the issue: StanfordSpezi/SpeziLLM#107