Support AWS plugin for TTS, STT and LLM #1302

jayeshp19 · 2024-12-26T20:22:29Z

This PR implements AWS plugin for TTS and STT

…-stt

changeset-bot · 2024-12-26T20:22:33Z

🦋 Changeset detected

Latest commit: c2e1384

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
livekit-plugins-aws	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

livekit-plugins/install_local.sh

theomonnom · 2025-01-13T12:21:18Z

livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/_utils.py

+    return credentials.access_key, credentials.secret_key
+
+
+TTS_SPEECH_ENGINE = Literal["standard", "neural", "long-form", "generative"]


We should move this to another file. Check how we do it for other TTS/STT

theomonnom · 2025-01-13T12:21:54Z

livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/tts.py

+
+                response = await client.synthesize_speech(**_strip_nones(params))
+
+                if "AudioStream" in response:


nit avoid the extra indent here

theomonnom · 2025-01-13T12:23:03Z

livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/stt.py

+                except Exception as e:
+                    logger.exception(f"an error occurred while streaming inputs: {e}")
+
+            handler = TranscriptEventHandler(stream.output_stream, self._event_ch)


Why do we create a separate class?

theomonnom · 2025-01-13T12:24:47Z

livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/tts.py

+        self,
+        *,
+        voice: str | None = "Ruth",
+        language: TTS_LANGUAGE | None = None,


Suggested change

language: TTS_LANGUAGE | None = None,

language: TTS_LANGUAGE | None = None,

We should always allow a str too here, we can't guarantee we will update the languages quickly

theomonnom · 2025-01-13T12:25:05Z

livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/tts.py

+        *,
+        voice: str | None = "Ruth",
+        language: TTS_LANGUAGE | None = None,
+        output_format: TTS_OUTPUT_FORMAT = "pcm",


I don't think it makes sense to expose the output format. we only support pcm

we do support mp3

…-stt

theomonnom · 2025-01-24T09:54:04Z

livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/_utils.py

+
+    # If API key and secret are provided, create a session with them
+    if api_key and api_secret:
+        session = boto3.Session(


Is this making network calls?

boto3.Session() doesn’t make network calls, but session.get_credentials() does if API keys and secrets aren’t cached., but we’re calling it during initialization, it’s a one-time operation.

theomonnom · 2025-01-24T09:54:59Z

livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/stt.py

+                try:
+                    async for frame in self._input_ch:
+                        if isinstance(frame, rtc.AudioFrame):
+                            await stream.input_stream.send_audio_event(
+                                audio_chunk=frame.data.tobytes()
+                            )
+                    await stream.input_stream.end_stream()
+
+                except Exception as e:
+                    logger.exception(f"an error occurred while streaming inputs: {e}")
+
+            async def handle_transcript_events():
+                try:
+                    async for event in stream.output_stream:
+                        if isinstance(event, TranscriptEvent):
+                            self._process_transcript_event(event)
+                except Exception as e:
+                    logger.exception(
+                        f"An error occurred while handling transcript events: {e}"
+                    )


Instead of using try — finally. We have an utility for it here

theomonnom · 2025-01-24T09:55:28Z

livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/stt.py

+            finally:
+                await utils.aio.gracefully_cancel(*tasks)
+        except Exception as e:
+            logger.exception(f"An error occurred while streaming inputs: {e}")


I think this is swallowing exceptions? In this case the baseclass will not try to reconnect on failure

theomonnom · 2025-01-24T09:57:43Z

livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/tts.py

+    def get_client(self):
+        """Returns a client creator context."""
+        return self._session.create_client(
+            "polly",
+            region_name=self._opts.speech_region,
+            aws_access_key_id=self._api_key,
+            aws_secret_access_key=self._api_secret,
+        )


Should we hide this?

theomonnom · 2025-01-24T09:58:00Z

livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/tts.py

+from typing import Any, Callable
+
+import aiohttp
+from aiobotocore.session import AioSession, get_session  # type: ignore


They don't support types?

They don't have official support for types. I found this thread boto/boto3#2213 and this library https://github.com/youtype should we include it?

jenfic · 2025-02-05T18:43:41Z

Getting the following error when calling a function:
ValidationException:http://internal.amazon.com/coral/com.amazon.bedrock/
"message":"The toolConfig field must be defined when using toolUse and toolResult content blocks."

Karan-Rajesh-Nair · 2025-02-06T13:08:48Z

Hey @jayeshp19 i was going thorugh the error and wanted to see if these solutions could help

Added imports for proper type support
from mypy_boto3_bedrock_runtime import BedrockRuntimeClient
from boto3.session import Session
Fixed the client initialization to use proper typing:
self._session: Session = Session(
aws_access_key_id=self._api_key,
aws_secret_access_key=self._api_secret,
region_name=region
)
self._client: BedrockRuntimeClient = self._session.client("bedrock-runtime")
Updated the type hint in LLMStream:
client: BedrockRuntimeClient,

and the last one for converse_stream would we be able to use invoke_model_with_response_stream in _run method.

vanics · 2025-02-06T18:27:52Z

Good work so far. I am looking forward to this one.

sunilvb · 2025-02-06T22:10:37Z

Just tested this PR. Overall, amazing speeds when using aws.LLM(), aws.TTS(), and aws.STT() !!

Here are a couple of errors from amazon_transcribe:

2025-02-06 17:05:51,236 - ERROR livekit.plugins.aws - Error in handle_transcript_events
Traceback (most recent call last):
File "..//livekit/agents/utils/log.py", line 16, in async_fn_logs
return await fn(*args, **kwargs)
File "..//livekit/plugins/aws/stt.py", line 161, in handle_transcript_events
async for event in stream.output_stream:
File "..//amazon_transcribe/eventstream.py", line 666, in aiter
parsed_event = self._parser.parse(event)
File "..//amazon_transcribe/deserialize.py", line 161, in parse
raise self._parse_event_exception(raw_event)
amazon_transcribe.exceptions.BadRequestException: Your request timed out because no new audio was received for 15 seconds. {"pid": 71376, "job_id": "AJ_ctSPy6HPRp57"}
2025-02-06 17:05:51,239 - ERROR livekit.agents.pipeline - Error in _recognize_task
Traceback (most recent call last):
File "..//livekit/agents/utils/log.py", line 16, in async_fn_logs
return await fn(*args, **kwargs)
File "..//livekit/agents/pipeline/human_input.py", line 150, in _recognize_task
await asyncio.gather(*tasks)
File "..//livekit/agents/pipeline/human_input.py", line 136, in _stt_stream_co
async for ev in stt_stream:
File "..//livekit/agents/stt/stt.py", line 321, in anext
raise exc from None
File "..//livekit/agents/stt/stt.py", line 219, in _main_task
return await self._run()
File "..//livekit/plugins/aws/stt.py", line 170, in _run
await asyncio.gather(*tasks)
File "..//livekit/agents/utils/log.py", line 16, in async_fn_logs
return await fn(*args, **kwargs)
File "..//livekit/plugins/aws/stt.py", line 161, in handle_transcript_events
async for event in stream.output_stream:
File "..//amazon_transcribe/eventstream.py", line 666, in aiter
parsed_event = self._parser.parse(event)
File "..//amazon_transcribe/deserialize.py", line 161, in parse
raise self._parse_event_exception(raw_event)
amazon_transcribe.exceptions.BadRequestException: Your request timed out because no new audio was received for 15 seconds. {"pid": 71376, "job_id": "AJ_ctSPy6HPRp57"}

sunilvb · 2025-02-06T22:14:19Z

Also, seeing the same error as @jenfic when using the function calls with aws.LLM():

ValidationException:http://internal.amazon.com/coral/com.amazon.bedrock/
"message":"The toolConfig field must be defined when using toolUse and toolResult content blocks."

Not sure if pytest could/should catch these errors.

sunilvb · 2025-02-06T22:16:55Z

@jayeshp19 let me know if you need further help testing. I can help spin-up all the services needed in AWS... (wink, wink).

meetakshay99 · 2025-02-07T13:48:24Z

I tried using this AWS plugin for TTS but could not hear any audio.
I am using it with PipelineAgent and could see this printed to logs

2025-02-07 13:43:07,366 - DEBUG livekit.agents.pipeline - speech playout started {"speech_id": "112916673385", "pid": 358, "job_id": "AJ_xSYnj6Kpfpbk"}
2025-02-07 13:43:07,929 - DEBUG livekit.agents.pipeline - speech playout finished {"speech_id": "112916673385", "interrupted": false, "pid": 358, "job_id": "AJ_xSYnj6Kpfpbk"}
2025-02-07 13:43:07,930 - DEBUG livekit.agents.pipeline - committed agent speech {"agent_transcript": "today", "interrupted": false, "speech_id": "112916673385", "pid": 358, "job_id": "AJ_xSYnj6Kpfpbk"}

but did not hear any audio.

Below are the details sent to aws while creating the TTS object:
{'api_key': 'my_api_key', 'api_secret': 'my_secret_key', 'speech_region': 'eu-west-1', 'voice': 'Matthew', 'speech_engine': 'neural'}
What may I be doing wrong?

…-stt

dasxran and others added 15 commits July 14, 2024 14:26

base files for AWS plugins

23efbf4

aws workflow

70fcfc3

rename aws folder

fa9a9f5

Polly TTS

7282f51

Update test_tts with aws

7d07511

Merge branch 'livekit:main' into dex/aws

485e410

Merge branch 'livekit:main' into dex/aws

c1676ed

aws transcribe

b19b2c8

livekit-agents v0.8.0

32e8f32

Merge remote-tracking branch 'dasxran/dex/aws' into aws-tts-stt

0d40699

ruff check & yml changes

8d7210f

setup changes

0d84256

Merge branch 'main' of https://github.com/livekit/agents into aws-tts…

9c0c373

…-stt

Merge branch 'main' of https://github.com/livekit/agents into aws-tts…

b2b5614

…-stt

updates

c2d26fc

jayeshp19 added 4 commits December 27, 2024 02:42

updates

ca7d609

updates

84741d9

updates

660d783

updates

81e2ebd

theomonnom reviewed Jan 13, 2025

View reviewed changes

livekit-plugins/install_local.sh Show resolved Hide resolved

updates

93f8b19

theomonnom reviewed Jan 13, 2025

View reviewed changes

updates

69b286b

theomonnom reviewed Jan 13, 2025

View reviewed changes

jayeshp19 added 2 commits January 13, 2025 19:37

updates

9d52b78

updates

71f450d

Merge branch 'main' of https://github.com/livekit/agents into aws-tts…

55768ae

…-stt

jayeshp19 changed the title ~~[draft] Support AWS plugin for TTS and STT~~ Support AWS plugin for TTS and STT Jan 20, 2025

jayeshp19 marked this pull request as ready for review January 20, 2025 09:51

jayeshp19 added 5 commits January 20, 2025 15:25

changeset

5af4053

Merge branch 'main' of https://github.com/livekit/agents into aws-tts…

33c787d

…-stt

updates

23de909

updates

6b8e706

Merge branch 'main' of https://github.com/livekit/agents into aws-tts…

1b56f02

…-stt

theomonnom reviewed Jan 24, 2025

View reviewed changes

jayeshp19 added 4 commits January 25, 2025 15:51

updates

827a09c

supprot bedrock llms

fa2efdd

ruff

c735e47

updates

e8557e1

jayeshp19 changed the title ~~Support AWS plugin for TTS and STT~~ Support AWS plugin for TTS, STT and LLM Jan 27, 2025

jayeshp19 added 6 commits January 27, 2025 20:21

updates

3e4869b

updates

c46554c

updates

1a07bb6

updates

c6687d6

updates

c9dabff

update test llm

fe986ab

davidzhao mentioned this pull request Feb 7, 2025

plugins: add AWS Transcribe and Polly TTS plugin #453

Closed

jayeshp19 added 2 commits February 10, 2025 16:57

Merge branch 'main' of https://github.com/livekit/agents into aws-tts…

adf5a91

…-stt

updates

c2e1384

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support AWS plugin for TTS, STT and LLM #1302

Support AWS plugin for TTS, STT and LLM #1302

jayeshp19 commented Dec 26, 2024 •

edited

Loading

changeset-bot bot commented Dec 26, 2024 •

edited

Loading

theomonnom Jan 13, 2025 •

edited

Loading

theomonnom Jan 13, 2025

theomonnom Jan 13, 2025

theomonnom Jan 13, 2025

theomonnom Jan 13, 2025

jayeshp19 Jan 13, 2025

theomonnom Jan 13, 2025

theomonnom Jan 24, 2025

jayeshp19 Jan 24, 2025

theomonnom Jan 24, 2025

theomonnom Jan 24, 2025

theomonnom Jan 24, 2025

theomonnom Jan 24, 2025

jayeshp19 Jan 27, 2025 •

edited

Loading

jenfic commented Feb 5, 2025

Karan-Rajesh-Nair commented Feb 6, 2025 •

edited

Loading

vanics commented Feb 6, 2025

sunilvb commented Feb 6, 2025 •

edited

Loading

sunilvb commented Feb 6, 2025

sunilvb commented Feb 6, 2025

meetakshay99 commented Feb 7, 2025 •

edited

Loading

		return credentials.access_key, credentials.secret_key


		TTS_SPEECH_ENGINE = Literal["standard", "neural", "long-form", "generative"]


		response = await client.synthesize_speech(**_strip_nones(params))

		if "AudioStream" in response:

	language: TTS_LANGUAGE \| None = None,
	language: TTS_LANGUAGE \| None = None,

Support AWS plugin for TTS, STT and LLM #1302

Are you sure you want to change the base?

Support AWS plugin for TTS, STT and LLM #1302

Conversation

jayeshp19 commented Dec 26, 2024 • edited Loading

changeset-bot bot commented Dec 26, 2024 • edited Loading

🦋 Changeset detected

theomonnom Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayeshp19 Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

jenfic commented Feb 5, 2025

Karan-Rajesh-Nair commented Feb 6, 2025 • edited Loading

vanics commented Feb 6, 2025

sunilvb commented Feb 6, 2025 • edited Loading

sunilvb commented Feb 6, 2025

sunilvb commented Feb 6, 2025

meetakshay99 commented Feb 7, 2025 • edited Loading

jayeshp19 commented Dec 26, 2024 •

edited

Loading

changeset-bot bot commented Dec 26, 2024 •

edited

Loading

theomonnom Jan 13, 2025 •

edited

Loading

jayeshp19 Jan 27, 2025 •

edited

Loading

Karan-Rajesh-Nair commented Feb 6, 2025 •

edited

Loading

sunilvb commented Feb 6, 2025 •

edited

Loading

meetakshay99 commented Feb 7, 2025 •

edited

Loading