feat: loading characters from db at load and runtime (conflicts resolved)

[odilitime]

Originally #551, this is a resubmission

Relates to:

Loading Characters from db(sqlite/postgres) during runtime (via a REST API call) and
during start/load time.

Risks

Medium
None of these changes will affect existing workflow as its handled by env variables
FETCH_FROM_DB (bool) - default: null/false - For loading characters from DB
AGENT_RUNTIME_MANAGEMENT (bool) - default: null/false - For loading characters from Db at runtime

From security standpoint, all character specific secrets (EG: TWITTER_PASSWORD or OPENAI_API_KEY) is AES-256 encrypted and stored in DB

Background

For a production ready - multi-character Eliza setup, we want to load new characters in runtime via API calls.
We do not want to restart the Eliza server for each new character.

ASSUMPTION:
An api or script exists to store the characters in DB.
The api uses the same stringToUUID method to create consistent UUIDs for a character and store in DB.

What does this PR do?

• Allows loading characters from DB during start
• Allows loading characters via REST API from DB during runtime
• The characters are stored in TEXT/JSONB format - similar to the character Agent file in sqlite and postgres respectively
• Securely encrypts each character's secrets using AES-256 encryption and then stores in DB (Decrypt after fetch)
• Proper error handling for all scenarios
  • Character already loaded in runtime
  • Character does not exist in db

What kind of change is this?

This is a new Feature

Why?
Currently

1. For adding any new agent, we need to restart the Eliza server => All the agents and its clients are loaded again - All previous tweets (incase of twitter client), interactions are processed again. Any existing direct or telegram conversation is lost.
2. Not a straight-forward mechanism to add new agents - Now with a REST API - load new agents.

Why are we doing this?
To take a step towards multi-character production setup

Code Changes made and why?

1. Created a new table in postgres and sqlite - Added in the seed file/script of both DBs
   export type CharacterTable = { id: UUID; // created from stringToUUID - unique and consistent for name name: string; characterState: Character; // A JSONB or TEXT - that maps the character secretsIV?: Secrets; // Initialisation Vector for each secrets };
   Also added the above in packages/core/src/types.ts
2. in SqliteAdapter and PostgresAdapter - created a new function to load from this characters table
3. in Agents/src/index.ts ->

• Assign Directclient to a global variable - along with set and get methods. This is to allow the same direct client be used for character(agent) runtime creation
• A function loadCharactersFromDb loadCharactersFromDb(
  characterNames?: string
  ): Promise<Character[]>
  • if any characterNames argument received, it fetches only those characters from db
  • else fetches all characters
  • This handles encryption and decryption of secrets if secrets are present in a character
  • uses env.ENCRYPTION_KEY
• An express server
  • The server starts only when the env AGENT_RUNTIME_MANAGEMENT == true
  • Why not use the same express server in DirectClient? DirectClient does not have access to the DB methods and all the agent-specific handling methods
  • ENDPOINT: "/load/:agentName" METHOD: Post
  • PORT: default. - 3001, overwritten by env AGENT_PORT

4. in packages/client-direct/src/index.ts

• Added ENDPOINT: "/load/:agentName" METHOD: Post
  • This endpoint (is a proxy) that routes request to the agent server's route
  • Before proxying, checks if AGENT_RUNTIME_MANAGEMENT ==true and if agents already in runtime

5. created a file packages/core/src/crypt.ts

• This handles the encryption and decryption methods

6. created scripts that parses a character json file or all files in a folder and inserts in DB

• Location: scripts/importCharactersInDB/[postgres/sqlite]/insertInDb.js
  Requires env variable
  POSTGRES_URL= # if postgres ENCRYPTION_KEY="" INPUT_PATH=characters # the folder path to load the characters from SQLITE_DB_PATH= #if you want to change db path Default: agent/data/db.sqlite

Documentation changes needed?

Not needed necessarily.
New ENV variables and explanations are added in .env.example

Testing

I have tested the scenarios in detailed testing steps in both sqlite and postgres.

Where should a reviewer start?

• agent/src/index.js
• Line 418 and 531
  packages/client-direct/src/index.js
• Line 271

Detailed testing steps

INIT STEP:

1. creating character table(use the schema.sql or sqliteTables.ts )
2. loading characters in DB (used the above mentioned scripts in scripts/importCharactersInDB/[postgres/sqlite]/insertInDb.js -

• also added to the characters tate and trump - TWITTER_USERNAME, TWITTER_PASSWORD, TWITTER_EMAIL to its settings.secrets and twitter to client) to test secrets encryption and decryption

I have tested the following scenarios

1. Fetching from database during start

• set env FETCH_FROM_DB=true
• pnpm start
• Will function as usual
• if we want to test postgres, set env POSTGRES_URL=''

2. Loading an agent during runtime

• set env FETCH_FROM_DB=false #if true, we can't test load as all characters in db will be loaded
• set env AGENT_RUNTIME_MANAGEMENT=true
• set env ENCRYPTION_KEY= #use the same encryption key used in insertInDB.js script
• set env AGENT_PORT=4000 #can be empty if we want to pick default port 3001
  pnpm start
• This will load with the default character Eliza
• curl --location --request POST 'http://localhost:3000/load/trump' - replace port and character name if using different characters
• SUB SCENARIO 1: agent is loaded and we get success code 200
• SUB SCENARIO 2: agent is already loaded and we get error code 409
• SUB SCENARIO 3: agent does not exist in db and we get error code 404
• SUB SCENARIO 4: Error during agent load - like incorrect twitter - error code 500
• if we want to test postgres, set env POSTGRES_URL=''

[HashWarlock]

I like this a lot. In the current design, are characters able to be published to the DB through an API call? I can imagine an interesting way to leverage this is to have a published docker image and pair it with a docker compose file to load the character file into the DB on startup and then initialize the agent without needing to build a new image.

If I am going to test this and want to load a character file, can I make a call to an S3 API storing a character file and have that be fetched and loaded in the DB? Then have my command to be something likepnpm start --character=https://s3-bucket/folder/to/chillguy.character.json

[odilitime]

not yet but I like that direction. The issue with the current REST loading is that a crash will lose all your current agents and you have to know which ones to reload, so I like this DB direction. This PR is a bit over engineered, thinking about merging and then cleaning up.

[monilpat]

I would really LOVE to get this merged in post hopefully the final round of merge conflicts :)