[BUG] ./opensearch-docker-entrypoint.sh: Killed when pushing document

[Sinan-Karakaya]

Describe the bug

I run a FastAPI server that calls an opensearch container that is used for RAG. When I start pushing the data through the use of opensearch-py, I get this error without any other log indicating that an error happened:

[2024-03-14T15:37:14,180][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [2e6e2506c04f] Starting ML sync up job...
[2024-03-14T15:37:24,222][INFO ][o.o.m.c.MLSyncUpCron     ] [2e6e2506c04f] ML configuration already initialized, no action needed
[2024-03-14T15:37:41,555][INFO ][o.o.p.PluginsService     ] [2e6e2506c04f] PluginService:onIndexModule index:[kuzco/WLWbX0uVQYGsSiuRgNq25A]
[2024-03-14T15:37:41,574][INFO ][o.o.c.m.MetadataCreateIndexService] [2e6e2506c04f] [kuzco] creating index, cause [api], templates [], shards [1]/[0]
[2024-03-14T15:37:41,600][INFO ][o.o.p.PluginsService     ] [2e6e2506c04f] PluginService:onIndexModule index:[kuzco/WLWbX0uVQYGsSiuRgNq25A]
[2024-03-14T15:37:41,623][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [2e6e2506c04f] Detected cluster change event for destination migration
[2024-03-14T15:37:41,649][INFO ][o.o.c.r.a.AllocationService] [2e6e2506c04f] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[kuzco][0]]]).
[2024-03-14T15:37:41,662][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [2e6e2506c04f] Detected cluster change event for destination migration
/usr/share/opensearch/./opensearch-docker-entrypoint.sh: line 69:    29 Killed                  "$@" "${opensearch_opts[@]}"

Here is the error on the side of the python server (sorry for the formatting, that's the doing of the logger library...):

{"version": "1.1", "short_message": "Creating index...", "timestamp": 1710430661.5429306, "level": 6, "host": "337681eb990f"}
{"version": "1.1", "short_message": "Index created", "timestamp": 1710430661.6673136, "level": 6, "host": "337681eb990f"}
{"version": "1.1", "short_message": "Fetching data...", "timestamp": 1710430661.6673856, "level": 6, "host": "337681eb990f"}
{"version": "1.1", "short_message": "Data fetched", "timestamp": 1710430661.8305678, "level": 6, "host": "337681eb990f"}
{"version": "1.1", "short_message": "Pushing data, don't panic if this takes a long time...", "timestamp": 1710430661.830729, "level": 6, "host": "337681eb990f"}
{"version": "1.1", "short_message": "Error pushing data", "timestamp": 1710430672.223734, "level": 3, "host": "337681eb990f", "full_message": "Traceback (most recent call last):\n  File \"/usr/lib/python3.11/site-packages/urllib3/connection.py\", line 198, in _new_conn\n    sock = connection.create_connection(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/urllib3/util/connection.py\", line 60, in create_connection\n    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/socket.py\", line 962, in getaddrinfo\n    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name does not resolve\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/lib/python3.11/site-packages/opensearchpy/connection/http_urllib3.py\", line 264, in perform_request\n    response = self.pool.urlopen(\n               ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 847, in urlopen\n    retries = retries.increment(\n              ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/urllib3/util/retry.py\", line 445, in increment\n    raise reraise(type(error), error, _stacktrace)\n          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/urllib3/util/util.py\", line 39, in reraise\n    raise value\n  File \"/usr/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 793, in urlopen\n    response = self._make_request(\n               ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 496, in _make_request\n    conn.request(\n  File \"/usr/lib/python3.11/site-packages/urllib3/connection.py\", line 400, in request\n    self.endheaders()\n  File \"/usr/lib/python3.11/http/client.py\", line 1293, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/usr/lib/python3.11/http/client.py\", line 1052, in _send_output\n    self.send(msg)\n  File \"/usr/lib/python3.11/http/client.py\", line 990, in send\n    self.connect()\n  File \"/usr/lib/python3.11/site-packages/urllib3/connection.py\", line 238, in connect\n    self.sock = self._new_conn()\n                ^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/urllib3/connection.py\", line 205, in _new_conn\n    raise NameResolutionError(self.host, self, e) from e\nurllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPConnection object at 0xffff73d728d0>: Failed to resolve 'kuzco-opensearch' ([Errno -2] Name does not resolve)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/app/main/rag.py\", line 75, in recreate\n    os_client.index(index=OPENSEARCH_INDEX_NAME, body=document, refresh=True)\n  File \"/usr/lib/python3.11/site-packages/opensearchpy/client/utils.py\", line 181, in _wrapped\n    return func(*args, params=params, headers=headers, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/opensearchpy/client/__init__.py\", line 402, in index\n    return self.transport.perform_request(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/opensearchpy/transport.py\", line 446, in perform_request\n    raise e\n  File \"/usr/lib/python3.11/site-packages/opensearchpy/transport.py\", line 409, in perform_request\n    status, headers_response, data = connection.perform_request(\n                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/opensearchpy/connection/http_urllib3.py\", line 279, in perform_request\n    raise ConnectionError(\"N/A\", str(e), e)\nopensearchpy.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0xffff73d728d0>: Failed to resolve 'kuzco-opensearch' ([Errno -2] Name does not resolve)) caused by: NameResolutionError(<urllib3.connection.HTTPConnection object at 0xffff73d728d0>: Failed to resolve 'kuzco-opensearch' ([Errno -2] Name does not resolve))"}

Here is the docker compose that I use + the minimal code:

version: '3'

services:
  fastapi-server:
    container_name: fastapi-server
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 8080:8080
    env_file:
      - path: .env
        required: true

  node-opensearch:
    container_name: node-opensearch
    image: opensearchproject/opensearch:2.12.0
    env_file:
      - path: .env
        required: true
    environment:
      - discovery.type=single-node
      - plugins.security.disabled=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200
      - 9600:9600

  ollama:
    container_name: ollama
    image: ollama/ollama:latest
    ports:
      - '11434:11434'

os_client = OpenSearch(
    hosts=[OPENSEARCH_URL],
    http_auth=(OPENSEARCH_USERNAME, OPENSEARCH_PASSWORD),
    use_ssl=False,
    verify_certs=False,
)
INDEX_SETTINGS = {
    "settings": {
        "index": {"knn": True},
        "number_of_shards": 1,
        "number_of_replicas": 0,
    },
    "mappings": {
        "properties": {
            "vector_field": {
                "type": "knn_vector",
                "dimension": 4096,
                "method": {"name": "hnsw", "space_type": "l2", "engine": "faiss"},
            }
        }
    },
}

if os_client.indices.exists(index=OPENSEARCH_INDEX_NAME):
        logger.info("Index already exists: deleting...")
        os_client.indices.delete(index=OPENSEARCH_INDEX_NAME)
    logger.info("Creating index...")
    os_client.indices.create(index=OPENSEARCH_INDEX_NAME, body=INDEX_SETTINGS)
    logger.info("Index created")

try:
     res = embedding_function.embed_query(thematic)
     text = {
          "thematic": thematic,
          "extensions": thematics[thematic],
       }
       document = {
             "vector_field": res,
              "text": str(text),
              "thematic": thematic,
              "extensions": thematics[thematic],
        }
        os_client.index(index=OPENSEARCH_INDEX_NAME, body=document, refresh=True)
except Exception as e:
    logger.error("Error pushing data", exc_info=e)

Related component

Other

To Reproduce

1. Create the containers
2. Try to execute the following code
3. Errors appears and the docker get killed

Expected behavior

The documents should be pushed normally and without errors.

Additional Details

Plugins
Security plugins disabled
Using opensearch-py

Host/Environment (please complete the following information):

• OS: MacOS Sonoma 14.3 - M1 chip
• python 3.12.2

[dblock]

I'll move this to the devops repo, but what you need to get is the full server-side log from the OpenSearch instance.

[rishabh6788]

@Sinan-Karakaya Please share the server-side logs to debug this further.

[bshien]

Closing for now due to no activity for a few months, feel free to reopen