Add vector search documentation

[kolchfa-aws]

Adds a vector search section

Checklist

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
  For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

[naveentatikonda]

Faiss HNSW with PQ also requires training

[naveentatikonda]

Need to add a card for binary vectors along with byte vectors
https://opensearch.org/docs/latest/field-types/supported-field-types/knn-vector#binary-vectors

[kolchfa-aws]

@naveentatikonda Thanks! I addressed both comments. Could you review this commit a5e8b8d.

[naveentatikonda]

@kolchfa-aws changes looks good. Thanks for making those changes.

For binary vectors, we also need to add memory estimation. Here is the formula for HNSW
1.1 * (dimension / 8 + 8 * M) bytes/vector

For IVF, I guess it is 1.1 * (((dimension / 8) * num_vectors) + (nlist * dimension / 8)). @jmazanec15 can you pls confirm ?

[jmazanec15]

Yeah that looks good to me

[kolchfa-aws]

Added both formulas.

[jmazanec15]

Hi @kolchfa-aws, this looks awesome!

In general, I think we should start moving the more low-level/expert details (like quantization and method configuration), out of vector search section and into detailed field reference section. Here is some high level feedback:

1. Itd be good to showcase/highlight some high level features on the vector search splash page (https://kolchfa-aws.github.io/vector-search). In particular: filtering, multi-vector per document support (nested), automatic embedding generation, low-memory search, and hybrid search
2. For vector search techniques, can we add sections on sparse vectors and hybrid search?
3. Can we add knn query type in query dsl?
4. Can we have a page for space types and only maintain one table - possibly in field reference? Also, can we move engine/method details into field reference and each engine have its own detailed section in mapping reference.
5. For examples/tutorials, can we use basic mapping instead of complex mapping:

PUT /test-index
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vector1": {
        "type": "knn_vector",
        "dimension": 3,
        "space_type": "l2"
      }
    }
  }
}

In performance tuning, we can mention picking a specific engine or specifying overriding method parameters for expert level fine tuning and point to reference docs.
6. For quantization, can we put low level details into an expert section? Instead, can we just mention in performance tuning for memory optimization, we use quantization to achieve 32x, 16x, 8x, 4x, 2x compression_levels. I think we can even renaming the disk-based section, memory optimized vector search. This is example mapping:

PUT my-vector-index
{
  "settings" : {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "my_vector_field": {
        "type": "knn_vector",
        "dimension": 8,
        "space_type": "innerproduct",
        "data_type": "float",
        "mode": "on_disk",
        "compression_level": "16x"
      }
    }
  }
}

Quantization is a bit ugly for users to have to understand. So I think its better to belong in detailed field reference. We can say, for further fine tuning of the quantization methods, see field reference.
7. Can we move SIMD optimization section to performance tuning: https://kolchfa-aws.github.io/vector-search/creating-vector-index/vector-field/#simd-optimization-for-the-faiss-engine? End users dont really need to know about this when reading about vector data types
8. In optimizing vector search performance, can we add page on cluster sizing? Basically, this should point to memory evaluation formulas and picking node configurations.

[kolchfa-aws]

Suggested change

	`name` | Yes | N/A | No | The nearest neighbor method. Valid values are `hnsw` and `ivf`. Not every engine combination supports each of the methods. For a list of supported methods, see the specific engine section.
	`name` | Yes | N/A | No | The nearest neighbor method. Valid values are `hnsw` and `ivf`. Not every engine combination supports each of the methods. For a list of supported methods, see the section for a specific engine.

[kolchfa-aws]

Suggested change

`space_type` | No | `l2` | No | The vector space used to calculate the distance between vectors. Valid values are `l1`, `l2`, `linf`, `cosinesimil`, `innerproduct`, `hamming`, and `hammingbit`. Not every method/engine combination supports each of the spaces. For a list of supported spaces, see the specific engine section. Note: This value can also be specified at the top level of the mapping. For more information, see [Spaces]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-spaces/).

`space_type` | No | `l2` | No | The vector space used to calculate the distance between vectors. Valid values are `l1`, `l2`, `linf`, `cosinesimil`, `innerproduct`, `hamming`, and `hammingbit`. Not every method/engine combination supports each of the spaces. For a list of supported spaces, see the section for a specific engine. Note: This value can also be specified at the top level of the mapping. For more information, see [Spaces]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-spaces/).

[kolchfa-aws]

Suggested change

	`parameters` | No | `null` | No | The parameters used for the nearest neighbor method. For more information, see the specific engine section.
	`parameters` | No | `null` | No | The parameters used for the nearest neighbor method. For more information, see the section for a specific engine.

[kolchfa-aws]

Suggested change

	`encoder` | No | flat | No | Encoder definition for encoding vectors.
	`encoder` | No | flat | No | An encoder definition for encoding vectors.

[kolchfa-aws]

Suggested change

	`m` | No | 16 | No | The number of bidirectional links created for each new element. Impacts memory consumption significantly. Keep between 2 and 100.
	`m` | No | 16 | No | The number of bidirectional links created for each new element. Impacts memory consumption significantly. Keep between `2` and `100`.

[kolchfa-aws]

Suggested change

	There are several options to choose from when building your `knn_vector` field. To determine the correct methods and parameters, you should first understand the requirements of your workload and what trade-offs you are willing to make. Factors to consider are (1) query latency, (2) query quality, (3) memory limits, and (4) indexing latency.
	There are several options to choose from when building your `knn_vector` field. To select the correct method and parameters, you should first understand the requirements of your workload and what trade-offs you are willing to make. Factors to consider are (1) query latency, (2) query quality, (3) memory limits, and (4) indexing latency.

[kolchfa-aws]

Suggested change

	Having a replica doubles the total number of vectors.
	Using a replica doubles the total number of vectors.

[kolchfa-aws]

Suggested change

	Not every method/engine combination supports each of the spaces. For a list of supported spaces, see the specific engine section in the [method documentation]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-methods-engines/).
	Not every method/engine combination supports each of the spaces. For a list of supported spaces, see the section for a specific engine in the [method documentation]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-methods-engines/).

[kolchfa-aws]

Suggested change

`space_type` | String | The vector space used to calculate the distance between vectors. Valid values are `l1`, `l2`, `linf`, `cosinesimil`, `innerproduct`, `hamming`, and `hammingbit`. Not every method/engine combination supports each of the spaces. For a list of supported spaces, see the specific engine section. Note: This value can also be specified within the `method`. Optional. For more information, see [Spaces]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-spaces/).

`space_type` | String | The vector space used to calculate the distance between vectors. Valid values are `l1`, `l2`, `linf`, `cosinesimil`, `innerproduct`, `hamming`, and `hammingbit`. Not every method/engine combination supports each of the spaces. For a list of supported spaces, see the section for a specific engine. Note: This value can also be specified within the `method`. Optional. For more information, see [Spaces]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-spaces/).

[kolchfa-aws]

Suggested change

	description: "Generate embeddings outside of OpenSearch using your favorite embedding utility."
	description: "Generate embeddings outside of OpenSearch using your favorite embedding utility."

[kolchfa-aws]

Suggested change

	description: "Configure a machine learning model that will automatically generate embeddings from your text at ingest time and query time."
	description: "Configure a machine learning model that will automatically generate embeddings from your text at ingestion time and query time."

[kolchfa-aws]

Suggested change

	In OpenSearch, you can either bring your own vectors or let OpenSearch generate them automatically from your data. Automated embedding generation integrated into OpenSearch reduces data preprocessing effort at ingestion and search time.
	In OpenSearch, you can either bring your own vectors or let OpenSearch generate them automatically from your data. Letting OpenSearch automatically generate your embeddings reduces data preprocessing effort at ingestion and search time.

[kolchfa-aws]

Suggested change

	OpenSearch automatically generates vector embeddings from your data using a machine learning (ML) model.
	Use this option to let OpenSearch automatically generate vector embeddings from your data using a machine learning (ML) model.

[kolchfa-aws]

Suggested change

	description: "Use text chunking to ensure adherence to token limit for embedding models."
	description: "Use text chunking to ensure adherence to embedding model token limits."

[kolchfa-aws]

Suggested change

	* The coordinator node picks up final size number of neighbors from the neighbors returned by each shard.
	* The coordinator node selects the final `size` neighbors from the neighbors returned by each shard.

[kolchfa-aws]

Suggested change

	`knn.circuit_breaker.triggered` | Dynamic | `false` | True when memory usage exceeds the `knn.circuit_breaker.unset.percentage` value.
	`knn.circuit_breaker.triggered` | Dynamic | `false` | `true` when memory usage exceeds the `knn.circuit_breaker.unset.percentage` value.

[kolchfa-aws]

Suggested change

`knn.faiss.avx2.disabled` | Static | `false` | A static setting that specifies whether to disable the SIMD-based `libopensearchknn_faiss_avx2.so` library and load the non-optimized `libopensearchknn_faiss.so` library for the Faiss engine on machines with x64 architecture. For more information, see [SIMD optimization]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-methods-engines/#simd-optimization).

`knn.faiss.avx2.disabled` | Static | `false` | A static setting that specifies whether to disable the SIMD-based `libopensearchknn_faiss_avx2.so` library and load the non-optimized `libopensearchknn_faiss.so` library for the Faiss engine on machines with x64 architecture. For more information, see [Single Instruction Multiple Data (SIMD) optimization]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/knn-methods-engines/#simd-optimization).

[kolchfa-aws]

Suggested change

	The following table lists all available index-level k-NN settings. For information about updating these settings, see [Index-level index setting]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index-settings/#index-level-index-settings).
	The following table lists all available index-level k-NN settings. For information about updating these settings, see [Index-level index settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index-settings/#index-level-index-settings).

[kolchfa-aws]

Suggested change

	With cosine similarity, it is not valid to pass a zero vector (`[0, 0, ...]`) as input. This is because the magnitude of such a vector is 0, which raises a `divide by 0` exception in the corresponding formula. Requests containing the zero vector will be rejected, and a corresponding exception will be thrown.
	When using cosine similarity, it is not valid to pass a zero vector (`[0, 0, ...]`) as input. This is because the magnitude of such a vector is 0, which raises a `divide by 0` exception in the corresponding formula. Requests containing the zero vector will be rejected, and a corresponding exception will be thrown.