Instructional Guide for Hybrid Search service¶

Hybrid Search combines both keyword and semantic search capabilities, allowing for more accurate and contextually relevant search results. Our Hybrid Search features allow you to easily add semantic field information: simply upload your raw data, and there's no need to calculate embeddings separately before adding them to the index.

This instructional guide offers detailed steps for deploying, using, and destroying the Hybrid Search service with search-hq CLI commands. Follow the sections below for a comprehensive walkthrough of each process. The guide is divided into three sections:

Deployment: Step-by-step instructions to deploy the Hybrid Search service.
Usage: Guidance on how to effectively use the Hybrid Search service.
Destruction: Instruction to destroy the Hybrid Search service.

Prerequisites¶

Launch an Instance from Search HQ Launch Pack¶

The Search HQ Launch Pack is an AMI available on AWS Marketplace. Before proceeding with the steps below, you need to launch an instance using this AMI. Follow the Setup Instructions to set up the instance, enabling you to use the search-hq CLI commands.

Deployment¶

Follow these steps to deploy the Hybrid Search service.

Step 1: Configure Your YAML File¶

Make sure your YAML configuration files (conf/project.yaml, conf/api.yaml, conf/hybrid_search.yaml, and conf/model.yaml) are correctly set up.

For detailed information, refer to the following instructional guides:

conf/project.yaml¶

##  You can configure the project name.
project_name: your-project-name  # Replace 'your-project-name' with the name of your project.


##  You can configure the AWS region.
aws:
  region_name: ap-northeast-1  # AWS region where resources will be deployed. Modify if needed (e.g., 'ap-northeast-1').


##  You need to use the existing VPC.
vpc:
  vpc_id: your-vpc-id  # Replace 'your-vpc-id' with the ID of your existing VPC.


##  You need to use the existing OpenSearch.
opensearch:
  domain: your-opensearch-domain  # Replace 'your-opensearch-domain' with the domain VPC endpoint for OpenSearch.
  domain_name: your-opensearch-domain-name  # Replace 'your-opensearch-domain-name' with the specific domain name.

  ##  If the fine-grained access control in your OpenSearch is enabled,
  ##  provide the necessary information in the 'auth' section as required.
  ##  By default, this setting is disabled. To enable it, uncomment the 'auth' section.
  # auth:
  #   type: aws  ##  The authentication type. Options: `aws` for AWS IAM authentication or `basic` for account and password. If `basic` is selected, you need to provide master_user_secret_arn.
  #   master_user_secret_arn: auth-arn  # The ARN of the SSM Parameter Store or AWS Secrets Manager where the master user’s account and password information is stored.

  ##  You can build OpenSearch Auto Scaling mechanism by configuring the auto_scaling section.
  ##  By default, this setting is disabled. To enable it, uncomment the 'auto_scaling' section.
  # auto_scaling:
  #   min_instance_count: 2  # Minimum number of instances for auto scaling.
  #   max_instance_count: 12  # Maximum number of instances for auto scaling.
  #   scale_down_cpu_util_threshold: 30  # CPU utilization threshold to scale down instances.
  #   scale_up_cpu_util_threshold: 80  # CPU utilization threshold to scale up instances.

conf/api.yaml¶

##  You can specify the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs.
# cloud_watch_role_name: your-cloud-watch-role-name  # Uncomment and replace 'your-cloud-watch-role-name' with your actual CloudWatch role name, if you already have one.


##  You can name custom domain name for your API.
domain_name_system:
  domain_name: your-domain-name  # Replace 'your-domain-name' with your actual domain name (e.g., example.com)
  sub_domain_name: your-sub-domain-name  # Replace 'your-sub-domain-name' with your actual subdomain name (e.g., www)
  certificate_arn: your-certificate-arn  # Replace 'your-certificate-arn' with the ARN of your SSL/TLS certificate (e.g., arn:aws:acm:region:account-id:certificate/certificate-id)


##  You can set up provisioned concurrency for Lambda functions.
provisioned_concurrency:
  ##  The api router Lambda function is responsible for handling API requests.
  ##  By default, this setting is disabled. To enable it, uncomment the 'api_router' section.
  # api_router:
  #   min_capacity: 1  # Minimum number of instances to keep running (adjust based on your needs)
  #   max_capacity: 3  # Maximum number of instances allowed (adjust based on your needs)
  #   utilization_target: 0.8  ##  Target utilization percentage (adjust based on your needs, e.g., 0.8 for 80%)

  ##  The log handler Lambda function is responsible for saving logs.
  ##  By default, this setting is disabled. To enable it, uncomment the 'log_handler' section.
  # log_handler:
  #   min_capacity: 1  # Minimum number of instances to keep running (adjust based on your needs)
  #   max_capacity: 3  # Maximum number of instances allowed (adjust based on your needs)
  #   utilization_target: 0.8  # Target utilization percentage (adjust based on your needs, e.g., 0.8 for 80%)


##  You can enable personalized search.
##  By default, this setting is disabled. To enable it, uncomment the 'personalized_ranking' section.
# personalized_ranking:
#   campaign_arn: your-campaign-arn  # Replace 'your-campaign-arn' with your actual campaign ARN from AWS Personalize

conf/hybrid_search.yaml¶

hybrid_queries:
  # Add your configuration here
  # Example:
  - index: your_index_name  # Replace 'your_index_name' with the actual name of your index
    embedding_mappings:
      - embedding_field: your-embedding-field  # Replace 'your_embedding_field' with the field name for the embeddings
        model_name: your_model_name  # Replace 'your_model_name' with the name of the embedding model to use
        src_fields:
          - source_field_1  # Replace 'source_field_1' with the source field from your data
          - source_field_2  # Add more source fields as needed
          ...
      ...
  ...

conf/model.yaml¶

##  You can use the following models to capture the semantics behind search queries and
##  the content being searched for Hybrid Search. Please refer to conf/hybrid_search.yaml
##  to check which models be used, and ensure the corresponding model section is uncommented.

##  Configuration for bge-large-en-v1.5 model
# bge_large_en_v1_5:
#   initial_instance_count: 1  # Set the initial number of instances (adjust based on your needs)
#   instance_type: "ml.g4dn.xlarge"  # Choose the appropriate instance type (e.g., "ml.g4dn.xlarge")


##  Configuration for splade-v3 model
# splade_v3:
#   initial_instance_count: 1  # Set the initial number of instances (adjust based on your needs)
#   instance_type: "ml.g4dn.xlarge"  # Choose the appropriate instance type (e.g., "ml.g4dn.xlarge")


##  Configuration for cohere.embed-english-v3 model
# cohere_embed_english_v3:
#   query_concurrency: 4  # Set the concurrency for queries (adjust based on your needs)


##  Configuration for cohere.embed-multilingual-v3 model
# cohere_embed_multilingual_v3:
#   query_concurrency: 4  # Set the concurrency for queries (adjust based on your needs)

Step 2: Deploy the Hybrid Search service¶

1. Deploy the OpenSearch API Gateway service¶

search-hq opensearch-gateway deploy

This command uses the configurations from your conf/project.yaml and conf/api.yaml files to deploy the OpenSearch API Gateway service, which enables the use of API endpoints.

2. Deploy the embedding model and synchronize configuration information to AWS¶

search-hq hybrid-search deploy-model

This command uses the configurations from your conf/hybrid_search.yaml and conf/model.yaml files to deploy the embedding model and synchronize the configuration information to AWS. This ensures that the Hybrid Search service is set up correctly.

Ingest Data¶

Step 1: Create or Update an Index Schema to Support KNN¶

Case 1: If you have an existing index and want to add semantic fields¶

Follow these steps to add semantic fields to an existing index:

Retrieve the existing index information:

curl --location 'https://<HOST>/raw-backend/<INDEX>' \
--header 'X-Api-Key: <API_KEY>'

Replace <HOST>, <INDEX>, and <API_KEY> with the appropriate values:

<HOST>: The endpoint URL of your OpenSearch service. This should be the base URL where your OpenSearch instance is accessible.
<INDEX>: The name of the index you want to retrieve information for. This is the specific index whose configuration and mappings you want to inspect.
<API_KEY>: Your API key used for authentication. This ensures that you have the necessary permissions to access the index information.

This command sends a request to the OpenSearch API to fetch the current configuration and mappings of the specified index.

After executing the above command, you will receive a response similar to this:

{
    "<INDEX>": {
        "aliases": {},
        "mappings": {
            "properties": {
                "field_1": {
                    "type": "text"
                },
                "field_2": {
                    "type": "text"
                }
            }
        },
        "settings": {
            "index": {
                "replication": {
                    "type": "DOCUMENT"
                },
                "number_of_shards": "<NUMBER_OF_SHARDS>",
                "provided_name": "<INDEX>",
                "knn": "true",
                "creation_date": "1721097800047",
                "number_of_replicas": "<NUMBER_OF_REPLICAS>",
                "uuid": "9UYi9hTWTNm2n2EK5g9jzg",
                "version": {
                    "created": "136347827"
                }
            }
        }
    }
}

Clean up the response to match the format required by the OpenSearch Create Index API:

{
  "settings": {
    "index": {
      "number_of_shards": "<NUMBER_OF_SHARDS>",
      "number_of_replicas": "<NUMBER_OF_REPLICAS>"
    }
  },
  "mappings": {
    "properties": {
      "field_1": {
        "type": "text"
      },
      "field_2": {
        "type": "text"
      }
    }
  },
  "aliases": {
    "sample-alias1": {}
  }
}

Save the cleaned response to a local file path, such as <schema_file_path>.jsonl.

Generate the semantic index schema:
```
search-hq hybrid-search generate-semantic-index-schema --index <INDEX> --schema_file_path <PATH_TO_SCHEMA_FILE>
```
Replace <INDEX> and <PATH_TO_SCHEMA_FILE> with the appropriate values:
- <INDEX>: The name of the index for which you want to generate the semantic index schema. This should match the index name in your OpenSearch setup where you want to apply semantic search capabilities.
- <PATH_TO_SCHEMA_FILE>: The local file path to the schema file that contains the base index configuration. This file should be formatted in JSON and include the settings and mappings that define how the index is structured and configured.
This command generates a semantic index schema based on your specified index and schema file, extending the index to support KNN operations and incorporating any semantic fields as defined in hybrid_search.yaml configuration.

For example, the extended index schema might look like this:
```
{
  "settings": {
    "index": {
      "number_of_shards": "<NUMBER_OF_SHARDS>",
      "number_of_replicas": "<NUMBER_OF_REPLICAS>",
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "book_id": {
        "type": "text"
      },
      "book_title": {
        "type": "text"
      },
      "book_description": {
        "type": "text"
      },
      "book_title_dense_embed": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "engine": "lucene",
          "name": "hnsw",
          "parameters": {},
          "space_type": "cosinesimil"
        }
      },
      "book_title_sparse_embed": {
        "type": "rank_features"
      }
    }
  },
  "aliases": {
    "sample-alias1": {}
  }
}
```
This schema includes:
- KNN Setting: The knn setting in the settings section will be set to true, enabling KNN search capabilities for the index.
- Embedding Fields: Based on your hybrid_search.yaml configuration, semantic fields will be added.
  - knn_vector for dense embeddings, suitable for high-dimensional vectors.
  - rank_features for sparse embeddings, suitable for lower-dimensional vectors.
Ensure that the embedding fields and their configurations are correctly defined in hybrid_search.yaml to align with the desired search functionalities.

Case 2: Creating a New Index with Semantic Fields¶

If you don't have an existing index, you can create a new KNN index schema by yourself. Use the following structure to create a new KNN index schema:

Define the semantic index schema:
1. Settings: The knn setting in the settings section should be set to true, enabling KNN search capabilities for the index.
2. Mappings: Define the properties of the index fields:
  - <text_field_name>: A text field used for storing textual data. Replace <text_field_name> with a name relevant to your data, such as title or description.
  - <dense_embedding_field_name>: A knn_vector field for dense embeddings. Replace <dense_embedding_field_name> with a descriptive name for the embeddings, such as embedding_vector or semantic_embedding. Specify the vector dimension and method details. For reference, see the OpenSearch KNN Index Documentation.
  - <sparse_embedding_field_name>: A rank_features field for sparse embeddings. Replace <sparse_embedding_field_name> with a descriptive name, such as feature_vector or term_frequency.
3. Aliases: Include any index aliases as needed to simplify querying or to support multiple index versions.
For more detailed information on creating and configuring indices, refer to the OpenSearch Index API Documentation.

Here’s an example of how you might structure the schema:
```
{
    "settings": {
      "index": {
        "number_of_shards": "<NUMBER_OF_SHARDS>",
        "number_of_replicas": "<NUMBER_OF_REPLICAS>",
        "knn": true
      }
    },
    "mappings": {
      "properties": {
        "<text_field_name>": {
          "type": "text"
        },
        "<dense_embedding_field_name>": {
          "type": "knn_vector",
          "dimension": <DIMENSION>,
          "method": {
            "engine": "lucene",
            "name": "hnsw",
            "parameters": {},
            "space_type": "cosinesimil"
          }
        },
        "<sparse_embedding_field_name>": {
          "type": "rank_features"
        }
      }
    },
    "aliases": {
      "sample-alias1": {}
    }
}
```

Step 2: Create the new KNN index¶

Use the previously defined schema to create the index. Send a PUT request to the Raw Backend API with the following command:

curl --location --request PUT 'https://<HOST>/raw-backend/<INDEX>' \
--header 'X-Api-Key: <API_KEY>' \
--header 'Content-Type: application/json' \
--data '<INDEX_SCHEMA>'

Replace <HOST>, <INDEX>, <API_KEY>, and <INDEX_SCHEMA> with the appropriate values:

<HOST>: Your OpenSearch service endpoint.
<INDEX>: The name of the index you want to create.
<API_KEY>: Your API key for authentication.
<INDEX_SCHEMA>: The JSON schema that defines the index settings and mappings, as generated in the previous steps.

This command will create the new KNN index using the schema you defined, allowing it to support KNN operations and include the necessary fields for your semantic search.

Step 3: Ingest data into OpenSearch¶

Prepare Your Bulk Request: Format your data according to the OpenSearch bulk API requirements. The <BULK_REQUEST> should include a series of newline-delimited JSON objects representing the documents to be ingested.
Send the Bulk Request: Use the following command to ingest data into your OpenSearch index:
```
curl --location 'https://<HOST>/application/bulk' \
--header 'X-Api-Key: <API_KEY>' \
--header 'Content-Type: application/json' \
--data '<BULK_REQUEST>'
```
Replace <HOST>, <API_KEY>, and <BULK_REQUEST> with the appropriate values:
- <HOST>: Your OpenSearch service endpoint.
- <API_KEY>: Your API key for authentication.
- <BULK_REQUEST>: The JSON payload containing the bulk request data, which should be formatted according to the OpenSearch bulk API specifications.
This command will send your data to OpenSearch in bulk, allowing for efficient indexing and updating of documents in your new KNN-enabled index.

Usage¶

We provide three API endpoints

Bulk API: The Bulk API allows you to add, update, or delete multiple documents within a single request. Compared to the native OpenSearch Bulk API, our Bulk API is more efficient at generating embeddings.
Hybrid Search API: The Hybrid Search API allows you to combine keyword and semantic search to improve search relevance. Users only need to specify dynamic tokens in the request, Search HQ will automatically retrieve the corresponding vectors based on configuration files. Furthermore, with personalized search settings configured, you can obtain customized search result rankings for specific users.
Raw Backend API: The Raw-Backend API enables you to interact directly with the native OpenSearch REST API for CRUD operations.

For details on how to use these APIs, refer to the sections below:

Bulk API¶

Endpoint URL¶

POST /application/bulk

Request body¶

{
  "index": "string",
  "bulk_request":  [
    {
      "id": "string",
      "op": "string",
      "doc": { ... }
    }
  ]
}

Field	Type	Required	Description
`index`	String	Required	The name of the index to which the bulk operation is to be applied.
`bulk_request`	Array	Required	A list of dictionaries representing the bulk operations. Each dictionary contains the fields `id`, `op`, and `doc`.
`bulk_request.id`	String	Required	The ID of the document.
`bulk_request.op`	String	Required	The operation to be performed. Accepted values are `"index"`, `"create"`, `"update"`, and `"delete"`.
`bulk_request.doc`	Object	Required	The document to be indexed, created, or updated. For delete operations, this field is not required.

When configured with hybrid_search.yaml, the Bulk API will automatically perform embedding inference for the specified fields. Embedding fields do not need to be pre-calculated and included in the doc.

Note

The bulk_request field should conform to the custom structure specified above, which differs from the native OpenSearch Bulk API.

Response body¶

The response body includes request body appended with index_response. The index_response is the response from OpenSearch bulk API

{
  "index": "book_semantic_index",
  "bulk_request": [
    {
      "id": "1",
      "op": "index",
      "doc": {
        "book_title": "Love Story",
        "book_description": "A classic love story."
      }
    }
  ],
  "index_response": {
    "took": 3,
    "errors": false,
    "items": [
      {
        "index": {
          "_index": "book_semantic_index",
          "_id": "1",
          "_version": 3,
          "result": "updated",
          "_shards": {
            "total": 2,
            "successful": 1,
            "failed": 0
          },
          "_seq_no": 2,
          "_primary_term": 1,
          "status": 200
        }
      }
    ]
  }
}

Example requests and responses¶

If hybrid_search.yaml is configured as follows:

hybrid_queries:
  - index: book_semantic_index
    embedding_mappings:
      - embedding_field: book_title_embed
        model_name: cohere_embed_english_v3
        src_fields:
          - book_title

This configuration specifies that for the book_semantic_index, the book_title field should be processed using the cohere_embed_english_v3 model to generate embeddings stored in the book_title_embed field.

The following request will automatically perform embedding inference for the book_title field and store the embeddings in the book_title_embed field.

Request¶

{
 "index": "book_semantic_index",
  "bulk_request": [
    { "id": "1", "op": "index", "doc": { "book_id": "BA01", "book_title": "Love Story", "book_description": "A classic love story.", "publication_year": 2024, "genre": "Romance" } },
    { "id": "2", "op": "index", "doc": { "book_id": "BA02", "book_title": "Endless Love", "book_description": "A tale of endless love and romance.", "publication_year": 2022, "genre": "Romance" } },
    { "id": "3", "op": "index", "doc": { "book_id": "BA03", "book_title": "Romantic Journey", "book_description": "A romantic journey through time.", "publication_year": 2021, "genre": "Romance" } }
  ]
}

Response¶

{
    "index": "book_semantic_index",
    "bulk_request": [
        {
            "id": "1",
            "op": "index",
            "doc": {
                "book_id": "BA01",
                "book_title": "Love Story",
                "book_description": "A classic love story.",
                "publication_year": 2024,
                "genre": "Romance"
            }
        },
        {
            "id": "2",
            "op": "index",
            "doc": {
                "book_id": "BA02",
                "book_title": "Endless Love",
                "book_description": "A tale of endless love and romance.",
                "publication_year": 2022,
                "genre": "Romance"
            }
        },
        {
            "id": "3",
            "op": "index",
            "doc": {
                "book_id": "BA03",
                "book_title": "Romantic Journey",
                "book_description": "A romantic journey through time.",
                "publication_year": 2021,
                "genre": "Romance"
            }
        }
    ],
    "index_response": {
        "took": 31,
        "errors": false,
        "items": [
            {
                "index": {
                    "_index": "book_semantic_index",
                    "_id": "1",
                    "_version": 1,
                    "result": "created",
                    "_shards": {
                        "total": 1,
                        "successful": 1,
                        "failed": 0
                    },
                    "_seq_no": 0,
                    "_primary_term": 1,
                    "status": 201
                }
            },
            {
                "index": {
                    "_index": "book_semantic_index",
                    "_id": "2",
                    "_version": 1,
                    "result": "created",
                    "_shards": {
                        "total": 1,
                        "successful": 1,
                        "failed": 0
                    },
                    "_seq_no": 1,
                    "_primary_term": 1,
                    "status": 201
                }
            },
            {
                "index": {
                    "_index": "book_semantic_index",
                    "_id": "3",
                    "_version": 1,
                    "result": "created",
                    "_shards": {
                        "total": 1,
                        "successful": 1,
                        "failed": 0
                    },
                    "_seq_no": 2,
                    "_primary_term": 1,
                    "status": 201
                }
            }
        ]
    }
}

The indexed result will include a book_title_embed field that contains the KNN vector, as shown in the following example:

{
  "hits": [
      {
          "_index": "book_semantic_index",
          "_id": "1",
          "_score": 1.0,
          "_source": {
              "book_id": "BA01",
              "book_title": "Love Story",
              "book_description": "A classic love story.",
              "publication_year": 2024,
              "genre": "Romance",
              "book_title_embed": [
                  -0.01789856,
                  0.0036449432,
                  0.0071411133,
                  ...
              ]
          }
      },
      {
          "_index": "book_semantic_index",
          "_id": "2",
          "_score": 1.0,
          "_source": {
              "book_id": "BA02",
              "book_title": "Endless Love",
              "book_description": "A tale of endless love and romance.",
              "publication_year": 2022,
              "genre": "Romance",
              "book_title_embed": [
                  0.046447754,
                  0.039215088,
                  -0.035217285,
                  ...
              ]
          }
      },
      {
          "_index": "book_semantic_index",
          "_id": "3",
          "_score": 1.0,
          "_source": {
              "book_id": "BA03",
              "book_title": "Romantic Journey",
              "book_description": "A romantic journey through time.",
              "publication_year": 2021,
              "genre": "Romance",
              "book_title_embed": [
                  0.030960083,
                  -0.014144897,
                  -0.018478394
                  ...
              ]
          }
      }
  ]
}

Hybrid Search API¶

Endpoint URL¶

POST /application/hybrid-search

Request Body¶

{
  "index": "string",
  "search_query": "string",
  "search_request_template": {
    ...
  },
  "settings": {
    "weights": "array of float",
    "normalization": "min_max" | "l2",
    "combination": "arithmetic_mean" | "geometric_mean" | "harmonic_mean"
  },
  "user_id": "string",
  "business_rules": {
    "is_personalized_enabled": false
  },
  "is_logging_enable": true
}

Field	Type	Required	Description
`index`	String	Required	The name of the index on which the search is performed.
`search_query`	String	Required	The query string for searching.
`search_request_template`	Object	Required	The dictionary containing parameters for the search request. This should adhere to the expected structure of the OpenSearch API.
`settings`	Object	Optional	Settings for Hybrid Search.
`settings.weights`	Array of Float	Optional	Specifies the weights for each query. Valid values are in the range [0.0, 1.0], where a weight closer to 1.0 indicates higher importance. The number of values must match the number of queries, and the sum must equal 1.0. If not provided, all queries are given equal weight.
`settings.normalization`	String	Optional	Technique for normalizing scores. Valid options are `min_max` and `l2`. Default is `min_max`.
`settings.combination`	String	Optional	Technique for combining scores. Valid options are `arithmetic_mean`, `geometric_mean`, and `harmonic_mean`. Default is `arithmetic_mean`.
`user_id`	String	Optional	ID of the user making the request. Useful for tracking or personalization. Default is `None`.
`business_rules`	Object	Optional	Allows additional rules for the search request. Currently, only the `is_personalized_enabled` rule is supported, defaulting to `False`.
`business_rules.is_personalized_enabled`	Boolean	Optional	Whether personalized search is enabled. If `True`, search results may be influenced by the user's preferences. Default is `False`.
`is_logging_enable`	Boolean	Optional	Whether to log the search request. Default is `True`.

Search Request Template¶

The search_request_template defines the structure of the search query. The query section uses an OpenSearch Hybrid Query that can include up to 5 subqueries. Each subquery retrieves search results independently and combines them according to the specified settings.

Semantic Search Subqueries:

KNN Subquery:

This subquery performs approximate KNN search using OpenSearch. Based on the hybrid_search.yaml, the system processes the specified text field through the embedding model to generate embeddings. The dynamic token in the query is replaced with the generated embeddings, and the search is performed on the designated field.

Example:
```
{
  "knn": {
    "<DENSE_EMBEDDING_FIELD>": {
      "vector": "[<DENSE_EMBEDDING_FIELD>]",
      "k": 200
    }
  }
}
```
In this example, "[<DENSE_EMBEDDING_FIELD>]" will be replaced with the embeddings generated from the specified text field using the embedding model defined in the configuration file. This is then used for KNN search on the field <DENSE_EMBEDDING_FIELD>.
Sparse Embedding Subquery:

This subquery performs a search using sparse embeddings, which are stored as rank_features in OpenSearch. The system converts the specified text field into a sparse embedding (which are represented as keyword-weight pairs) using the embedding model. The dynamic token in the query is replaced with the generated sparse embedding, and the search is conducted on the field designated for sparse encoding.

Example:
```
{
  "bool": {
    "should": "[<SPARSE_EMBEDDING_FIELD>]"
  }
}
```
In this example, "[<SPARSE_EMBEDDING_FIELD>]" will be replaced with the sparse embedding created from the specified text field, where the vector is a rank_features data type. This is then used for the search on the field <SPARSE_EMBEDDING_FIELD>.

Dynamic Token Format in Search Request Template

The dynamic token format in the search request template body should be "[<EMBEDDING_FIELD_NAME>]", where <EMBEDDING_FIELD_NAME> is the name of the semantic field containing the embedding data to be used. Ensure that you use double quotes and square brackets around the field name. This format is crucial for the system to correctly replace the token with the appropriate embedding data during the search.

Response Body¶

The response body includes request body appended with search_response. The search_response is the response from OpenSearch search API

{
  "index": "string",
  "search_query": "string",
  "search_request_template": {
    ...
  },
  "settings": {
    "weights": "array of float",
    "normalization": "min_max" | "l2",
    "combination": "arithmetic_mean" | "geometric_mean" | "harmonic_mean"
  },
  "user_id": "string",
  "business_rules": {
    "is_personalized_enabled": false
  },
  "is_logging_enable": true,
  "search_response": {
    "took": 3,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 3,
            "relation": "eq"
        },
        "max_score": 0.76324844,
        "hits": [
            {
                "_index": "book_semantic_index",
                "_id": "2",
                "_score": 0.76324844,
                "_source": {
                    "book_title": "Endless Love",
                    "book_description": "A tale of endless love and romance."
                }
            },
            {
                "_index": "book_semantic_index",
                "_id": "1",
                "_score": 0.5005,
                "_source": {
                    "book_title": "Love Story",
                    "book_description": "A classic love story."
                }
            }
        ]
    },
    "profile": {
        "shards": []
    }
  }
}

Example requests and responses¶

For example, consider a scenario where you want to perform a search combining keyword search with both dense and sparse semantic search techniques. The hybrid_search.yaml file might look like this:

- index: book_semantic_index
  embedding_mappings:
    - embedding_field: book_title_sparse_embed
      model_name: splade_v3
      src_fields:
        - book_title
    - embedding_field: book_title_dense_embed
      model_name: cohere_embed_english_v3
      src_fields:
        - book_title

In this configuration:

book_title_sparse_embed field uses the splade_v3 model for generating sparse embeddings, sourcing from the book_title field.
book_title_dense_embed field uses the cohere_embed_english_v3 model for generating dense embeddings, also sourcing from the book_title field.

Here's how you would structure the search request using this configuration:

Request Body¶

{
  "index": "book_semantic_index",
  "search_query": "endless love",
  "search_request_template": {
    "_source": {
      "includes": [
        "book_title",
        "book_description"
      ]
    },
    "size": 10,
    "query": {
      "hybrid": {
        "queries": [
          // keyword subquery on field "book_title"
          {
            "match": {
              "book_title": {
                "query": "endless love"
              }
            }
          },
          // KNN subquery on field "book_title_dense_embed"
          {
            "knn": {
              "book_title_dense_embed": {
                "vector": "[book_title_dense_embed]",
                "k": 100
              }
            }
          },
          // sparse embedding subquery on field "book_title_sparse_embed"
          {
            "bool": {
              "should": "[book_title_sparse_embed]"
            }
          }
        ]
      }
    }
  },
  "settings": {
    "weights": [
      0.1,
      0.6,
      0.3
    ],
    "normalization": "min_max",
    "combination": "arithmetic_mean"
  },
  "user_id": "user_id_123",
  "business_rules": {
    "is_personalized_enabled": false
  },
  "is_logging_enable": true
}

Response Body¶

{
    "index": "book_semantic_index",
    "search_query": "endless love",
    "search_request_template": {
        "_source": {
            "includes": [
                "book_title",
                "book_description",
                "genre",
                "publication_year"
            ]
        },
        "size": 5,
        "query": {
            "hybrid": {
                "queries": [
                    {
                        "match": {
                            "book_title": {
                                "query": "endless love"
                            }
                        }
                    },
                    {
                        "knn": {
                            "book_title_cohere": {
                                "vector": "[book_title_cohere]",
                                "k": 100
                            }
                        }
                    },
                    {
                        "bool": {
                            "should": "[book_title_splade]"
                        }
                    }
                ]
            }
        },
        "search_pipeline": {
            "request_processors": [
                {
                    "oversample": {
                        "sample_factor": 40.0
                    }
                }
            ],
            "response_processors": [
                {
                    "truncate_hits": {
                        "description": "Truncates back to the original size before oversample increased it."
                    }
                }
            ],
            "phase_results_processors": [
                {
                    "normalization-processor": {
                        "normalization": {
                            "technique": "min_max"
                        },
                        "combination": {
                            "technique": "arithmetic_mean",
                            "parameters": {
                                "weights": [
                                    0.1,
                                    0.6,
                                    0.3
                                ]
                            }
                        }
                    }
                }
            ]
        }
    },
    "settings": {
        "weights": [
            0.1,
            0.6,
            0.3
        ],
        "normalization": "min_max",
        "combination": "arithmetic_mean"
    },
    "user_id": "user_id_123",
    "business_rules": {
        "is_personalized_enabled": false
    },
    "is_logging_enable": true,
    "search_response": {
        "took": 512,
        "timed_out": false,
        "_shards": {
            "total": 1,
            "successful": 1,
            "skipped": 0,
            "failed": 0
        },
        "hits": {
            "total": {
                "value": 10,
                "relation": "eq"
            },
            "max_score": 1.0,
            "hits": [
                {
                    "_index": "book_semantic_index",
                    "_id": "2",
                    "_score": 1.0,
                    "_source": {
                        "publication_year": 2022,
                        "book_title": "Endless Love",
                        "genre": "Romance",
                        "book_description": "A tale of endless love and romance."
                    }
                },
                {
                    "_index": "book_semantic_index",
                    "_id": "1",
                    "_score": 0.43359107,
                    "_source": {
                        "publication_year": 2024,
                        "book_title": "Love Story",
                        "genre": "Romance",
                        "book_description": "A classic love story."
                    }
                },
                {
                    "_index": "book_semantic_index",
                    "_id": "10",
                    "_score": 0.4100484,
                    "_source": {
                        "publication_year": 2022,
                        "book_title": "Poetry of Love",
                        "genre": "Romance",
                        "book_description": "A beautiful poetry collection about love."
                    }
                },
                {
                    "_index": "book_semantic_index",
                    "_id": "4",
                    "_score": 0.340065,
                    "_source": {
                        "publication_year": 2020,
                        "book_title": "Love in the Time of Chaos",
                        "genre": "Romance",
                        "book_description": "A love story set in chaotic times."
                    }
                },
                {
                    "_index": "book_semantic_index",
                    "_id": "3",
                    "_score": 0.24266393,
                    "_source": {
                        "publication_year": 2021,
                        "book_title": "Romantic Journey",
                        "genre": "Romance",
                        "book_description": "A romantic journey through time."
                    }
                }
            ]
        },
        "profile": {
            "shards": []
        }
    }
}

If you need to apply additional conditions, you can modify the query to include these filters. For example:

To filter results by a specific publication_year or genre, incorporate filter clauses into the query. This customization allows you to refine search results according to specific criteria.

Request Body:

"search_request_template" = {
  "_source": {
      "includes": [
        "book_title",
        "book_description",
        "genre",
        "publication_year"
      ]
  },
  "size": 10,
  "query": {
    "hybrid": {
      "queries":  [
        // keyword subquery with filter
        {
          "bool": {
            "must": {
              "match": {
                "book_title": {
                  "query": "endless love"
                }
              }
            },
            "filter": [
              {
                "term": {
                  "genre": "Romance"
                }
              },
              {
                "range": {
                  "publication_year": {
                    "gte": 2018,
                    "lte": 2020
                  }
                }
              }
            ]
          }
        },
        // knn subquery with filter
        {
          "knn": {
            "book_title_dense_embed": {
              "vector": "[book_title_dense_embed]",
              "k": 200,
              "filter": {
                "bool": {
                  "must": [
                    {
                      "term": {
                        "genre": "Romance"
                      }
                    },
                    {
                      "range": {
                        "publication_year": {
                          "gte": 2018,
                          "lte": 2020
                        }
                      }
                    }
                  ]
                }
              }
            }
          }
        },
        // sparse embedding subquery with filter
        {
          "bool": {
            "should": "[book_title_sparse_embed]",
            "filter": [
              {
                "term": {
                  "genre": "Romance"
                }
              },
              {
                "range": {
                  "publication_year": {
                    "gte": 2018,
                    "lte": 2020
                  }
                }
              }
            ]
          }
        }
      ]
    }
  }
}

Response Body:

{
    "index": "book_semantic_index",
    "search_query": "love",
    "search_request_template": {
        "_source": {
            "includes": [
                "book_title",
                "book_description",
                "genre",
                "publication_year"
            ]
        },
        "size": 10,
        "query": {
            "hybrid": {
                "queries": [
                    {
                        "bool": {
                            "must": {
                                "match": {
                                    "book_title": {
                                        "query": "endless love"
                                    }
                                }
                            },
                            "filter": [
                                {
                                    "term": {
                                        "genre": "Romance"
                                    }
                                },
                                {
                                    "range": {
                                        "publication_year": {
                                            "gte": 2018,
                                            "lte": 2020
                                        }
                                    }
                                }
                            ]
                        }
                    },
                    {
                        "knn": {
                            "book_title_cohere": {
                                "vector": "[book_title_cohere]",
                                "k": 200,
                                "filter": {
                                    "bool": {
                                        "must": [
                                            {
                                                "term": {
                                                    "genre": "Romance"
                                                }
                                            },
                                            {
                                                "range": {
                                                    "publication_year": {
                                                        "gte": 2018,
                                                        "lte": 2020
                                                    }
                                                }
                                            }
                                        ]
                                    }
                                }
                            }
                        }
                    },
                    {
                        "bool": {
                            "should": "[book_title_splade]",
                            "filter": [
                                {
                                    "term": {
                                        "genre": "Romance"
                                    }
                                },
                                {
                                    "range": {
                                        "publication_year": {
                                            "gte": 2018,
                                            "lte": 2020
                                        }
                                    }
                                }
                            ]
                        }
                    }
                ]
            }
        },
        "search_pipeline": {
            "request_processors": [
                {
                    "oversample": {
                        "sample_factor": 20.0
                    }
                }
            ],
            "response_processors": [
                {
                    "truncate_hits": {
                        "description": "Truncates back to the original size before oversample increased it."
                    }
                }
            ],
            "phase_results_processors": [
                {
                    "normalization-processor": {
                        "normalization": {
                            "technique": "min_max"
                        },
                        "combination": {
                            "technique": "arithmetic_mean"
                        }
                    }
                }
            ]
        }
    },
    "settings": {
        "weights": null,
        "normalization": "min_max",
        "combination": "arithmetic_mean"
    },
    "user_id": "24",
    "business_rules": {
        "is_personalized_enabled": false
    },
    "is_logging_enable": true,
    "search_response": {
        "took": 4,
        "timed_out": false,
        "_shards": {
            "total": 1,
            "successful": 1,
            "skipped": 0,
            "failed": 0
        },
        "hits": {
            "total": {
                "value": 1,
                "relation": "eq"
            },
            "max_score": 1.0,
            "hits": [
                {
                    "_index": "book_semantic_index",
                    "_id": "4",
                    "_score": 1.0,
                    "_source": {
                        "publication_year": 2020,
                        "book_title": "Love in the Time of Chaos",
                        "genre": "Romance",
                        "book_description": "A love story set in chaotic times."
                    }
                }
            ]
        }
    }
}

Raw Backend API¶

Endpoint URLs¶

You can use the following endpoints to interact with the OpenSearch REST API:

POST /raw-backend/<endpoint>: Creates or modifies resources.
GET /raw-backend/<endpoint>: Retrieves information about resources.
PUT /raw-backend/<endpoint>: Updates resources or configuration.
DELETE /raw-backend/<endpoint>: Deletes resources.

Replace <endpoint> with the specific OpenSearch REST API path.

Request Body¶

The request body varies based on the specific OpenSearch REST API endpoint being called. For detailed information on the request format, refer to the OpenSearch REST API documentation.

Response Body¶

The response body also varies depending on the OpenSearch REST API endpoint. For details on the response format, consult the OpenSearch REST API documentation.

Example Requests and Responses¶

Create (POST)¶

Endpoint: POST /raw-backend/book_semantic_index

Request Body:

{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "mappings": {
    "properties": {
      "book_title": {
        "type": "text"
      },
      "book_description": {
        "type": "text"
      }
    }
  }
}

This request creates a new index called book_semantic_index with specified settings and mappings.

Response Body:

{
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "book_semantic_index"
}

Read (GET)¶

Endpoint: GET /raw-backend/book_semantic_index

Request Body: None

This request retrieves the details of the book_semantic_index, including its settings and mappings.

Response Body:

{
    "book_semantic_index": {
        "aliases": {},
        "mappings": {
            "properties": {
                "book_description": {
                    "type": "text"
                },
                "book_title": {
                    "type": "text"
                }
            }
        },
        "settings": {
            "index": {
                "replication": {
                    "type": "DOCUMENT"
                },
                "number_of_shards": "1",
                "provided_name": "book_semantic_index",
                "creation_date": "1722400154716",
                "number_of_replicas": "0",
                "uuid": "NrRI96H4QhKALsgj6SG6_g",
                "version": {
                    "created": "136347827"
                }
            }
        }
    }
}

Update (PUT)¶

Endpoint: PUT /raw-backend/book_semantic_index/_mapping

Request Body:

{
  "properties":{
    "book_category":{
      "type": "text"
    },
    "year":{
      "type": "integer"
    }
  }
}

This request updates the index mapping by adding new fields book_category and year to the book_semantic_index.

Response Body:

{
    "acknowledged": true
}

Delete (DELETE)¶

Endpoint: DELETE /raw-backend/book_semantic_index

Request Body: None

This request deletes the book_semantic_index from OpenSearch. This operation removes the index and all its associated data

Response Body:

{
    "acknowledged": true
}

Destruction¶

Step 1: Destroy the Hybrid Search service¶

To destroy the Hybrid Search service, execute the following command:

search-hq opensearch-gateway destroy --destroy_infra=False

This command will disable and remove the Hybrid Search service.

Note

If you are using only the Hybrid Search service, set destroy_infra to True to completely remove the service. However, if you are using other Search HQ services simultaneously, set destroy_infra to False to prevent shared resources from being deleted.

Step 2: Destroy the associated resources (Optional)¶

By default, the destroy command will retain the Amazon S3 bucket and AWS Systems Manager Parameter Store associated with your search-hq deployment. You can choose to manually delete these resources if needed.

To remove associated resources, follow these additional steps:

Remove Objects from Amazon S3 Bucket: To delete the entire Amazon S3 bucket associated with your search-hq deployment, use the following command. This will remove the entire bucket and all objects within it.
```
python -m operation.cli.s3 delete_bucket
```
If you only need to delete a single object from the Amazon S3 bucket, use:
```
python -m operation.cli.s3 delete_object --key <S3_OBJECT_KEY>
```
Replace <S3_OBJECT_KEY> with the key of the object you want to delete from the Amazon S3 bucket.
Remove Parameters from AWS Systems Manager Parameter Store: To delete all parameters associated with your search-hq deployment from AWS Systems Manager Parameter Store, use the following command. This will remove all parameters.
```
python -m operation.cli.ssm delete_ssm_parameters
```
If you only need to delete a single parameter, use:
```
python -m operation.cli.ssm delete_ssm_parameter --parameter_name <PARAMETER_NAME>
```
Replace <PARAMETER_NAME> with thename of the parameter you want to delete from the AWS Systems Manager Parameter Store.

By following the instructions in this guide, you can efficiently deploy, use, and destroy the Hybrid Search service. Make sure your YAML configurations are correct and use the provided search-hq commands to manage the service as needed.