Skip to content

Instructional Guide for OpenSearch API Gateway service

The OpenSearch API Gateway provides a structured and efficient way to manage search-related operations. It offers seamless integration with other AWS resources, enhancing the overall functionality and efficiency beyond just OpenSearch itself. The gateway supports various endpoints, such as bulk operations, standard search queries, Hybrid Search, and direct interaction with the native OpenSearch REST API.

This instructional guide provides detailed steps to deploy, use, and destroy the OpenSearch API Gateway service using the search-hq CLI commands. Follow the sections below for a comprehensive walkthrough on each process. The guide is divided into three sections:

  • Deployment: Step-by-step instructions to deploy the OpenSearch API Gateway service.
  • Usage: Guidance on how to effectively use the OpenSearch API Gateway service.
  • Destruction: Instruction to destroy the OpenSearch API Gateway service.

Prerequisites

Launch an Instance from Search HQ Launch Pack

The Search HQ Launch Pack is an AMI available on AWS Marketplace. Before proceeding with the steps below, you need to launch an instance using this AMI. Follow the Setup Instructions to set up the instance, enabling you to use the search-hq CLI commands.

Deployment

Follow these steps to deploy the OpenSearch API Gateway service.

Step 1: Configure Your YAML File

Ensure your YAML configuration files (conf/project.yaml and conf/api.yaml) are correctly set up.

For detailed information, refer to Instructional Guide for Project Configuration and Instructional Guide for API Configuration.

conf/project.yaml

##  You can configure the project name.
project_name: your-project-name  # Replace 'your-project-name' with the name of your project.


##  You can configure the AWS region.
aws:
  region_name: ap-northeast-1  # AWS region where resources will be deployed. Modify if needed (e.g., 'ap-northeast-1').


##  You need to use the existing VPC.
vpc:
  vpc_id: your-vpc-id  # Replace 'your-vpc-id' with the ID of your existing VPC.


##  You need to use the existing OpenSearch.
opensearch:
  domain: your-opensearch-domain  # Replace 'your-opensearch-domain' with the domain VPC endpoint for OpenSearch.
  domain_name: your-opensearch-domain-name  # Replace 'your-opensearch-domain-name' with the specific domain name.

  ##  If the fine-grained access control in your OpenSearch is enabled,
  ##  provide the necessary information in the 'auth' section as required.
  ##  By default, this setting is disabled. To enable it, uncomment the 'auth' section.
  # auth:
  #   type: aws  ##  The authentication type. Options: `aws` for AWS IAM authentication or `basic` for account and password. If `basic` is selected, you need to provide master_user_secret_arn.
  #   master_user_secret_arn: auth-arn  # The ARN of the SSM Parameter Store or AWS Secrets Manager where the master user’s account and password information is stored.

  ##  You can build OpenSearch Auto Scaling mechanism by configuring the auto_scaling section.
  ##  By default, this setting is disabled. To enable it, uncomment the 'auto_scaling' section.
  # auto_scaling:
  #   min_instance_count: 2  # Minimum number of instances for auto scaling.
  #   max_instance_count: 12  # Maximum number of instances for auto scaling.
  #   scale_down_cpu_util_threshold: 30  # CPU utilization threshold to scale down instances.
  #   scale_up_cpu_util_threshold: 80  # CPU utilization threshold to scale up instances.

conf/api.yaml

##  You can specify the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs.
# cloud_watch_role_name: your-cloud-watch-role-name  # Uncomment and replace 'your-cloud-watch-role-name' with your actual CloudWatch role name, if you already have one.


##  You can name custom domain name for your API.
domain_name_system:
  domain_name: your-domain-name  # Replace 'your-domain-name' with your actual domain name (e.g., example.com)
  sub_domain_name: your-sub-domain-name  # Replace 'your-sub-domain-name' with your actual subdomain name (e.g., www)
  certificate_arn: your-certificate-arn  # Replace 'your-certificate-arn' with the ARN of your SSL/TLS certificate (e.g., arn:aws:acm:region:account-id:certificate/certificate-id)


##  You can set up provisioned concurrency for Lambda functions.
provisioned_concurrency:
  ##  The api router Lambda function is responsible for handling API requests.
  ##  By default, this setting is disabled. To enable it, uncomment the 'api_router' section.
  # api_router:
  #   min_capacity: 1  # Minimum number of instances to keep running (adjust based on your needs)
  #   max_capacity: 3  # Maximum number of instances allowed (adjust based on your needs)
  #   utilization_target: 0.8  ##  Target utilization percentage (adjust based on your needs, e.g., 0.8 for 80%)

  ##  The log handler Lambda function is responsible for saving logs.
  ##  By default, this setting is disabled. To enable it, uncomment the 'log_handler' section.
  # log_handler:
  #   min_capacity: 1  # Minimum number of instances to keep running (adjust based on your needs)
  #   max_capacity: 3  # Maximum number of instances allowed (adjust based on your needs)
  #   utilization_target: 0.8  # Target utilization percentage (adjust based on your needs, e.g., 0.8 for 80%)


##  You can enable personalized search.
##  By default, this setting is disabled. To enable it, uncomment the 'personalized_ranking' section.
# personalized_ranking:
#   campaign_arn: your-campaign-arn  # Replace 'your-campaign-arn' with your actual campaign ARN from AWS Personalize

Step 2: Deploy the OpenSearch API Gateway service

Execute the following command to deploy the OpenSearch API Gateway service:

search-hq opensearch-gateway deploy

This command will use the configurations from your conf/project.yaml and conf/api.yaml files to set up the OpenSearch API Gateway service.

Upon successful deployment, you will see the API key pair in the console:

API
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ API URL                                                    ┃ API Key                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ https://example.execute-api.ap-northeast-1.amazonaws.com/  │ myapikeyabcdefghijklmnop  |
└────────────────────────────────────────────────────────────┴───────────────────────────┘

You can use this API key to make requests to the OpenSearch API Gateway. For more information on making API requests, please refer to the Usage section.

Usage

We provide three API endpoints:

  • Bulk API: The Bulk API allows you to add, update, or delete multiple documents within a single request.
  • Search API: The Search API allows you to execute search requests to retrieve data from your cluster. With personalized search settings configured, you can obtain customized search result rankings for specific users.
  • Raw Backend API: The Raw-Backend API enables you to interact directly with the native OpenSearch REST API for CRUD operations.

For details on how to use these APIs, refer to the sections below:

Bulk API

Endpoint URL

POST /application/bulk

Request body

{
  "index": "string",
  "bulk_request":  [
    {
      "id": "string",
      "op": "index" | "create" | "update" | "delete",
      "doc": { ... }
    }
  ]
}
Field Type Required Description
index String Required The name of the index to which the bulk operation is to be applied.
bulk_request Array Required A list of dictionaries representing the bulk operations. Each dictionary contains the fields id, op, and doc.
bulk_request.id String Required The ID of the document.
bulk_request.op String Required The operation to be performed. Accepted values are index, create, update, and delete.
bulk_request.doc Object Required The document to be indexed, created, or updated. For delete operations, this field is not required.

Note

The bulk_request field should conform to the custom structure specified above, which differs from the native OpenSearch Bulk API.

Response body

The response body includes request body appended with index_response. The index_response is the response from OpenSearch bulk API

{
  "index": "book_semantic_index",
  "bulk_request": [
    {
      "id": "1",
      "op": "index",
      "doc": {
        "book_title": "Love Story",
        "book_description": "A classic love story."
      }
    }
  ],
  "index_response": {
    "took": 3,
    "errors": false,
    "items": [
      {
        "index": {
          "_index": "book_semantic_index",
          "_id": "1",
          "_version": 3,
          "result": "updated",
          "_shards": {
            "total": 2,
            "successful": 1,
            "failed": 0
          },
          "_seq_no": 2,
          "_primary_term": 1,
          "status": 200
        }
      }
    ]
  }
}

Example requests and responses

Request
{
  "index": "book_semantic_index",
  "bulk_request": [
    { "id": "1", "op": "index", "doc": { "book_title": "Love Story", "book_description": "A classic love story." } },
    { "id": "2", "op": "create", "doc": { "book_title": "Endless Love", "book_description": "A timeless romance." } },
    { "id": "1", "op": "update", "doc": { "book_description": "An updated classic love story." } },
    { "id": "3", "op": "delete" }
  ]
}
Response
{
    "index": "book_semantic_index",
    "bulk_request": [
        {
            "id": "1",
            "op": "index",
            "doc": {
                "book_title": "Love Story",
                "book_description": "A classic love story."
            }
        },
        {
            "id": "2",
            "op": "create",
            "doc": {
                "book_title": "Endless Love",
                "book_description": "A timeless romance."
            }
        },
        {
            "id": "1",
            "op": "update",
            "doc": {
                "book_description": "An updated classic love story."
            }
        },
        {
            "id": "3",
            "op": "delete",
            "doc": null
        }
    ],
    "index_response": {
        "took": 260,
        "errors": false,
        "items": [
            {
                "index": {
                    "_index": "book_semantic_index",
                    "_id": "1",
                    "_version": 1,
                    "result": "created",
                    "_shards": {
                        "total": 2,
                        "successful": 1,
                        "failed": 0
                    },
                    "_seq_no": 0,
                    "_primary_term": 1,
                    "status": 201
                }
            },
            {
                "create": {
                    "_index": "book_semantic_index",
                    "_id": "2",
                    "_version": 1,
                    "result": "created",
                    "_shards": {
                        "total": 2,
                        "successful": 1,
                        "failed": 0
                    },
                    "_seq_no": 0,
                    "_primary_term": 1,
                    "status": 201
                }
            },
            {
                "update": {
                    "_index": "book_semantic_index",
                    "_id": "1",
                    "_version": 2,
                    "result": "updated",
                    "_shards": {
                        "total": 2,
                        "successful": 1,
                        "failed": 0
                    },
                    "_seq_no": 1,
                    "_primary_term": 1,
                    "status": 200
                }
            },
            {
                "delete": {
                    "_index": "book_semantic_index",
                    "_id": "3",
                    "_version": 1,
                    "result": "not_found",
                    "_shards": {
                        "total": 2,
                        "successful": 1,
                        "failed": 0
                    },
                    "_seq_no": 0,
                    "_primary_term": 1,
                    "status": 404
                }
            }
        ]
    }
}

Search API

Endpoint URL

POST /application/search

Request body

{
  "index": "string",
  "search_request": {
    ...
  },
  "user_id": "string",
  "business_rules": {
    "is_personalized_enabled": "boolean"
  },
  "is_logging_enable": "boolean"
}
Field Type Required Description
index String Required The name of the index on which the search is to be performed.
search_request Object Required The dictionary containing the parameters of the search request. This field should conform to the expected structure of the OpenSearch API.
user_id String Optional The ID of the user making the search request. This can be used for tracking or personalization purposes. Default is None.
business_rules Object Optional It allows specifying additional rules to be applied to the search request. So far, we only support the is_personalized_enabled rule, which defaults to False.
business_rules.is_personalized_enabled Boolean Optional Whether personalized search is enabled. When set to True, the search results may be influenced by the user's preferences. Default is False.
is_logging_enable Boolean Optional Whether the search request should be logged. Default is True.

Response body

The response body includes request body appended with search_response. The search_response is the response from OpenSearch search API

{
  "index": "string",
  "search_request": {
    ...
  },
  "user_id": "string",
  "business_rules": {
    "is_personalized_enabled": "boolean"
  },
  "is_logging_enable": "boolean",
  "search_response": {
    "took": 3,
    "timed_out": false,
    "_shards": {
      "total": 1,
      "successful": 1,
      "skipped": 0,
      "failed": 0
    },
    "hits": {
      "total": {
        "value": 66,
        "relation": "eq"
      },
      "max_score": 1.0,
      "hits": [
        {
          "_index": "book_semantic_index",
          "_id": "1",
          "_score": 1.0,
          "_source": {
            "book_title": "Love Story"
          }
        },
        {
          "_index": "book_semantic_index",
          "_id": "2",
          "_score": 1.0,
          "_source": {
            "book_title": "Endless Love"
          }
        },
        {
          "_index": "book_semantic_index",
          "_id": "3",
          "_score": 1.0,
          "_source": {
            "book_title": "The Love Hypothesis"
          }
        }
      ]
    }
  }
}

Example requests and responses

Request
{
  "index": "book_semantic_index",
  "search_request": {
    "_source": [
      "book_title"
    ],
    "size": 3,
    "query": {
      "match": {
        "book_title": "love"
      }
    }
  },
  "user_id": "user_id_123",
  "business_rules": {
    "is_personalized_enabled": false
  },
  "is_logging_enable": true
}
Response
{
  "index": "book_semantic_index",
  "search_request": {
    "_source": [
      "book_title"
    ],
    "size": 3,
    "query": {
      "match": {
        "book_title": "love"
      }
    }
  },
  "user_id": "user_id_123",
  "business_rules": {
    "is_personalized_enabled": false
  },
  "is_logging_enable": true,
  "search_response": {
    "took": 3,
    "timed_out": false,
    "_shards": {
      "total": 1,
      "successful": 1,
      "skipped": 0,
      "failed": 0
    },
    "hits": {
      "total": {
        "value": 66,
        "relation": "eq"
      },
      "max_score": 1.0,
      "hits": [
        {
          "_index": "book_semantic_index",
          "_id": "1",
          "_score": 1.0,
          "_source": {
            "book_title": "Love Story"
          }
        },
        {
          "_index": "book_semantic_index",
          "_id": "2",
          "_score": 1.0,
          "_source": {
            "book_title": "Endless Love"
          }
        },
        {
          "_index": "book_semantic_index",
          "_id": "3",
          "_score": 1.0,
          "_source": {
            "book_title": "The Love Hypothesis"
          }
        }
      ]
    }
  }
}

Raw Backend API

Endpoint URLs

You can use the following endpoints to interact with the OpenSearch REST API:

  • POST /raw-backend/<endpoint>: Creates or modifies resources.
  • GET /raw-backend/<endpoint>: Retrieves information about resources.
  • PUT /raw-backend/<endpoint>: Updates resources or configuration.
  • DELETE /raw-backend/<endpoint>: Deletes resources.

Replace <endpoint> with the specific OpenSearch REST API path.

Request Body

The request body varies based on the specific OpenSearch REST API endpoint being called. For detailed information on the request format, refer to the OpenSearch REST API documentation.

Response Body

The response body also varies depending on the OpenSearch REST API endpoint. For details on the response format, consult the OpenSearch REST API documentation.

Example Requests and Responses

Create (POST)

Endpoint: POST /raw-backend/book_semantic_index

Request Body:

{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "mappings": {
    "properties": {
      "book_title": {
        "type": "text"
      },
      "book_description": {
        "type": "text"
      }
    }
  }
}

This request creates a new index called book_semantic_index with specified settings and mappings.

Response Body:

{
    "acknowledged": true,
    "shards_acknowledged": true,
    "index": "book_semantic_index"
}
Read (GET)

Endpoint: GET /raw-backend/book_semantic_index

Request Body: None

This request retrieves the details of the book_semantic_index, including its settings and mappings.

Response Body:

{
    "book_semantic_index": {
        "aliases": {},
        "mappings": {
            "properties": {
                "book_description": {
                    "type": "text"
                },
                "book_title": {
                    "type": "text"
                }
            }
        },
        "settings": {
            "index": {
                "replication": {
                    "type": "DOCUMENT"
                },
                "number_of_shards": "1",
                "provided_name": "book_semantic_index",
                "creation_date": "1722400154716",
                "number_of_replicas": "0",
                "uuid": "NrRI96H4QhKALsgj6SG6_g",
                "version": {
                    "created": "136347827"
                }
            }
        }
    }
}
Update (PUT)

Endpoint: PUT /raw-backend/book_semantic_index/_mapping

Request Body:

{
  "properties":{
    "book_category":{
      "type": "text"
    },
    "year":{
      "type": "integer"
    }
  }
}

This request updates the index mapping by adding new fields book_category and year to the book_semantic_index.

Response Body:

{
    "acknowledged": true
}
Delete (DELETE)

Endpoint: DELETE /raw-backend/book_semantic_index

Request Body: None

This request deletes the book_semantic_index from OpenSearch. This operation removes the index and all its associated data

Response Body:

{
    "acknowledged": true
}

Destruction

Step 1: Destroy the OpenSearch API Gateway service

To destroy the OpenSearch API Gateway service, execute the following command:

search-hq opensearch-gateway destroy --destroy_infra=False

This command will disable and remove the OpenSearch API Gateway service.

Note

If you are using only the OpenSearch API Gateway service, set destroy_infra to True to completely remove the service. However, if you are using other Search HQ services simultaneously, set destroy_infra to False to prevent shared resources from being deleted.

Step 2: Destroy the associated resources (Optional)

By default, the destroy command will retain the Amazon S3 bucket and AWS Systems Manager Parameter Store associated with your search-hq deployment. You can choose to manually delete these resources if needed.

To remove associated resources, follow these additional steps:

  • Remove Objects from Amazon S3 Bucket: To delete the entire Amazon S3 bucket associated with your search-hq deployment, use the following command. This will remove the entire bucket and all objects within it.

    python -m operation.cli.s3 delete_bucket

    If you only need to delete a single object from the Amazon S3 bucket, use:

    python -m operation.cli.s3 delete_object --key <S3_OBJECT_KEY>

    Replace <S3_OBJECT_KEY> with the key of the object you want to delete from the Amazon S3 bucket.

  • Remove Parameters from AWS Systems Manager Parameter Store: To delete all parameters associated with your search-hq deployment from AWS Systems Manager Parameter Store, use the following command. This will remove all parameters.

    python -m operation.cli.ssm delete_ssm_parameters

    If you only need to delete a single parameter, use:

    python -m operation.cli.ssm delete_ssm_parameter --parameter_name <PARAMETER_NAME>

    Replace <PARAMETER_NAME> with the name of the parameter you want to delete from the AWS Systems Manager Parameter Store.

By following the steps outlined in this guide, you can deploy, use and destroy the OpenSearch API Gateway service efficiently. Ensure your YAML configurations are correct, and use the provided search-hq commands to manage the service as needed.