API Generator (#177)

* API Generator

Signed-off-by: Theo Truong <theotr@amazon.com>

* # Took advantage of the provided gem folder to grep for folders representing namespaces. We don't need to hardcode existing namespaces anymore!

Signed-off-by: Theo Truong <theotr@amazon.com>

---------

Signed-off-by: Theo Truong <theotr@amazon.com>

.rubocop.yml

@@ -7,6 +7,8 @@ require:
 AllCops:
   TargetRubyVersion: 2.5
   NewCops: enable
+  Exclude:
+    - 'api_generator/**/*'
 
 RSpec/ImplicitExpect:
   Enabled: false

CHANGELOG.md

@@ -4,6 +4,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 ## [Unreleased]
 ### Added
 - Added `remote_store.restore` action ([#176](https://github.com/opensearch-project/opensearch-ruby/pull/176))
+- Added API Generator ([#139](https://github.com/opensearch-project/opensearch-ruby/issues/139))
 ### Changed
 - Merged `opensearch-transport`, `opensearch-api`, and `opensearch-dsl` into `opensearch-ruby` ([#133](https://github.com/opensearch-project/opensearch-ruby/issues/133))
 - Bumped `mocha` gem from 1.x.x to 2.x.x ([#178](https://github.com/opensearch-project/opensearch-ruby/pull/178))

DEVELOPER_GUIDE.md

@@ -6,6 +6,7 @@
     - [Build and Test](#build-and-test)
     - [Integration Tests](#integration-tests)
     - [Linter](#linter)
+    - [Generate API Actions](#generate-api-actions)
   - [Submitting Changes](#submitting-changes)
 
 # Developer Guide
@@ -85,6 +86,10 @@ rubocop -a
 rubocop --auto-gen-config
 ```
 
+### Generate API Actions
+
+All changes to the API actions should be done via the `api_generator`. For more information, see the [API Generator's USER_GUIDE](./api_generator/USER_GUIDE.md).
+
 ## Submitting Changes
 
 See [CONTRIBUTING](CONTRIBUTING.md).

api_generator/.rubocop.yml

@@ -0,0 +1,21 @@
+require: rubocop-rake
+AllCops:
+  Include:
+    - 'lib/**/*.rb'
+    - 'Rakefile'
+  NewCops: enable
+
+Metrics/CyclomaticComplexity:
+  Enabled: false
+Metrics/MethodLength:
+  Enabled: false
+Metrics/AbcSize:
+  Enabled: false
+Metrics/PerceivedComplexity:
+  Enabled: false
+
+Layout/EmptyLineAfterGuardClause:
+  Enabled: false
+
+Style/MultilineBlockChain:
+  Enabled: false

api_generator/.ruby-version

@@ -0,0 +1 @@
+3.1.0

api_generator/USER_GUIDE.md

@@ -0,0 +1,48 @@
+This API Generator generates API actions for the OpenSearch Ruby client based off of [OpenSearch OpenAPI specification](https://github.com/opensearch-project/opensearch-api-specification/blob/main/OpenSearch.openapi.json). All changes to the API actions should be done via the this generator. If you find an error in the API actions, it most likely comes from the spec. So, please submit a new issue to the [OpenSearch API specification](https://github.com/opensearch-project/opensearch-api-specification/issues/new/choose) repo first.
+
+---
+
+### Usage
+This generator should be run everytime the OpenSearch API Specification is updated to propagate the changes to the Ruby client. For now, this must be done manually:
+- Create a new branch from `main`
+- Download the latest OpenSearch API Specification from [The API Spec Repo](https://github.com/opensearch-project/opensearch-api-specification/blob/main/OpenSearch.openapi.json)
+- Run the generator with the API Spec downloaded previously (see below)
+- Run Rubocop with `-a` flag to remove redundant spacing from the generated code `rubocop -a`
+- Commit and create a PR to merge the updated API actions into `main`.
+
+### Generate API Actions
+Install all dependencies
+```bash
+bundle install
+```
+
+Import the API Generator and load the OpenSearch OpenAPI specification into a generator instance
+```ruby
+require './lib/api_generator'
+generator = ApiGenerator.new('./OpenSearch.openapi.json')
+```
+
+The `generate` method accepts the path to the root directory of the `opensearch-ruby` gem as a parameter. By default, it points to the parent directory of the folder containing the generator script. For example to generate all actions into the `tmp` directory:
+```ruby
+generator.generate('./tmp')
+```
+
+You can also target a specific API version by passing in the version number as a parameter. For example to generate all actions for version `1.0` into the `tmp` directory:
+```ruby
+generator.generate(version: '1.0')
+```
+
+The generator also support incremental generation. For example, to generate all actions of the `cat` namespace:
+```ruby
+generator.generate(namespace: 'cat')
+```
+
+To limit it to specific actions of a namespace:
+```ruby
+generator.generate(namespace: 'cat', actions: %w[aliases allocation])
+```
+
+Note that the root namespace is presented by an empty string `''`. For example, to generate all actions of the root namespace for OS version 2.3:
+```ruby
+generator.generate(version: '2.3', namespace: '')
+```

api_generator/gemfile

@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
+
+gem 'rake'
+gem 'rubocop', '~> 1.44', require: false
+gem 'rubocop-rake', require: false
+gem 'openapi3_parser'
+gem 'mustache', '~> 1'
+gem 'awesome_print'
+gem 'activesupport', '~> 7'

api_generator/lib/action.rb

@@ -0,0 +1,52 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# frozen_string_literal: true
+
+require_relative 'operation'
+require_relative 'version'
+require_relative 'parameter'
+
+# A collection of operations that comprise a single API Action
+class Action
+  attr_reader :group, :name, :namespace, :http_verbs, :urls, :description, :external_docs,
+              :parameters, :path_params, :query_params,
+              :body, :body_description, :body_required
+
+  # @param [Array<Operation>] operations
+  def initialize(operations)
+    @operations = operations
+    @group = operations.first.group
+    @name = operations.first.action
+    @namespace = operations.first.namespace
+    @http_verbs = operations.map(&:http_verb).uniq
+    @urls = operations.map(&:url).uniq
+    @description = operations.map(&:description).find(&:present?)
+    @external_docs = operations.map(&:external_docs).find(&:present?)
+    @external_docs = nil if @external_docs == 'https://opensearch.org/docs/latest'
+
+    dup_params = operations.flat_map(&:parameters)
+    @path_params = dup_params.select { |p| p.in == 'path' }
+    path_param_names = @path_params.map(&:name).to_set
+    @query_params = dup_params.select { |p| p.in == 'query' && !path_param_names.include?(p.name) }
+    @parameters = @path_params + @query_params
+    @parameters.each { |p| p.spec.node_data['required'] = p.name.in?(required_components) }
+
+    @body = operations.map(&:request_body).find(&:present?)
+    @body_required = 'body'.in?(required_components)
+    @body_description = @body&.content&.[]('application/json')&.schema&.description if @body.present?
+  end
+
+  # @return [Set<String>] The names of input components that are required by the action.
+  # A component is considered required if it is required by all operations that make up the action.
+  def required_components
+    @required_components ||= @operations.map do |op|
+      set = Set.new(op.parameters.select(&:required?).map(&:name))
+      set.add('body') if op.request_body&.required?
+      set
+    end.reduce(&:intersection)
+  end
+end

api_generator/lib/action_generator.rb

@@ -0,0 +1,127 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# frozen_string_literal: true
+
+require_relative 'base_generator'
+require_relative 'action'
+
+# Generate an API Action via Mustache
+class ActionGenerator < BaseGenerator
+  self.template_file = './templates/action.mustache'
+  attr_reader :module_name, :method_name, :valid_params_constant_name,
+              :method_description, :argument_descriptions, :external_docs
+
+  # Actions that use perform_request_simple_ignore_404
+  SIMPLE_IGNORE_404 = %w[exists
+                         indices.exists
+                         indices.exists_alias
+                         indices.exists_template
+                         indices.exists_type].to_set.freeze
+
+  # Actions that use perform_request_complex_ignore_404
+  COMPLEX_IGNORE_404 = %w[delete
+                          get
+                          indices.flush_synced
+                          indices.delete_template
+                          indices.delete
+                          security.get_role
+                          security.get_user
+                          snapshot.status
+                          snapshot.get
+                          snapshot.get_repository
+                          snapshot.delete_repository
+                          snapshot.delete
+                          update
+                          watcher.delete_watch].to_set.freeze
+
+  # Actions that use perform_request_ping
+  PING = %w[ping].to_set.freeze
+
+  # @param [Pathname] output_folder
+  # @param [Action] action
+  def initialize(output_folder, action)
+    super(output_folder)
+    @action = action
+    @urls = action.urls.map { |u| u.split('/').select(&:present?) }.uniq
+    @external_docs = action.external_docs
+    @module_name = action.namespace&.camelize
+    @method_name = action.name.underscore
+    @valid_params_constant_name = "#{action.name.upcase}_QUERY_PARAMS"
+    @method_description = action.description
+    @argument_descriptions = params_desc + [body_desc].compact
+  end
+
+  def url_components
+    @urls.max_by(&:length)
+         .map { |e| e.starts_with?('{') ? "_#{e[/{(.+)}/, 1]}" : "'#{e}'" }
+         .join(', ')
+  end
+
+  def http_verb
+    case @action.http_verbs.sort
+    when %w[get post]
+      'body ? OpenSearch::API::HTTP_POST : OpenSearch::API::HTTP_GET'
+    when %w[post put]
+      diff_param = @urls.map(&:to_set).sort_by(&:size).reverse.reduce(&:difference).first
+      "_#{diff_param[/{(.+)}/, 1]} ? OpenSearch::API::HTTP_PUT : OpenSearch::API::HTTP_POST"
+    else
+      "OpenSearch::API::HTTP_#{@action.http_verbs.first.upcase}"
+    end
+  end
+
+  def required_args
+    @action.required_components.map { |arg| { arg: } }
+           .tap { |args| args.last&.[]=('_blank_line', true) }
+  end
+
+  def path_params
+    @action.path_params.map { |p| { name: p.name, listify: p.is_array } }
+           .tap { |args| args.last&.[]=('_blank_line', true) }
+  end
+
+  def query_params
+    @action.query_params.map { |p| { name: p.name } }
+  end
+
+  def listify_query_params
+    @action.query_params.select(&:is_array).map { |p| { name: p.name } }
+           .tap { |args| args.first&.[]=('_blank_line', true) }
+  end
+
+  def perform_request
+    args = 'method, url, params, body, headers'
+    return "perform_request_simple_ignore_404(#{args})" if SIMPLE_IGNORE_404.include?(@action.group)
+    return "perform_request_complex_ignore_404(#{args}, arguments)" if COMPLEX_IGNORE_404.include?(@action.group)
+    return "perform_request_ping(#{args})" if PING.include?(@action.group)
+    "perform_request(#{args}).body"
+  end
+
+  private
+
+  def output_file
+    create_folder(*[@output_folder, @action.namespace].compact).join("#{@action.name}.rb")
+  end
+
+  def params_desc
+    @action.parameters.map do |p|
+      { data_type: p.ruby_type,
+        name: p.name,
+        required: p.required?,
+        description: p.description,
+        default: p.default,
+        deprecated: p.deprecated? }
+    end
+  end
+
+  def body_desc
+    return unless @action.body.present?
+    { data_type: :Hash,
+      name: :body,
+      description: @action.body_description,
+      required: @action.body_required }
+  end
+end

api_generator/lib/api_generator.rb

@@ -0,0 +1,59 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+# frozen_string_literal: true
+
+require 'openapi3_parser'
+require_relative 'action'
+require_relative 'action_generator'
+require_relative 'spec_generator'
+require_relative 'namespace_generator'
+require_relative 'index_generator'
+
+# Generate API endpoints for OpenSearch Ruby client
+class ApiGenerator
+  HTTP_VERBS = %w[get post put patch delete patch head].freeze
+
+  # @param [String] openapi_spec location of the OpenSearch API spec file [required]
+  def initialize(openapi_spec)
+    @spec = Openapi3Parser.load_file(openapi_spec)
+  end
+
+  # @param [String] gem_folder location of the API Gem folder (default to the parent folder of the generator)
+  # @param [String] version target OpenSearch version to generate like "2.5" or "3.0"
+  # @param [String] namespace namespace to generate (Default to all namespaces. Use '' for root)
+  # @param [Array<String>] actions list of actions in the specified namespace to generate (Default to all actions)
+  def generate(gem_folder = '../', version: nil, namespace: nil, actions: nil)
+    gem_folder = Pathname gem_folder
+    namespaces = existing_namespaces(gem_folder)
+    target_actions(version, namespace, actions).each do |action|
+      ActionGenerator.new(gem_folder.join('lib/opensearch/api/actions'), action).generate
+      SpecGenerator.new(gem_folder.join('spec/opensearch/api/actions'), action).generate
+      NamespaceGenerator.new(gem_folder.join('lib/opensearch/api/namespace'), action.namespace).generate(namespaces)
+    end
+    IndexGenerator.new(gem_folder.join('lib/opensearch'), namespaces).generate
+  end
+
+  private
+
+  def target_actions(version, namespace, actions)
+    namespace = namespace.to_s
+    actions = Array(actions).map(&:to_s).to_set unless actions.nil?
+
+    operations = @spec.paths.flat_map do |url, path|
+      path.to_h.slice(*HTTP_VERBS).compact.map do |verb, operation_spec|
+        operation = Operation.new operation_spec, url, verb
+        operation.part_of?(version, namespace, actions) ? operation : nil
+      end
+    end.compact
+
+    operations.group_by(&:group).values.map { |ops| Action.new ops }
+  end
+
+  def existing_namespaces(gem_folder)
+    gem_folder.join('lib/opensearch/api/actions').children.select(&:directory?).map(&:basename).map(&:to_s).to_set
+  end
+end