Getting Started with Elasticsearch, Amazon OpenSearch, NEST and .NET

You click the button and you can’t believe what your eyes see — data. But, not just data, lots of data and that data was returned in a blink of an eye. It seems to good to be true, and it would be if the data was fetched from an average SQL based database. But, in this case, the data came from Elasticsearch!

Elasticsearch is a distributed, open search, and analytics engine for textual, numerical, geospatial, structured, and unstructured data. Elasticsearch is known for its speed and scalability as well as its simple REST APIs that make integrations fairly intuitive. NEST, the official .NET client for Elasticsearch, takes it one step further, making developing Elasticsearch integrations even easier.

---

---

The Solution

In this tutorial, we’ll build out simple .NET applications that demonstrate how easy one can get started with Elasticsearch.

Remember, for any example solution from AWS with .NET, we focus on the code that exemplifies the problem we are trying to solve. We don’t include logging, input validation, exception handling, etc., and we embed the configuration data within classes instead of using environment variables, configuration files, key/value stores and the like. These items should not be skipped for proper solutions.

Prerequisites

To complete this solution, you will need the .NET CLI which is included in the .NET SDK. In addition, you will need to install Docker and Docker Compose.

Warning: some AWS services may have fees associated with them.

Infrastructure

Before we dig into the code, let’s create a docker compose file, named, docker-compose.yaml that will let us spin up the infrastructure that we need. The contents of the docker-compose.yaml file should have the following content:

Notes:

• The Elasticsearch container uses a lot of memory. My test rig has 32GB of RAM and Elastic seems to be using about half of it.
• Using Kibana is optional, but provides a great tool to test your solution and give insight into the data that Elasticsearch contains.
• The docker-compose.yaml file was tested on a rig that runs Ubuntu 20.04.

Abstractions

First, let’s create our abstractions. The first step is to create a class library that will hold all of our interfaces for the sample applications.

$ dotnet new classlib ––name Abstractions

This library has one dependency, NEST, and we will add it like so:

$ dotnet add Abstractions/ package NEST ––version 7.12.1

For this solution we will need two interfaces, IElasticSearchClientFactory and IElasticSearchConnectionSettingsFactory.

Let’s first take a look at IElasticSearchClientFactory. The IElasticSearchClientFactory defines one method, GetClient.

Our second interface, IElasticSearchConnectionSettingsFactory, defines one method named, GetSettings.

Common Domain

Next we’ll create a library that will contain common models that are used by various constructs. We will create a class library named, Domain, that will contain the models.

$ dotnet new classlib ––name Domain

For this example solution, we will have only one model, Vehicle.

NEST is a high level client that maps all requests and responses as types, and comes with a strongly typed query DSL that maps 1 to 1 with the Elasticsearch query DSL.

www.elastic.co

Elasticsearch/NEST Implementations

Let’s now create implementations from the aforementioned interfaces. First, let’s create a class library that will be home to these implementations.

$ dotnet new classlib ––name Elasticsearch

This library also has a dependency on NEST, and we will add it like so:

$ dotnet add Elasticsearch/ package NEST ––version 7.12.1

The Elasticsearch project also has a dependency on the Abstractions project as well as the Domain project and we can add the references through the CLI.

$ dotnet add Elasticsearch/ reference Abstractions/ Domain/

The first implementation, ElasticSearchStaticConnectionSettingsFactory, will implement the IElasticSearchConnectionSettingsFactory. This implementation is responsible for providing the settings to connect to an Elasticsearch datastore. As you can see here, the connection data is stored directly in the class. A better implementation would be to pull the connection data from configuration files or a configuration store like Consul. You can also see that we set up a default mapping for the Vehicle type. You can find out more about default mapping in this article about index name inference.

The second implementation, ElasticSearchClientFactory, implements the IElasticSearchClientFactory interface. ElasticSearchClientFactory takes a dependency on an IElasticSearchConnectionSettingsFactory based class and is responsible for creating and providing the Elasticsearch client.

These factories are very simple, as we are building prototype level applications. Production level applications would require factories that are much more complex.

Data Creation Application

With our supporting libraries and constructs in place we can now build out our example apps.

The first example application that we will build will create data in the Elasticsearch data store. We’ll create the console app using the following command:

$ dotnet new console ––name DataCreate

This application will require some dependencies and we can set the dependencies like so:

$ dotnet add DataCreate/ reference Abstractions/ Domain/ Elasticsearch/

We’ll now open the Program.cs file within the newly created console app and start changing the Main method.

First, we will need to instantiate an IElasticSearchConnectionSettingsFactory based class, ElasticSearchStaticConnectionSettingsFactory.

We then instantiate an IElasticSearchClientFactory based class, ElasticSearchClientFactory, passing in the ElasticSearchStaticConnectionSettingsFactory class.

We can now create a local variable of type IElasticClient, named client, using the GetClient method from the ElasticSearchClientFactory class.

With the client in place, let’s instantiate the data that we want to create.

Now, with the client set and the data that we want to create instantiated, we can use the client to send the data to the Elasticsearch data store.

Let’s take a look at the application in its entirety.

Data Search Application

The next example application that we will build will search for data in the Elasticsearch data store. We’ll create the console app using the following command at the CLI:

$ dotnet new console ––name DataSearch

This application will also require some dependencies and we can set the dependencies like so:

$ dotnet add DataSearch/ reference Abstractions/ Domain/ Elasticsearch/

Again, we’ll need to open the Program.cs file within the newly created console app and start editing the Main method.

Just like we did in the previous app, we need to instantiate the Elasticsearch client following the steps below.

Instantiate an IElasticSearchConnectionSettingsFactory based class, ElasticSearchStaticConnectionSettingsFactory.

We then instantiate an IElasticSearchClientFactory based class, ElasticSearchClientFactory, passing in the ElasticSearchStaticConnectionSettingsFactory class.

Create a local variable of type IElasticClient, named client, using the GetClient method from the ElasticSearchClientFactory class.

Now that we have a client, let’s create a search request. Here we will instantiate a new SearchRequest object and set a few properties.

The From property, defines the offset from the first result you want to fetch. In this example, we are going to start at the beginning and we will use the value of 0.

The Size property configures the maximum hits to be returned. Think of a hit as a search result.

We’ll also use a Match query to fetch the data, querying for hits where the vehicle make equals “BMW”.

Now that the SearchRequest object is ready, we’ll send the request and set the result to a local variable.

Let’s take a look at the complete Program.cs file.

You click the button and you can’t believe what your eyes see — data. But, not just data, lots of data and that data was returned in a blink of an eye.

With all of the projects in place, let’s create a .NET solution for all the projects, add the projects and then kick off a build for the solution.

Create the solution file:

$ dotnet new sln ––name elasticsearch

Add the projects to the solution file:

$ dotnet sln add Abstractions/ Domain/ Elasticsearch/ DataCreate/ DataSearch/

Build the solution:

$ dotnet build

Test Drive

Now that we have two test apps built, let’s take things for a test drive.

First, let’s start by pulling up the infrastructure using docker compose. Run the following command from the command line within the directory/folder that contains the docker compose file. This command will install and spin up the Elasticsearch datastore as well as Kibana.

On Linux and MacOS based machines, use the following syntax:

$ docker-compose up

Windows systems require the following syntax:

$ docker compose up

With an Elasticsearch datastore now available for us to test, let’s create some data with the first app we created by running the following command:

$ dotnet run ––project ./DataCreate

When this command completes you should see a message like the following. Hit the enter key to terminate the app.

Id: 4dd6a196-8698-4d81-b10c-68351a333cb0

With the data now in place, we can query the data by running the DataSearch app.

$ dotnet run ––project ./DataSearch

On the completion of the command, you can see the data that was just added with the DataCreate app. Hit the enter key to terminate the app.

Vehicle Id: 4dd6a196-8698-4d81-b10c-68351a333cb0
Year: 2020
Make: BMW
Model: 330i

Challenges

Now that you have a working solution, let’s take things a little further…

• Modify the solution to use configuration and dependency injection.
• Rerun the DataCreate app and modify the From and Size properties of the Search Request in the DataSearch app, then rerun that app to see the changes in the data that is returned.
• Create apps or modify the previous apps to update data as well as delete data.
• Test this solution on AWS OpenSearch (see below).
• Use Kibana to create, update, delete and query data in your Elasticsearch datastore.

Moving to Amazon OpenSearch

Amazon now provides a straightforward article on how to get started with Amazon OpenSearch (the new Amazon ES). Follow the directions in that article to get setup and note the points about development environments.

We have completed the solution, learning to spin up a local Elasticsearch datastore as well as a local Kibana install. You also learned how to create and search for data in an Elasticsearch datastore using NEST and .NET.

Elasticsearch is a distributed open search and analytics engine for textual, numerical, geospatial, structured, and unstructured data

Want to know more about Elasticsearch, NEST, and OpenSearch?

Elasticsearch

Elasticsearch.Net and NEST Introduction

Amazon OpenSearch Service

Get all the source code at GitHub!