Getting Started with Elasticsearch, Amazon OpenSearch, NEST and .NET
You click the button and you can’t believe what your eyes see – data. But, not just data, lots of data and that data was returned in a blink of an eye. It seems to good to be true, and it would be if the data was fetched from an average SQL based database. But, in this case, the data came from Elasticsearch!
Elasticsearch is a distributed, open search, and analytics engine for textual, numerical, geospatial, structured, and unstructured data. Elasticsearch is known for its speed and scalability as well as its simple REST APIs that make integrations fairly intuitive. NEST, the official .NET client for Elasticsearch, takes it one step further, making developing Elasticsearch integrations even easier.
The Solution
In this tutorial, we’ll build out simple .NET applications that demonstrate how easy one can get started with Elasticsearch.
Remember, for any example solution from AWS with .NET, we focus on the code that exemplifies the problem we are trying to solve. We don’t include logging, input validation, exception handling, etc., and we embed the configuration data within classes instead of using environment variables, configuration files, key/value stores and the like. These items should not be skipped for proper solutions.
Prerequisites
To complete this solution, you will need the .NET CLI which is included in the .NET SDK. In addition, you will need to install Docker and Docker Compose.
Warning: some AWS services may have fees associated with them.
Infrastructure
Before we dig into the code, let’s create a docker compose file, named, docker-compose.yaml that will let us spin up the infrastructure that we need. The contents of the docker-compose.yaml file should have the following content:
version: "3.5"
services:
elasticsearch-local:
image: "docker.elastic.co/elasticsearch/elasticsearch:7.12.1"
container_name: elasticsearch-local
environment:
– discovery.type=single-node
volumes:
– ~/docker-volumes/elasticsearch:/usr/share/elasticsearch/data
ports:
– 9200:9200
kibana-local:
image: "docker.elastic.co/kibana/kibana:7.12.0"
container_name: kibana-local
environment:
SERVER_NAME: "kibana.example.org"
ELASTICSEARCH_HOSTS: "http://elasticsearch-local:9200"
ports:
– 5601:5601
Notes:
- The Elasticsearch container uses a lot of memory. My test rig has 32GB of RAM and Elastic seems to be using about half of it.
- Using Kibana is optional, but provides a great tool to test your solution and give insight into the data that Elasticsearch contains.
- The docker-compose.yaml file was tested on a rig that runs Ubuntu 20.04.
Abstractions
First, let’s create our abstractions. The first step is to create a class library that will hold all of our interfaces for the sample applications.
$ dotnet new classlib --name Abstractions
This library has one dependency, NEST, and we will add it like so:
$ dotnet add Abstractions/ package NEST --version 7.12.1
For this solution we will need two interfaces, IElasticSearchClientFactory and IElasticSearchConnectionSettingsFactory.
Let’s first take a look at IElasticSearchClientFactory. The IElasticSearchClientFactory defines one method, GetClient.
using Nest;
namespace Abstractions
{
public interface IElasticSearchClientFactory
{
IElasticClient GetClient();
}
}
Our second interface, IElasticSearchConnectionSettingsFactory, defines one method named, GetSettings.
using Nest;
namespace Abstractions
{
public interface IElasticSearchConnectionSettingsFactory
{
ConnectionSettings GetSettings();
}
}
Common Domain
Next we’ll create a library that will contain common models that are used by various constructs. We will create a class library named, Domain, that will contain the models.
$ dotnet new classlib --name Domain
For this example solution, we will have only one model, Vehicle.
namespace Domain
{
public class Vehicle
{
public Guid Id { get; set; }
public short Year { get; set; }
public string Make { get; set; }
public string Model { get; set; }
}
}
NESTis a high level client that maps all requests and responses as types, and comes with a strongly typed query DSL that maps 1 to 1 with the Elasticsearch query DSL.
Elasticsearch/NEST Implementations
Let’s now create implementations from the aforementioned interfaces. First, let’s create a class library that will be home to these implementations.
$ dotnet new classlib --name Elasticsearch
This library also has a dependency on NEST, and we will add it like so:
$ dotnet add Elasticsearch/ package NEST --version 7.12.1
The Elasticsearch project also has a dependency on the Abstractions project as well as the Domain project and we can add the references through the CLI.
$ dotnet add Elasticsearch/ reference Abstractions/ Domain/
The first implementation, ElasticSearchStaticConnectionSettingsFactory, will implement the IElasticSearchConnectionSettingsFactory. This implementation is responsible for providing the settings to connect to an Elasticsearch datastore. As you can see here, the connection data is stored directly in the class. A better implementation would be to pull the connection data from configuration files or a configuration store like Consul. You can also see that we set up a default mapping for the Vehicle type. You can find out more about default mapping in this article about index name inference.
using System;
using Abstractions;
using Domain;
using Elasticsearch.Net;
using Nest;
namespace Elasticsearch
{
public class ElasticSearchStaticConnectionSettingsFactory :
IElasticSearchConnectionSettingsFactory
{
public ConnectionSettings GetSettings()
{
var elasticUris = new[]{
new Uri("http://localhost:9200"),
};
var connectionPool = new SingleNodeConnectionPool(elasticUris[0]);
var settings = new ConnectionSettings(connectionPool)
.DefaultMappingFor<Vehicle>(
m =>
{
m.IndexName("vehicles");
m = m.IdProperty("Id");
return m;
}
);
return settings;
}
}
}
The second implementation, ElasticSearchClientFactory, implements the IElasticSearchClientFactory interface. ElasticSearchClientFactory takes a dependency on an IElasticSearchConnectionSettingsFactory based class and is responsible for creating and providing the Elasticsearch client.
using Abstractions;
using Nest;
namespace Elasticsearch
{
public class ElasticSearchClientFactory : IElasticSearchClientFactory
{
private readonly IElasticSearchConnectionSettingsFactory
_connectionSettingsFactory;
public ElasticSearchClientFactory(
IElasticSearchConnectionSettingsFactory connectionSettingsFactory
)
{
_connectionSettingsFactory = connectionSettingsFactory;
}
public IElasticClient GetClient()
{
ConnectionSettings settings = _connectionSettingsFactory.GetSettings();
IElasticClient client = new ElasticClient(settings);
return client;
}
}
}
These factories are very simple, as we are building prototype level applications. Production level applications would require factories that are much more complex.
Data Creation Application
With our supporting libraries and constructs in place we can now build out our example apps.
The first example application that we will build will create data in the Elasticsearch data store. We’ll create the console app using the following command:
$ dotnet new console --name DataCreate
This application will require some dependencies and we can set the dependencies like so:
$ dotnet add DataCreate/ reference Abstractions/ Domain/ Elasticsearch/
We’ll now open the Program.cs file within the newly created console app and start changing the Main method.
First, we will need to instantiate an IElasticSearchConnectionSettingsFactory based class, ElasticSearchStaticConnectionSettingsFactory.
IElasticSearchConnectionSettingsFactory elasticSearchConnectionSettingsFactory = new
ElasticSearchStaticConnectionSettingsFactory();
We then instantiate an IElasticSearchClientFactory based class, ElasticSearchClientFactory, passing in the ElasticSearchStaticConnectionSettingsFactory class.
IElasticSearchClientFactory elasticSearchClientFactory =
new ElasticSearchClientFactory(elasticSearchConnectionSettingsFactory);
We can now create a local variable of type IElasticClient, named client, using the GetClient method from the ElasticSearchClientFactory class.
IElasticClient client = elasticSearchClientFactory.GetClient();
With the client in place, let’s instantiate the data that we want to create.
Vehicle vehicle = new Vehicle{
Id = Guid.NewGuid(),
Year = 2020,
Make = "BMW",
Model = "330i"
};
Now, with the client set and the data that we want to create instantiated, we can use the client to send the data to the Elasticsearch data store.
IndexResponse asyncIndexResponse = await client.IndexDocumentAsync(vehicle);
Let’s take a look at the application in its entirety.
using System;
using System.Threading.Tasks;
using Abstractions;
using Domain;
using Nest;
using Elasticsearch;
namespace DataCreate
{
class Program
{
static async Task Main(string[] args)
{
IElasticSearchConnectionSettingsFactory
elasticSearchConnectionSettingsFactory =
new ElasticSearchStaticConnectionSettingsFactory();
IElasticSearchClientFactory elasticSearchClientFactory =
new ElasticSearchClientFactory(elasticSearchConnectionSettingsFactory);
IElasticClient client = elasticSearchClientFactory.GetClient();
Vehicle vehicle = new Vehicle{
Id = Guid.NewGuid(),
Year = 2020,
Make = "BMW",
Model = "330i"
};
IndexResponse asyncIndexResponse = await client.IndexDocumentAsync(vehicle);
Console.WriteLine("Id: " + asyncIndexResponse.Id);
Console.ReadLine();
}
}
}
Data Search Application
The next example application that we will build will search for data in the Elasticsearch data store. We’ll create the console app using the following command at the CLI:
$ dotnet new console --name DataSearch
This application will also require some dependencies and we can set the dependencies like so:
$ dotnet add DataSearch/ reference Abstractions/ Domain/ Elasticsearch/
Again, we’ll need to open the Program.cs file within the newly created console app and start editing the Main method.
Just like we did in the previous app, we need to instantiate the Elasticsearch client following the steps below.
Instantiate an IElasticSearchConnectionSettingsFactory based class, ElasticSearchStaticConnectionSettingsFactory.
IElasticSearchConnectionSettingsFactory elasticSearchConnectionSettingsFactory =
new ElasticSearchStaticConnectionSettingsFactory();
We then instantiate an IElasticSearchClientFactory based class, ElasticSearchClientFactory, passing in the ElasticSearchStaticConnectionSettingsFactory class.
IElasticSearchClientFactory elasticSearchClientFactory =
new ElasticSearchClientFactory(elasticSearchConnectionSettingsFactory);
Create a local variable of type IElasticClient, named client, using the GetClient method from the ElasticSearchClientFactory class.
IElasticClient client = elasticSearchClientFactory.GetClient();
Now that we have a client, let’s create a search request. Here we will instantiate a new SearchRequest object and set a few properties.
The From property, defines the offset from the first result you want to fetch. In this example, we are going to start at the beginning and we will use the value of 0.
The Size property configures the maximum hits to be returned. Think of a hit as a search result.
We’ll also use a Match query to fetch the data, querying for hits where the vehicle make equals “BMW”.
var searchRequest = new SearchRequest<Vehicle>()
{
From = 0,
Size = 1,
Query = new MatchQuery
{
Field = Infer.Field<Vehicle>(f => f.Make),
Query = "BMW"
}
};
Now that the SearchRequest object is ready, we’ll send the request and set the result to a local variable.
var searchResponse = await client.SearchAsync<Vehicle>(searchRequest);
Let’s take a look at the complete Program.cs file.
namespace DataSearch
{
class Program
{
static async Task Main(string[] args)
{
IElasticSearchConnectionSettingsFactory
elasticSearchConnectionSettingsFactory = new
ElasticSearchStaticConnectionSettingsFactory();
IElasticSearchClientFactory elasticSearchClientFactory =
new ElasticSearchClientFactory(elasticSearchConnectionSettingsFactory);
IElasticClient client = elasticSearchClientFactory.GetClient();
var searchRequest = new SearchRequest<Vehicle>()
{
From = 0,
Size = 1,
Query = new MatchQuery
{
Field = Infer.Field<Vehicle>(f => f.Make),
Query = "BMW"
}
};
var searchResponse = await client.SearchAsync<Vehicle>(searchRequest);
Vehicle vehicle = searchResponse.Documents.FirstOrDefault();
Console.WriteLine("Vehicle Id: " + vehicle.Id);
Console.WriteLine("Year: " + vehicle.Year);
Console.WriteLine("Make: " + vehicle.Make);
Console.WriteLine("Model: " + vehicle.Model);
Console.ReadLine();
}
}
}
You click the button and you can’t believe what your eyes see — data. But, not just data, lots of data and that data was returned in a blink of an eye.
With all of the projects in place, let’s create a .NET solution for all the projects, add the projects and then kick off a build for the solution.
Create the solution file:
$ dotnet new sln --name elasticsearch
Add the projects to the solution file:
$ dotnet sln add Abstractions/ Domain/ Elasticsearch/ DataCreate/ DataSearch/
Build the solution:
$ dotnet build
Test Drive
Now that we have two test apps built, let’s take things for a test drive.
First, let’s start by pulling up the infrastructure using docker compose. Run the following command from the command line within the directory/folder that contains the docker compose file. This command will install and spin up the Elasticsearch datastore as well as Kibana.
On Linux and MacOS based machines, use the following syntax:
$ docker-compose up
Windows systems require the following syntax:
$ docker compose up
With an Elasticsearch datastore now available for us to test, let’s create some data with the first app we created by running the following command:
$ dotnet run --project ./DataCreate
When this command completes you should see a message like the following. Hit the enter key to terminate the app.
Id: 4dd6a196-8698-4d81-b10c-68351a333cb0
With the data now in place, we can query the data by running the DataSearch app.
$ dotnet run --project ./DataSearch
On the completion of the command, you can see the data that was just added with the DataCreate app. Hit the enter key to terminate the app.
Vehicle Id: 4dd6a196-8698-4d81-b10c-68351a333cb0
Year: 2020
Make: BMW
Model: 330i
Challenges
Now that you have a working solution, let’s take things a little further…
- Modify the solution to use configuration and dependency injection.
- Rerun the DataCreate app and modify the From and Size properties of the Search Request in the DataSearch app, then rerun that app to see the changes in the data that is returned.
- Create apps or modify the previous apps to update data as well as delete data.
- Test this solution on AWS OpenSearch (see below).
- Use Kibana to create, update, delete and query data in your Elasticsearch datastore.
Moving to Amazon OpenSearch
Amazon now provides a straightforward article on how to get started with Amazon OpenSearch (the new Amazon ES). Follow the directions in that article to get setup and note the points about development environments.
We have completed the solution, learning to spin up a local Elasticsearch datastore as well as a local Kibana install. You also learned how to create and search for data in an Elasticsearch datastore using NEST and .NET.
Elasticsearch is a distributed open search and analytics engine for textual, numerical, geospatial, structured, and unstructured data
Want to know more about Elasticsearch, NEST, and OpenSearch?