Skip to content

1. Install self-hosted EraSearch

Estimated time to read: 5 minutes

Self-hosted EraSearch lets you run and manage EraSearch on your own cloud infrastructure. Use it to integrate EraSearch into your existing setup, lower costs, and get the most out of your log data.

This two-part series is intended for developer audiences, and it outlines how to install and use self-hosted EraSearch. In this series, you’ll:

  • Set up and run EraSearch on your cloud infrastructure.
  • Write data to EraSearch.
  • Explore and query data using the EraSearch API.

Note

This series works with self-hosted EraSearch. If you're looking for the fastest way to set up and use EraSearch, get started with EraSearch on EraCloud.

Before you begin

Before you start, reach out to us at Era Software to get the following:

  • eradb-x.x.x.tgz - Compressed files with Helm charts for EraSearch.
  • values-eradb.yaml - The Helm chart values file for EraSearch, customized for your deployment.
  • eradb-registry-xxxx.json - Authentication credentials for Era Software's container registry.

This guide assumes you're comfortable with Kubernetes, Helm, and EraSearch's high-level architecture. It also assumes you have the following in your environment:

  • Kubernetes version 1.20+ with persistent volumes, and Helm.
  • kubectl and the Helm CLI.

    Make sure they're installed on the machine you'll be deploying from.

  • An object storage provider.

    Self-hosted EraSearch works with Amazon Web Services (AWS) S3, Google Cloud Storage, Microsoft Azure Blob Storage, or an API-compatible alternative.

Step 1: Set up the Kubernetes namespace

Follow these steps to create a Kubernetes namespace for the EraSearch deployment:

  1. In your terminal, enter this command to create a new namespace, replacing NAMESPACE_NAME with your namespace name.

    $ kubectl create namespace NAMESPACE_NAME
    
  2. Create a secret for storing Era Software's container registry credentials in the namespace. Enter this command, replacing:

    • NAMESPACE_NAME with the namespace name you specified above.
    • REGISTRY_SECRET with the name of the JSON credential file you got from Era Software.
    • REGISTRY_EMAIL with the email address in that same file.
    $ kubectl create secret docker-registry eradb-registry \
        -n NAMESPACE_NAME \
        --docker-server=https://us.gcr.io \
        --docker-username=_json_key \
        --docker-password="$(cat path/to/REGISTRY_SECRET)" \
        --docker-email=REGISTRY_EMAIL
    

Step 2: Configure the Helm deployment

Make sure the values in values-eradb.yaml are right for your EraSearch deployment.

For every database service, review these values:

  • image.repository

    You may need to update the values if you're mirroring images from an internal registry instead of directly pulling them from Era Software's registry.

  • replicaCount, resources.cpu, and resources.memory

    Make sure the values are valid for your Kubernetes cluster and deployment size.

Next, review these maxwell.treasurer values, making sure they're valid for your environment and object storage budget:

  • monthly_budget

    The maximum USD amount you want to spend on object storage per month.

  • batch_size

    The maximum batch size, in bytes, after which EraSearch flushes data to object storage.

  • batch_delay_ms

    The time, in milliseconds, after which EraSearch flushes data to object storage.

For AWS deployments

If you're using AWS, also review the following:

  • maxwell.s3.region and maxwell.s3.bucket

    Set the values to the right bucket and region for your deployment.

  • maxwell.extraEnv

    Make sure this section includes valid AWS credentials under the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. For production deployments, Era Software recommends using Kubernetes service accounts with IAM pod roles instead of environment-provided credentials.

  • quarry.persistence.storageClass

    Set the value to the supported storage class. Era Software recommends starting with gp2. Visit AWS's Storage classes for more information.

For Microsoft Azure deployments

If you're using Microsoft Azure, also review the following:

  • maxwell.s3

    Make sure the section specifies your Azure storage bucket, region, and endpoint (including the URI and SAS token). You may also want to check the network security settings for your storage account, ensuring EraSearch deployment requests can authenticate.

    s3:
     bucket: your-container-name
     region: eastus2
     endpoint: https://XXXX.blob.core.windows.net/?sv=2020-08-04&ss=b&srt=sco&sp=rwdlacitfx&se=2023-03-04T14:33:22Z&st=2022-03-04T06:33:22Z&spr=https&sig=XXXXXXXX
    

    Note

    While the name s3 refers to Amazon's object storage service, the specification still works with Microsoft Azure Blob Storage.

  • maxwell.extraEnv

    In this section, set the following array elements:

    extraEnv:
     - name: MAXWELL_STORAGE
       value: azure_blob
     - name: MAXWELL_AZURE_CONTAINER
       value: your-container-name
    
  • quarry.persistence.storageClass

    Set the value to supported storage class. Era Software recommends starting with managed-premium unless you need provisioned IOPS. Visit Microsoft Azure's Storage classes for more information.

Step 3: Install EraSearch

With the values set and validated, follow these steps to install EraSearch:

  1. Enter the command below in your terminal, replacing:

    • NAME with a name for the EraSearch database release (for example, era).
    • X.X.X with the version of the Helm chart you got from Era Software.
    • NAMESPACE_NAME with the Kubernetes namespace you created above.
    $ helm install NAME ./eradb-X.X.X.tgz \
        --values ./values-eradb.yaml \ 
        --namespace NAMESPACE_NAME
    
  2. The installation takes a few minutes to complete. Enter this command to check its status:

    $ kubectl rollout status deployment NAME-quarry-deployment -n NAMESPACE_NAME
    

    Once the installation completes, you'll see this message:

    deployment "NAME-quarry-deployment" successfully rolled out
    

Step 4: Verify your installation

Follow these steps to verify your EraSearch installation:

  1. Connect to the quarry-deployment service. Enter the command below to open a local port forward, replacing:

    • NAMESPACE_NAME with the Kubernetes namespace you used above.
    • NAME with the name of the EraSearch database release you used above.
    $ kubectl port-forward -n NAMESPACE_NAME svc/NAME-quarry-deployment 9200:9200
    
  2. Visit http://localhost:9200. When EraSearch is deployed and the API is ready for data, you'll see this JSON document with metadata about your instance:

    {
       "name": "XXXXX",
       "cluster_name": "erasearch_cluster",
       "cluster_uuid": "XXXXX",
       "version": {
          "number": "XXXXX",
          [...]
          "lucene_version": "XXXXX"
       },
       "tagline": "You Know, for Search"
    }
    

You're all set. You've now installed self-hosted EraSearch in your environment.

Next steps

Next, visit 2. Write and query data to get the basics and use the EraSearch REST API. If you're ready to integrate EraSearch into your existing setup, visit the list of tools for writing real-time data.


Last update: November 28, 2022