Configuration reference

Estimated time to read: 17 minutes

This page describes EraSearch's configuration files and options. Use it to customize self-hosted EraSearch to your environment and logging needs.

Configuration files¶

For self-hosted EraSearch¶

Self-hosted EraSearch uses Helm – a package manager for Kubernetes – for configuring and deploying releases into Kubernetes clusters. All self-hosted instances come with these files:

• eradb-registry-xxxx.json - The authentication credentials for pulling images from Era Software's container registry.
• eradb-x.x.x.tgz - Compressed files with Helm charts for installing EraSearch into a Kubernetes cluster.
• values-eradb.yaml - The Helm values file, listing EraSearch's configuration options in YAML. Era Software customizes the file to your environment.

To customize EraSearch, edit the configuration options in the values file and specify that file when deploying EraSearch. The values file overrides the default configuration options in the Helm charts.

For EraSearch on EraCloud¶

Era Software configures and manages EraSearch on EraCloud for you. You don't need to configure or customize your EraSearch instance, and it's ready for production by default.

Using values files¶

To configure self-hosted EraSearch, edit the options in values-eradb.yaml. Then specify the file in the helm install or helm upgrade commands. To view the configuration options for an existing release, use helm get values.

helm install example¶

The command below installs EraSearch into a Kubernetes cluster.

The command installs the eradb-1.2.3.tgz charts into the logs-erasearch namespace. Helm uses the configurations in values-eradb.yaml to override default chart values, and the release is called myerasearch.

$ helm install myerasearch ./eradb-1.2.3.tgz \
    --values ./values-eradb.yaml \ 
    --namespace logs-erasearch

The install command returns this output:

deployment "myerasearch-quarry-deployment" successfully rolled out

helm upgrade example¶

The command below upgrades an existing EraSearch release in a Kubernetes cluster.

The command upgrades the myerasearch release in the logs-erasearch namespace. Helm applies the updated eradb-1.2.3.tgz charts and values-eradb.yaml file to the existing release.

$ helm upgrade myerasearch ./eradb-1.2.3.tgz \
    --values ./values-eradb.yaml \ 
    --namespace logs-erasearch

The upgrade command returns this output along with other deployment details:

Release myerasearch-quarry-deployment has been upgraded. Happy Helming!

helm get values example¶

This command displays the configuration options used in an EraSearch release (myerasearch):

$ helm get values myerasearch

The command returns USER-SUPPLIED VALUES followed by the configuration options. The response is organized by service and lists the options in alphabetical order.

A note on service names¶

This page uses two names for EraSearch services:

• Configuration option names use Era Software's internal names (for example, Maxwell).
• Configuration descriptions use descriptive names (for example, Storage Service).

That naming difference is temporary. In the future, option names and descriptions will use only descriptive names. Until then, here’s how the descriptive names map to internal names:

• Cache Service and API Service ➡ Quarry
• Coordinator Service ➡ Alexandria
• Storage Service ➡ Maxwell

Configuration options¶

The sections below list and describe the configuration options in values-eradb.yaml. They're organized by service.

Cache Service and API Service options¶

This section describes the following Cache Service and API Service options:

quarry:
  image:
    repository: <string>
    tag: <string>
  imagePullSecrets:
    - name: <string>
  replicaCount: <integer>
  resources:
    cpu: <integer>
    memory: <integer>
    disk: <integer>
  persistence:
    storageClass: <string>
  extraEnv:
    - name: <string>
      value: <integer>
  api:
    replicaCount: <integer>
    resources:
      cpu: <integer>
      memory: <integer>

quarry.image.repository¶

quarry.image.repository is the container registry storing the image for API Service and Cache Service.

The default value – set to Era Software's registry – works for most installations. Update this option only if you're mirroring images from an internal registry instead of pulling images from Era Software's registry.

quarry.image.tag¶

quarry.image.tag identifies the relevant API Service and Cache Service image to work with. You only need to update this option if instructed by Era Software support.

quarry.imagePullSecrets¶

quarry.imagePullSecrets references the secret that lets Kubernetes authenticate with Era Software's private registry. This value must match the name of the secret you created in the EraSearch namespace.

quarry.replicaCount¶

quarry.replicaCount is the number of Cache Service pods Kubernetes runs at the same time.

Increasing the value scales Cache Service, letting EraSearch handle larger data volumes and service more queries. The default, 1, is a good starting value for new EraSearch installations.

quarry.resources.cpu¶

quarry.resources.cpu sets compute processing resources, in Kubernetes CPU units, for the Cache Service container.

Your quarry.resources.cpu value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

quarry.resources.memory¶

quarry.resources.memory sets memory resources for the Cache Service container. The value is either a plain integer (in bytes) or an integer with a memory resource unit.

Your quarry.resources.memory value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

quarry.resources.disk¶

quarry.resources.disk sets disk resources for the Cache Service container. The value is either a plain integer (in bytes) or an integer with a disk resource unit.

Your quarry.resources.disk value depends on your expected data volumes, query patterns, and deployment size. Reach out to Era support if you have questions about sizing Cache Service disks.

quarry.persistence.storageClass¶

quarry.persistence.storageClass defines Cache Service's local storage in the Kubernetes cluster. In practice, Kubernetes uses quarry.persistence.storageClass to dynamically provision a persistent volume. Cache Service stores data in that persistent volume and uses it to service query results.

Your storage class depends on your cloud provider. For Amazon S3, Era Software recommends starting with gp2. Visit Amazon's EBS volume types for more information.

For Microsoft Azure Blob Storage, Era Software recommends starting with managed-premium unless you need provisioned IOPS. Visit Microsoft's Storage classes for more information.

quarry.extraEnv¶

quarry.extraEnv defines additional environment variables for EraSearch's Cache Service. Use this section to set an environment variable for optional feature flags, for example, QUARRY_EVICTION_TTL:

extraEnv:
  - name: QUARRY_EVICTION_TTL
    value: 7d

QUARRY_EVICTION_TTL is the time after which Cache Service removes documents from disk. The value is an integer with a human-readable time duration. If you don't set QUARRY_EVICTION_TTL, Cache Service keeps data on disk forever.

If you set QUARRY_EVICTION_TTL, about every minute, Cache Service checks for documents with timestamps older than QUARRY_EVICTION_TTL. It then removes those documents from disk. That removal process is called cache eviction.

Raising QUARRY_EVICTION_TTL results in the following:

• Increases how long Cache Service keeps documents on disk.

• Gives you the fastest possible queries on documents in the longer time range, as more data is available in the local cache.

• Increases disk-resource needs (set by quarry.resources.disk).

Lowering QUARRY_EVICTION_TTL results in the following:

• Decreases how long Cache Service keeps documents on disk.
• Gives you the fastest possible queries on documents in the shorter time range.

You can still query documents not on disk. Those queries are slower because EraSearch has to move the documents from object storage back to disk. - Lowers disk-resource needs (set by quarry.resources.disk).

quarry.api.replicaCount¶

quarry.api.replicaCount is the number of API Service pods Kubernetes runs at the same time.

Raising the value scales API Service, increasing how many write and query requests you can make to EraSearch. The default, 1, is a good starting value for new EraSearch installations.

quarry.api.resources.cpu¶

quarry.api.resources.cpu sets compute processing resources, in Kubernetes CPU units, for the API Service container.

Your quarry.api.resources.cpu value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

quarry.api.resources.memory¶

quarry.api.resources.memory sets memory resources for the API Service container. The value is either a plain integer (in bytes) or an integer with a memory resource unit.

Your quarry.api.resources.memory value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

Coordinator Service options¶

This section describes the following Coordinator Service options:

alexandria:
  image:
    repository: <string>
    tag: <string>
  imagePullSecrets:
    - name: <string>
  replicaCount: <integer>
  resources:
    cpu: <integer>
    memory: <integer>

alexandria.image.repository¶

alexandria.image.repository is the container registry storing the Coordinator Service image.

The default value – set to Era Software's registry – works for most installations. Update this option only if you're mirroring images from an internal registry instead of pulling images from Era Software's registry.

alexandria.image.tag¶

alexandria.image.tag identifies the relevant Coordinator Service image to work with. You only need to update this option if instructed by Era Software support.

alexandria.imagePullSecrets¶

alexandria.imagePullSecrets references the secret that lets Kubernetes authenticate with Era Software's private registry. This value must match the name of the secret you created in the EraSearch namespace.

alexandria.replicaCount¶

alexandria.replicaCount is the number of Coordinator Service pods Kubernetes runs at the same time.

Increasing the value scales Coordinator Service, letting EraSearch more efficiently process large amounts of incoming data. The default, 1, is a good starting value for new EraSearch installations.

alexandria.resources.cpu¶

alexandria.resources.cpu sets compute processing resources, in Kubernetes CPU units, for the Coordinator Service container.

Your alexandria.resources.cpu value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

alexandria.resources.memory¶

alexandria.resources.memory sets memory resources for the Coordinator Service container. The value is either a plain integer (in bytes) or an integer with a memory resource unit.

Your alexandria.resources.memory value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

Redis options¶

Coordinator Service stores object IDs (OIDs) in Redis. This section describes the following Redis configuration options:

stateless-redis:
  master:
    resources:
      limits:
        cpu: <integer>
        memory: <integer>
      requests:
        cpu: <integer>
        memory: <integer>

stateless-redis.master.resources.limits.cpu¶

stateless-redis.master.resources.limits.cpu sets a compute processing resource limit, in Kubernetes CPU units, for the Redis container. This value must be higher than stateless-redis.master.resources.requests.cpu.

Your stateless-redis.master.resources.limits.cpu value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

stateless-redis.master.resources.limits.memory¶

stateless-redis.master.resources.limits.memory sets a memory resource limit for the Redis container. The value is either a plain integer (in bytes) or an integer with a memory resource unit. The value must be higher than stateless-redis.master.resources.requests.memory.

Your stateless-redis.master.resources.limits.memory value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

stateless-redis.master.resources.requests.cpu¶

stateless-redis.master.resources.requests.cpu sets compute processing resources, in Kubernetes CPU units, for the Redis container. This value must be lower than stateless-redis.master.resources.limits.cpu.

Your stateless-redis.master.resources.requests.cpu value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

stateless-redis.master.resources.requests.memory¶

stateless-redis.master.resources.requests.memory sets memory resources for the Redis container. The value is either a plain integer (in bytes) or an integer with a memory resource unit. This value must be lower than stateless-redis.master.resources.limits.memory.

Your stateless-redis.master.resources.requests.memory value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

Storage Service options¶

This section describes the following Storage Service options:

maxwell:
  image:
    repository: <string>
    tag: <string>
  imagePullSecrets:
    - name: <string>
  replicaCount: <integer>
  resources:
    cpu: <integer>
    memory: <integer>
  s3:
    bucket: <string>
    endpoint: <string>    
    region: <string>
  treasurer:
   batch_delay_ms: <integer>
   batch_size: <integer>
   monthly_budget: <integer>
  extraEnv:
    - name: <string>
      value: <string>

maxwell.image.repository¶

maxwell.image.repository is the container registry storing the Storage Service image.

The default value – set to Era Software's registry – works for most installations. Update this option only if you're mirroring images from an internal registry instead of pulling images from Era Software's registry.

maxwell.image.tag¶

maxwell.image.tag identifies the relevant Storage Service image to work with. You only need to update this option if Era Software support suggests it.

maxwell.imagePullSecrets¶

maxwell.imagePullSecrets references the secret that lets Kubernetes authenticate with Era Software's private registry. This value must match the name of the secret you created in the EraSearch namespace.

maxwell.replicaCount¶

maxwell.replicaCount is the number of Storage Service pods Kubernetes runs at the same time.

Increasing the value scales Storage Service, letting EraSearch read and write more data to long-term storage. The default, 1, is a good starting value for new EraSearch installations.

maxwell.resources.cpu¶

maxwell.resources.cpu sets compute processing resources, in Kubernetes CPU units, for the Storage Service container.

Your maxwell.resources.cpu value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

maxwell.resources.memory¶

maxwell.resources.memory sets memory resources for the Storage Service container. The value is either a plain integer (in bytes) or an integer with a memory resource unit.

Your maxwell.resources.memory value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.

maxwell.s3.bucket¶

maxwell.s3.bucket is the container for storing EraSearch data in object storage. Examples of bucket names are mycompany.internal.private.erasearch and erasearch.

maxwell.s3.endpoint¶

maxwell.s3.endpoint is the URL for connecting to objects in your Microsoft Azure Blob Storage account. When configuring this option, make sure your account's network security settings let EraSearch authenticate requests.

This is an example of a Microsoft Azure Blob Storage endpoint (note that it includes the URI and SAS token):

https://mystorageaccount.blob.core.windows.net/?sv=2020-08-04&ss=b&srt=sco&sp=rwdlacitfx&se=2023-03-04T14:33:22Z&st=2022-03-04T06:33:22Z&spr=https&sig=XXXXXXXX

maxwell.s3.region¶

maxwell.s3.region is the geographical area for your object storage. For example, us-east-1 and us-west-2 are Amazon Web Services regions, and eastus2 and westus2 are Microsoft Azure Blob Storage regions.

maxwell.treasurer.batch_delay_ms¶

maxwell.treasurer.batch_delay_ms is the time, in milliseconds, after which the Storage Service flushes data to object storage. In practice, the Storage Service flushes data after the system reaches maxwell.treasurer.batch_delay_ms or maxwell.treasurer.batch_size, whichever happens first.

Increasing maxwell.treasurer.batch_delay_ms lowers data throughput and object-storage costs. Decreasing the value raises data throughput and object-storage costs.

maxwell.treasurer.batch_size¶

maxwell.treasurer.batch_size is the maximum batch size, in bytes, after which the Storage Service flushes data to object storage. In practice, the Storage Service flushes data after the system reaches maxwell.treasurer.batch_size or maxwell.treasurer.batch_delay_ms, whichever happens first.

Increasing maxwell.treasurer.batch_size lowers data throughput and object-storage costs. Decreasing the value raises data throughput and object storage-costs.

maxwell.treasurer.monthly_budget¶

maxwell.treasurer.monthly_budget is the target USD amount you want to spend on object storage per month. This option helps you target a budget by affecting how often Storage Service interacts with object storage.

In practice, Storage Service uses this option to calculate a request limit. The request limit is the number of object-storage requests Storage Service can make per month while staying in budget. If Storage Service's average request rate goes above the request limit, the service lowers the request rate. That change keeps costs closer to the target budget.

When applying maxwell.treasurer.monthly_budget, Storage Service:

• Doesn't flush data as often to object storage, letting Cache Service keep data in local storage longer.
• Overrides maxwell.treasurer.batch_delay_ms and maxwell.treasurer.batch_size to determine the object-storage request rate.
• Reports the new request rate in the maxwell_treasurer_until_ready metric.

If you don't want to use maxwell.treasurer.monthly_budget, you can disable it by setting it to a high number, for example, 60000.

maxwell.extraEnv¶

maxwell.extraEnv defines required environment variables for EraSearch's Storage Service.

For Amazon S3¶

Use the syntax below to set environment variables for your Amazon Web Services (AWS) credentials. Replace YOUR_KEY_ID and YOUR_SECRET with your AWS access key id and secret access key.

extraEnv:
  - name: AWS_ACCESS_KEY_ID
    value: YOUR_KEY_ID
  - name: AWS_SECRET_ACCESS_KEY
    value: YOUR_SECRET

For Microsoft Azure Blob Storage¶

Use the syntax below to set environment variables for your storage provider. Replace YOUR_CONTAINER_NAME with the name of the container storing EraSearch data.

extraEnv:
  - name: MAXWELL_STORAGE
    value: azure_blob
  - name: MAXWELL_AZURE_CONTAINER
    value: YOUR_CONTAINER_NAME

UI Service options¶

This section describes the following UI Service options:

ui:
  enabled: <boolean>
  image:
    tag: <string>
  imagePullSecrets:
    - name:  <string>
  existingSecretName:  <string>
  externalUrl:  <string>
  azureScopeName:  <string>
  ingress:
    enabled: <boolean>
    hosts:
      -  <string>
    tls:
      - hosts:
          -  <string>
        secretName: <string>

ui.enabled¶

ui.enabled controls whether the UI Service deployment is included or not. Set this option to true to deploy the UI Service.

ui.image.tag¶

ui.image.tag identifies the relevant UI Service image to work with. You only need to update this option if instructed by Era Software support.

ui.imagePullSecrets¶

ui.imagePullSecrets references the secret that lets Kubernetes authenticate with Era Software's private registry. This value must match the name of the secret you created in the EraSearch namespace.

ui.existingSecretName¶

ui.existingSecretName references the name of the Kubernetes secret that contains sensitive configuration options for the UI Service. Within this secret, the following keys must be set:

• app-signing-secret - A randomly-generated secret used to securely sign application cookies to prevent tampering.
• postgres-password - A password to use as the master password for the UI Service's backing PostgreSQL database.
• auth-sign-secret - A randomly-generated secret used for the UI Service components to securely communicate.
• azure-client-id - The Azure AD client ID to use for SSO.
• azure-client-secret - The Azure AD client secret to use for SSO.
• azure-tenant-id - The Azure AD tenant ID to use for SSO.

A sample secret is provided below for reference:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: erasearch-ui
  namespace: my-namespace
stringData:
  app-signing-secret: XXXXXXXXX
  auth-sign-secret: XXXXXXXXX
  azure-client-id: XXXXXXXXX
  azure-client-secret: XXXXXXXXX
  azure-tenant-id: XXXXXXXXX
  postgres-password: XXXXXXXXX

ui.externalUrl¶

ui.externalUrl is the URL for accessing the EraSearch UI. This property is used for establishing redirect URLs for SSO. This value must be a valid URL and match the URL defined within the ui.ingress.hosts array discussed below.

ui.azureScopeName¶

ui.azureScopeName references the name of the Azure AD scope created for the EraSearch UI.

ui.ingress¶

ui.ingress is the Kubernetes ingress object to use for serving the EraSearch UI. A sample ingress object is provided below:

ingress:
  annotations: {}
  enabled: true
  hosts:
    - erasearch.example.com
  tls:
    - hosts:
        - erasearch.example.com
      secretName: tls-erasearch.example.com