Configuration reference
Estimated time to read: 17 minutes
Acquisition notice
In October 2022, ServiceNow acquired Era Software. The documentation on this site is no longer maintained and is intended for existing Era Software users only.
To get the latest information about ServiceNow's observability solutions, visit their website and documentation.
This page describes EraSearch's configuration files and options. Use it to customize self-hosted EraSearch to your environment and logging needs.
Configuration files¶
For self-hosted EraSearch¶
Self-hosted EraSearch uses Helm – a package manager for Kubernetes – for configuring and deploying releases into Kubernetes clusters. All self-hosted instances come with these files:
eradb-registry-xxxx.json
- The authentication credentials for pulling images from Era Software's container registry.eradb-x.x.x.tgz
- Compressed files with Helm charts for installing EraSearch into a Kubernetes cluster.values-eradb.yaml
- The Helm values file, listing EraSearch's configuration options in YAML. Era Software customizes the file to your environment.
To customize EraSearch, edit the configuration options in the values file and specify that file when deploying EraSearch. The values file overrides the default configuration options in the Helm charts.
For EraSearch on EraCloud¶
Era Software configures and manages EraSearch on EraCloud for you. You don't need to configure or customize your EraSearch instance, and it's ready for production by default.
Using values files¶
To configure self-hosted EraSearch, edit the options in values-eradb.yaml
. Then specify the file in the helm install or helm upgrade commands. To view the configuration options for an existing release, use helm get values.
Note
You can also override default configuration options by setting specific values in the helm install
and helm upgrade
commands. Visit Helm's install and upgrade documentation for more information.
helm install
example¶
The command below installs EraSearch into a Kubernetes cluster.
The command installs the eradb-1.2.3.tgz
charts into the logs-erasearch
namespace. Helm uses the configurations in values-eradb.yaml
to override default chart values, and the release is called myerasearch
.
$ helm install myerasearch ./eradb-1.2.3.tgz \
--values ./values-eradb.yaml \
--namespace logs-erasearch
The install command returns this output:
helm upgrade
example¶
The command below upgrades an existing EraSearch release in a Kubernetes cluster.
The command upgrades the myerasearch
release in the logs-erasearch
namespace. Helm applies the updated eradb-1.2.3.tgz
charts and values-eradb.yaml
file to the existing release.
$ helm upgrade myerasearch ./eradb-1.2.3.tgz \
--values ./values-eradb.yaml \
--namespace logs-erasearch
The upgrade command returns this output along with other deployment details:
helm get values
example¶
This command displays the configuration options used in an EraSearch release (myerasearch
):
The command returns USER-SUPPLIED VALUES
followed by the configuration options. The response is organized by service and lists the options in alphabetical order.
A note on service names¶
This page uses two names for EraSearch services:
- Configuration option names use Era Software's internal names (for example, Maxwell).
- Configuration descriptions use descriptive names (for example, Storage Service).
That naming difference is temporary. In the future, option names and descriptions will use only descriptive names. Until then, here’s how the descriptive names map to internal names:
- Cache Service and API Service ➡ Quarry
- Coordinator Service ➡ Alexandria
- Storage Service ➡ Maxwell
Configuration options¶
The sections below list and describe the configuration options in values-eradb.yaml
. They're organized by service.
Cache Service and API Service options¶
This section describes the following Cache Service and API Service options:
quarry:
image:
repository: <string>
tag: <string>
imagePullSecrets:
- name: <string>
replicaCount: <integer>
resources:
cpu: <integer>
memory: <integer>
disk: <integer>
persistence:
storageClass: <string>
extraEnv:
- name: <string>
value: <integer>
api:
replicaCount: <integer>
resources:
cpu: <integer>
memory: <integer>
quarry.image.repository¶
Type | Default | Needs configuring |
---|---|---|
String | us.gcr.io/eradb-registry/quarry | Usually no |
quarry.image.repository
is the container registry storing the image for API Service and Cache Service.
The default value – set to Era Software's registry – works for most installations. Update this option only if you're mirroring images from an internal registry instead of pulling images from Era Software's registry.
quarry.image.tag¶
Type | Default | Needs configuring |
---|---|---|
String | 1.XX.X | Usually no |
quarry.image.tag
identifies the relevant API Service and Cache Service image to work with. You only need to update this option if instructed by Era Software support.
quarry.imagePullSecrets¶
Type | Default | Needs configuring |
---|---|---|
String | eradb-registry | Yes |
quarry.imagePullSecrets
references the secret that lets Kubernetes authenticate with Era Software's private registry. This value must match the name of the secret you created in the EraSearch namespace.
quarry.replicaCount¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1 | Yes |
quarry.replicaCount
is the number of Cache Service pods Kubernetes runs at the same time.
Increasing the value scales Cache Service, letting EraSearch handle larger data volumes and service more queries. The default, 1
, is a good starting value for new EraSearch installations.
quarry.resources.cpu¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1 | Yes |
quarry.resources.cpu
sets compute processing resources, in Kubernetes CPU units, for the Cache Service container.
Your quarry.resources.cpu value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
quarry.resources.memory¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1Gi | Yes |
quarry.resources.memory
sets memory resources for the Cache Service container. The value is either a plain integer (in bytes) or an integer with a memory resource unit.
Your quarry.resources.memory value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
quarry.resources.disk¶
Type | Default | Needs configuring |
---|---|---|
Integer | 10Gi | Yes |
quarry.resources.disk
sets disk resources for the Cache Service container. The value is either a plain integer (in bytes) or an integer with a disk resource unit.
Your quarry.resources.disk
value depends on your expected data volumes, query patterns, and deployment size. Reach out to Era support if you have questions about sizing Cache Service disks.
quarry.persistence.storageClass¶
Type | Default | Needs configuring |
---|---|---|
String | gp2 | Yes |
quarry.persistence.storageClass
defines Cache Service's local storage in the Kubernetes cluster. In practice, Kubernetes uses quarry.persistence.storageClass
to dynamically provision a persistent volume. Cache Service stores data in that persistent volume and uses it to service query results.
Your storage class depends on your cloud provider. For Amazon S3, Era Software recommends starting with gp2
. Visit Amazon's EBS volume types for more information.
For Microsoft Azure Blob Storage, Era Software recommends starting with managed-premium
unless you need provisioned IOPS. Visit Microsoft's Storage classes for more information.
quarry.extraEnv¶
Type | Default | Needs configuring |
---|---|---|
Array | '' | Usually no |
quarry.extraEnv
defines additional environment variables for EraSearch's Cache Service. Use this section to set an environment variable for optional feature flags, for example, QUARRY_EVICTION_TTL
:
QUARRY_EVICTION_TTL
is the time after which Cache Service removes documents from disk. The value is an integer with a human-readable time duration. If you don't set QUARRY_EVICTION_TTL
, Cache Service keeps data on disk forever.
If you set QUARRY_EVICTION_TTL
, about every minute, Cache Service checks for documents with timestamps older than QUARRY_EVICTION_TTL
. It then removes those documents from disk. That removal process is called cache eviction.
Raising QUARRY_EVICTION_TTL
results in the following:
- Increases how long Cache Service keeps documents on disk.
-
Gives you the fastest possible queries on documents in the longer time range, as more data is available in the local cache.
-
Increases disk-resource needs (set by quarry.resources.disk).
Lowering QUARRY_EVICTION_TTL
results in the following:
- Decreases how long Cache Service keeps documents on disk.
- Gives you the fastest possible queries on documents in the shorter time range.
You can still query documents not on disk. Those queries are slower because EraSearch has to move the documents from object storage back to disk. - Lowers disk-resource needs (set by quarry.resources.disk).
quarry.api.replicaCount¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1 | Yes |
quarry.api.replicaCount
is the number of API Service pods Kubernetes runs at the same time.
Raising the value scales API Service, increasing how many write and query requests you can make to EraSearch. The default, 1
, is a good starting value for new EraSearch installations.
quarry.api.resources.cpu¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1 | Yes |
quarry.api.resources.cpu
sets compute processing resources, in Kubernetes CPU units, for the API Service container.
Your quarry.api.resources.cpu value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
quarry.api.resources.memory¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1Gi | Yes |
quarry.api.resources.memory
sets memory resources for the API Service container. The value is either a plain integer (in bytes) or an integer with a memory resource unit.
Your quarry.api.resources.memory value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
Coordinator Service options¶
This section describes the following Coordinator Service options:
alexandria:
image:
repository: <string>
tag: <string>
imagePullSecrets:
- name: <string>
replicaCount: <integer>
resources:
cpu: <integer>
memory: <integer>
alexandria.image.repository¶
Type | Default | Needs configuring |
---|---|---|
String | us.gcr.io/eradb-registry/alexandria | Usually no |
alexandria.image.repository
is the container registry storing the Coordinator Service image.
The default value – set to Era Software's registry – works for most installations. Update this option only if you're mirroring images from an internal registry instead of pulling images from Era Software's registry.
alexandria.image.tag¶
Type | Default | Needs configuring |
---|---|---|
String | l.XX.X | Usually no |
alexandria.image.tag
identifies the relevant Coordinator Service image to work with. You only need to update this option if instructed by Era Software support.
alexandria.imagePullSecrets¶
Type | Default | Needs configuring |
---|---|---|
String | eradb-registry | Yes |
alexandria.imagePullSecrets
references the secret that lets Kubernetes authenticate with Era Software's private registry. This value must match the name of the secret you created in the EraSearch namespace.
alexandria.replicaCount¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1 | Yes |
alexandria.replicaCount
is the number of Coordinator Service pods Kubernetes runs at the same time.
Increasing the value scales Coordinator Service, letting EraSearch more efficiently process large amounts of incoming data. The default, 1, is a good starting value for new EraSearch installations.
alexandria.resources.cpu¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1 | Yes |
alexandria.resources.cpu
sets compute processing resources, in Kubernetes CPU units, for the Coordinator Service container.
Your alexandria.resources.cpu
value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
alexandria.resources.memory¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1Gi | Yes |
alexandria.resources.memory
sets memory resources for the Coordinator Service container. The value is either a plain integer (in bytes) or an integer with a memory resource unit.
Your alexandria.resources.memory
value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
Redis options¶
Coordinator Service stores object IDs (OIDs) in Redis. This section describes the following Redis configuration options:
stateless-redis:
master:
resources:
limits:
cpu: <integer>
memory: <integer>
requests:
cpu: <integer>
memory: <integer>
stateless-redis.master.resources.limits.cpu¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1 | No |
stateless-redis.master.resources.limits.cpu
sets a compute processing resource limit, in Kubernetes CPU units, for the Redis container. This value must be higher than stateless-redis.master.resources.requests.cpu
.
Your stateless-redis.master.resources.limits.cpu
value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
stateless-redis.master.resources.limits.memory¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1Gi | Yes |
stateless-redis.master.resources.limits.memory
sets a memory resource limit for the Redis container. The value is either a plain integer (in bytes) or an integer with a memory resource unit. The value must be higher than stateless-redis.master.resources.requests.memory
.
Your stateless-redis.master.resources.limits.memory
value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
stateless-redis.master.resources.requests.cpu¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1 | No |
stateless-redis.master.resources.requests.cpu
sets compute processing resources, in Kubernetes CPU units, for the Redis container. This value must be lower than stateless-redis.master.resources.limits.cpu
.
Your stateless-redis.master.resources.requests.cpu
value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
stateless-redis.master.resources.requests.memory¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1Gi | Yes |
stateless-redis.master.resources.requests.memory
sets memory resources for the Redis container. The value is either a plain integer (in bytes) or an integer with a memory resource unit. This value must be lower than stateless-redis.master.resources.limits.memory
.
Your stateless-redis.master.resources.requests.memory
value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
Storage Service options¶
This section describes the following Storage Service options:
maxwell:
image:
repository: <string>
tag: <string>
imagePullSecrets:
- name: <string>
replicaCount: <integer>
resources:
cpu: <integer>
memory: <integer>
s3:
bucket: <string>
endpoint: <string>
region: <string>
treasurer:
batch_delay_ms: <integer>
batch_size: <integer>
monthly_budget: <integer>
extraEnv:
- name: <string>
value: <string>
maxwell.image.repository¶
Type | Default | Needs configuring |
---|---|---|
String | us.gcr.io/eradb-registry/maxwell | Usually no |
maxwell.image.repository
is the container registry storing the Storage Service image.
The default value – set to Era Software's registry – works for most installations. Update this option only if you're mirroring images from an internal registry instead of pulling images from Era Software's registry.
maxwell.image.tag¶
Type | Default | Needs configuring |
---|---|---|
String | 1.XX.X | Usually no |
maxwell.image.tag
identifies the relevant Storage Service image to work with. You only need to update this option if Era Software support suggests it.
maxwell.imagePullSecrets¶
Type | Default | Needs configuring |
---|---|---|
String | eradb-registry | Yes |
maxwell.imagePullSecrets
references the secret that lets Kubernetes authenticate with Era Software's private registry. This value must match the name of the secret you created in the EraSearch namespace.
maxwell.replicaCount¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1 | Yes |
maxwell.replicaCount
is the number of Storage Service pods Kubernetes runs at the same time.
Increasing the value scales Storage Service, letting EraSearch read and write more data to long-term storage. The default, 1, is a good starting value for new EraSearch installations.
maxwell.resources.cpu¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1 | Yes |
maxwell.resources.cpu
sets compute processing resources, in Kubernetes CPU units, for the Storage Service container.
Your maxwell.resources.cpu
value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
maxwell.resources.memory¶
Type | Default | Needs configuring |
---|---|---|
Integer | 1G | Yes |
maxwell.resources.memory
sets memory resources for the Storage Service container. The value is either a plain integer (in bytes) or an integer with a memory resource unit.
Your maxwell.resources.memory
value depends on your Kubernetes cluster and deployment size. For more information, visit Kubernetes's Resource management for pods and containers.
maxwell.s3.bucket¶
Type | Default | Needs configuring |
---|---|---|
String | '' | Yes |
maxwell.s3.bucket
is the container for storing EraSearch data in object storage. Examples of bucket names are mycompany.internal.private.erasearch
and erasearch
.
Note
While the name s3.bucket
references Amazon's object storage service, the configuration also works with Microsoft Azure Blob Storage and Google Cloud Platform.
maxwell.s3.endpoint¶
Type | Default | Needs configuring |
---|---|---|
String | '' | Yes, for Microsoft Azure Blob Storage |
maxwell.s3.endpoint
is the URL for connecting to objects in your Microsoft Azure Blob Storage account. When configuring this option, make sure your account's network security settings let EraSearch authenticate requests.
This is an example of a Microsoft Azure Blob Storage endpoint (note that it includes the URI and SAS token):
https://mystorageaccount.blob.core.windows.net/?sv=2020-08-04&ss=b&srt=sco&sp=rwdlacitfx&se=2023-03-04T14:33:22Z&st=2022-03-04T06:33:22Z&spr=https&sig=XXXXXXXX
Note
While the name s3.endpoint references Amazon's object storage service, the configuration works with Microsoft Azure Blob Storage. You don't need to configure this option if you're working with Amazon S3.
maxwell.s3.region¶
Type | Default | Needs configuring |
---|---|---|
String | '' | Yes |
maxwell.s3.region
is the geographical area for your object storage. For example, us-east-1
and us-west-2
are Amazon Web Services regions, and eastus2
and westus2
are Microsoft Azure Blob Storage regions.
Note
While the name s3.region references Amazon's object storage service, the configuration also works with Microsoft Azure Blob Storage and Google Cloud Platform.
maxwell.treasurer.batch_delay_ms¶
Type | Default | Needs configuring |
---|---|---|
Integer | 250 | Yes |
maxwell.treasurer.batch_delay_ms
is the time, in milliseconds, after which the Storage Service flushes data to object storage. In practice, the Storage Service flushes data after the system reaches maxwell.treasurer.batch_delay_ms
or maxwell.treasurer.batch_size
, whichever happens first.
Increasing maxwell.treasurer.batch_delay_ms
lowers data throughput and object-storage costs. Decreasing the value raises data throughput and object-storage costs.
maxwell.treasurer.batch_size¶
Type | Default | Needs configuring |
---|---|---|
Integer | 65536 | Yes |
maxwell.treasurer.batch_size
is the maximum batch size, in bytes, after which the Storage Service flushes data to object storage. In practice, the Storage Service flushes data after the system reaches maxwell.treasurer.batch_size
or maxwell.treasurer.batch_delay_ms
, whichever happens first.
Increasing maxwell.treasurer.batch_size
lowers data throughput and object-storage costs. Decreasing the value raises data throughput and object storage-costs.
maxwell.treasurer.monthly_budget¶
Type | Default | Needs configuring |
---|---|---|
Integer | 500 | Yes |
maxwell.treasurer.monthly_budget
is the target USD amount you want to spend on object storage per month. This option helps you target a budget by affecting how often Storage Service interacts with object storage.
In practice, Storage Service uses this option to calculate a request limit. The request limit is the number of object-storage requests Storage Service can make per month while staying in budget. If Storage Service's average request rate goes above the request limit, the service lowers the request rate. That change keeps costs closer to the target budget.
When applying maxwell.treasurer.monthly_budget
, Storage Service:
- Doesn't flush data as often to object storage, letting Cache Service keep data in local storage longer.
- Overrides
maxwell.treasurer.batch_delay_ms
andmaxwell.treasurer.batch_size
to determine the object-storage request rate. - Reports the new request rate in the
maxwell_treasurer_until_ready
metric.
If you don't want to use maxwell.treasurer.monthly_budget
, you can disable it by setting it to a high number, for example, 60000
.
maxwell.extraEnv¶
Type | Default | Needs configuring |
---|---|---|
Array | '' | Yes |
maxwell.extraEnv
defines required environment variables for EraSearch's Storage Service.
For Amazon S3¶
Use the syntax below to set environment variables for your Amazon Web Services (AWS) credentials. Replace YOUR_KEY_ID
and YOUR_SECRET
with your AWS access key id and secret access key.
extraEnv:
- name: AWS_ACCESS_KEY_ID
value: YOUR_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
value: YOUR_SECRET
Note
For production deployments using AWS, Era Software doesn't recommend using environment variables to expose credentials. Instead, assign an AWS Identity and Access Management (IAM) role to a Kubernetes service account.
For Microsoft Azure Blob Storage¶
Use the syntax below to set environment variables for your storage provider. Replace YOUR_CONTAINER_NAME
with the name of the container storing EraSearch data.
extraEnv:
- name: MAXWELL_STORAGE
value: azure_blob
- name: MAXWELL_AZURE_CONTAINER
value: YOUR_CONTAINER_NAME
UI Service options¶
This section describes the following UI Service options:
ui:
enabled: <boolean>
image:
tag: <string>
imagePullSecrets:
- name: <string>
existingSecretName: <string>
externalUrl: <string>
azureScopeName: <string>
ingress:
enabled: <boolean>
hosts:
- <string>
tls:
- hosts:
- <string>
secretName: <string>
ui.enabled¶
Type | Default | Needs configuring |
---|---|---|
Boolean | false | Yes |
ui.enabled
controls whether the UI Service deployment is included or not. Set this option to true
to deploy the UI Service.
ui.image.tag¶
Type | Default | Needs configuring |
---|---|---|
String | 1.XX.X | Usually no |
ui.image.tag
identifies the relevant UI Service image to work with. You only need to update this option if instructed by Era Software support.
ui.imagePullSecrets¶
Type | Default | Needs configuring |
---|---|---|
String | eradb-registry | Yes |
ui.imagePullSecrets
references the secret that lets Kubernetes authenticate with Era Software's private registry. This value must match the name of the secret you created in the EraSearch namespace.
ui.existingSecretName¶
Type | Default | Needs configuring |
---|---|---|
String | erasearch-ui | Yes |
ui.existingSecretName
references the name of the Kubernetes secret that contains sensitive configuration options for the UI Service. Within this secret, the following keys must be set:
app-signing-secret
- A randomly-generated secret used to securely sign application cookies to prevent tampering.postgres-password
- A password to use as the master password for the UI Service's backing PostgreSQL database.auth-sign-secret
- A randomly-generated secret used for the UI Service components to securely communicate.azure-client-id
- The Azure AD client ID to use for SSO.azure-client-secret
- The Azure AD client secret to use for SSO.azure-tenant-id
- The Azure AD tenant ID to use for SSO.
A sample secret is provided below for reference:
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: erasearch-ui
namespace: my-namespace
stringData:
app-signing-secret: XXXXXXXXX
auth-sign-secret: XXXXXXXXX
azure-client-id: XXXXXXXXX
azure-client-secret: XXXXXXXXX
azure-tenant-id: XXXXXXXXX
postgres-password: XXXXXXXXX
ui.externalUrl¶
Type | Default | Needs configuring |
---|---|---|
String | https://erasearch.example.com | Yes |
ui.externalUrl
is the URL for accessing the EraSearch UI. This property is used for establishing redirect URLs for SSO. This value must be a valid URL and match the URL defined within the ui.ingress.hosts
array discussed below.
ui.azureScopeName¶
Type | Default | Needs configuring |
---|---|---|
String | `` | Yes |
ui.azureScopeName
references the name of the Azure AD scope created for the EraSearch UI.
ui.ingress¶
Type | Default | Needs configuring |
---|---|---|
Object | {} | Yes |
ui.ingress
is the Kubernetes ingress object to use for serving the EraSearch UI. A sample ingress object is provided below:
ingress:
annotations: {}
enabled: true
hosts:
- erasearch.example.com
tls:
- hosts:
- erasearch.example.com
secretName: tls-erasearch.example.com