Changelog
Estimated time to read: 5 minutes
Acquisition notice
In October 2022, ServiceNow acquired Era Software. The documentation on this site is no longer maintained and is intended for existing Era Software users only.
To get the latest information about ServiceNow's observability solutions, visit their website and documentation.
At Era Software, we're working to improve your log management experience. This page lists the new features and changes you get in the latest releases.
EraSearch 1.23.1¶
August 1, 2022
Version 1.23.1 is a patch release providing important performance improvements for common use cases.
New features¶
- Each Cache node now uses a Write-Ahead Log (WAL) to ensure quick recovery in case of unexpected or unclean shutdowns.
Improvements¶
-
The API mappings endpoint performance has been drastically improved over previous versions.
-
Query performance for documents within a single query task is up to 10x faster.
Breaking changes¶
- The API mappings endpoint now returns a maximum of 1,000 fields, where fields are sorted in order of occurrence. Previously all fields would be returned.
Bug fixes¶
- Shutdown: Startup after an "unclean shutdown" (caused by out-of-memory exceptions or SIGKILL) is significantly improved over previous versions.
EraSearch 1.23.0¶
July 22, 2022
Version 1.23.0 is a maintenance release improving EraSearch's performance and stability.
New features¶
-
Added support for the
from
parameter for paginating query results. -
Added the
QUARRY_QUERY_THREADS
environment variable to the API and Cache services. The variable lets you increase the number of blocking threads dedicated to servicing queries. By default, the number of threads matches the number of available CPU cores. - Added the
QUARRY_QUERY_TIMEOUT_SECONDS
environment variable to the API and Cache services. The variable lets you set a hard maximum duration for queries. EraSearch stops queries that exceed the timeout.
New metrics¶
EraSearch version 1.23.0 also offers the new metrics listed below. For more information about EraSearch metrics, visit Metrics reference.
-
quarry_aggregation_cache_entries_bytes
reports the number of bytes in the aggregation memory cache. -
quarry_aggregation_cache_entries_count
reports the number of entries in the aggregation memory cache. quarry_compaction_doc_count
reports the number of documents compacted by level label.quarry_compaction_failure_total
reports the number of compaction failures, labeled with the failure's cause.quarry_delete_file_duration_ns
reports how long an attempt at removing a file takes.quarry_delete_file_error_count
reports the number of files that may exist but error during deletion.quarry_delete_file_success_count
reports the number of successfully deleted files.quarry_delete_local_gc_count
reports how many times the system initiated local garbage collection (GC) on index-lifecycles.quarry_delete_local_gc_duration_ns
reports how long a local GC takes on index-lifecycle.quarry_delete_local_namespaces_try_later_count
reports the number of indexes scheduled for GC soon.quarry_delete_local_namespaces_try_later_current_count
is a gauge for a try-later GC queue depth.quarry_delete_update_gc_ranges_count
reports how many times GC ranges were updated.quarry_delete_update_gc_ranges_duration_ns
reports how long updating GC ranges took.quarry_delete_vacuum_namespace_visit_count
reports the number of indexes visited by a vacuum task.quarry_document_field_count
summarizes the number of fields per document using quantiles.quarry_document_token_count
summarizes the number of tokens per document using quantiles.quarry_index_bytes
reports the number of bytes in the index per index type and compaction level.quarry_jemalloc_*
report allocation details.quarry_process_resident_bytes
reports RSS, including memory maps.quarry_query_queue_duration_ns
reports the time each query task spends queued on the query thread pool.quarry_query_queued_total
reports the number of tasks ever queued on the query thread pool.quarry_query_tasks_current_count
reports the number of tasks running on the query thread pool.quarry_query_tasks_panicked_total
reports the total number of tasks that have panicked on the query thread pool.quarry_query_tasks_total
reports the total number of tasks ever executed on the query thread pool.
Improvements¶
-
Aggregations: Improve performance by adding special logic for counting documents with no filter.
-
Aggregations: Strictly bound memory for servicing aggregation queries.
- Compactions: Add support for L1->{L1,L2} compactions, not just L1->L2.
- Disk: Improve performance by issuing fewer syscalls when walking the filesystem to collect deleted files.
- Search: Improve performance for
phrase_prefix_query
queries. - Search: Reduce memory consumption while servicing queries and compactions.
- Search: Make several performance improvements for sparse queries.
- Storage: Validate Azure Blob Storage SAS URLs on startup for accuracy. Invalid URLs now trigger the Storage service to exit.
- Storage: Free system resources for indexes after ten minutes of inactivity.
- Storage: Make the
POST /_bulk
API endpoint returnHTTP 429
errors when the system exceeds the Storage service's treasurer settings. That change informs clients to retry requests.
Breaking changes¶
In EraSearch RBAC, actors no longer need the monitor
permission to access GET /_cat/indices
, GET /_license
, and GET /_xpack
. Those API endpoints are now available for all authenticated requests. For GET /_cat/indices
, EraSearch only returns indexes for which the actor has read
access.
All other breaking changes are related to metrics:
-
quarry_bulk_request_indexed_bytes
no longer uses per-index labels to reduce label cardinality. -
quarry_compaction_current_count
now has labelsfrom
andto
for compaction levels. quarry_compaction_io_failure
andquarry_compaction_upload_failure
have been replaced with the singlequarry_compaction_failure
metric. The new metric includes acause
specifying the reason for the failure.quarry_eviction_file_total_count
no longer uses per-index labels to reduce label cardinality.quarry_usage_write_bytes
no longer uses per-index tags to reduce label cardinality.
Bug fixes¶
-
Compactions: Fixed an issue preventing original documents from being rehydrated.
-
Compactions: Fixed issue where compactions would not be retried under certain circumstances.
- Deletion: Fixed issue where an infinite loop may be encountered while compacting documents during deletion.
- Deletion: Fixed issue where more indexes can be cleaned up at a given time.
- Ingest: Fixed issue causing excessive calls to the Coordinator and Cache nodes while servicing bulk ingest requests.
- Memory: Fixed a memory leak when fetching large numbers of documents at query time.
- Memory: Fixed issue causing excessive memory consumption for serving calls to the
_mapping
API. - Metrics: Fixed issue where summary metrics were not expired, which could cause integer overflow errors.
- Metrics: Fixed gauge-based metrics for concurrent operations -- they're accurate now.
- Query: Restricted
exists
queries to string field types. - Query: Restricted
percentile
query aggregations to numeric fields. - Shutdown: Added a maximum timeout for graceful HTTP shutdown, improving shutdown behavior when running in Kubernetes.
- Shutdown: Made the
liveness
API endpoint active during shutdown, ensuring the Cache nodes shut down cleanly. Previously, the Kubernetes scheduler may have forcefully shut down the Cache nodes, requiring recovery on startup. - Shutdown: Made the system more aggressively interrupt queries and GC during shutdown.
- Storage: Fixed an issue where EraSearch returned
HTTP 500 Internal Server Error
when an Azure Blob Storage range lookup returned no results. - Storage: Fixed an issue where EraSearch returned
HTTP 500
(instead ofHTTP 404
) if an Azure Blob Storage lookup contained no results. - Storage: Fixed an issue where Azure Blob Storage objects weren't accurately labeled, and couldn't be retrieved at query time.