Changelog

Estimated time to read: 5 minutes

At Era Software, we're working to improve your log management experience. This page lists the new features and changes you get in the latest releases.

EraSearch 1.23.1¶

August 1, 2022

Version 1.23.1 is a patch release providing important performance improvements for common use cases.

New features¶

• Each Cache node now uses a Write-Ahead Log (WAL) to ensure quick recovery in case of unexpected or unclean shutdowns.

Improvements¶

• The API mappings endpoint performance has been drastically improved over previous versions.

• Query performance for documents within a single query task is up to 10x faster.

Breaking changes¶

• The API mappings endpoint now returns a maximum of 1,000 fields, where fields are sorted in order of occurrence. Previously all fields would be returned.

Bug fixes¶

• Shutdown: Startup after an "unclean shutdown" (caused by out-of-memory exceptions or SIGKILL) is significantly improved over previous versions.

EraSearch 1.23.0¶

July 22, 2022

Version 1.23.0 is a maintenance release improving EraSearch's performance and stability.

New features¶

• Added support for the from parameter for paginating query results.

• Added the QUARRY_QUERY_THREADS environment variable to the API and Cache services. The variable lets you increase the number of blocking threads dedicated to servicing queries. By default, the number of threads matches the number of available CPU cores.

• Added the QUARRY_QUERY_TIMEOUT_SECONDS environment variable to the API and Cache services. The variable lets you set a hard maximum duration for queries. EraSearch stops queries that exceed the timeout.

New metrics¶

EraSearch version 1.23.0 also offers the new metrics listed below. For more information about EraSearch metrics, visit Metrics reference.

• quarry_aggregation_cache_entries_bytes reports the number of bytes in the aggregation memory cache.

• quarry_aggregation_cache_entries_count reports the number of entries in the aggregation memory cache.

• quarry_compaction_doc_count reports the number of documents compacted by level label.
• quarry_compaction_failure_total reports the number of compaction failures, labeled with the failure's cause.
• quarry_delete_file_duration_ns reports how long an attempt at removing a file takes.
• quarry_delete_file_error_count reports the number of files that may exist but error during deletion.
• quarry_delete_file_success_count reports the number of successfully deleted files.
• quarry_delete_local_gc_count reports how many times the system initiated local garbage collection (GC) on index-lifecycles.
• quarry_delete_local_gc_duration_ns reports how long a local GC takes on index-lifecycle.
• quarry_delete_local_namespaces_try_later_count reports the number of indexes scheduled for GC soon.
• quarry_delete_local_namespaces_try_later_current_count is a gauge for a try-later GC queue depth.
• quarry_delete_update_gc_ranges_count reports how many times GC ranges were updated.
• quarry_delete_update_gc_ranges_duration_ns reports how long updating GC ranges took.
• quarry_delete_vacuum_namespace_visit_count reports the number of indexes visited by a vacuum task.
• quarry_document_field_count summarizes the number of fields per document using quantiles.
• quarry_document_token_count summarizes the number of tokens per document using quantiles.
• quarry_index_bytes reports the number of bytes in the index per index type and compaction level.
• quarry_jemalloc_* report allocation details.
• quarry_process_resident_bytes reports RSS, including memory maps.
• quarry_query_queue_duration_ns reports the time each query task spends queued on the query thread pool.
• quarry_query_queued_total reports the number of tasks ever queued on the query thread pool.
• quarry_query_tasks_current_count reports the number of tasks running on the query thread pool.
• quarry_query_tasks_panicked_total reports the total number of tasks that have panicked on the query thread pool.
• quarry_query_tasks_total reports the total number of tasks ever executed on the query thread pool.

Improvements¶

• Aggregations: Improve performance by adding special logic for counting documents with no filter.

• Aggregations: Strictly bound memory for servicing aggregation queries.

• Compactions: Add support for L1->{L1,L2} compactions, not just L1->L2.
• Disk: Improve performance by issuing fewer syscalls when walking the filesystem to collect deleted files.
• Search: Improve performance for phrase_prefix_query queries.
• Search: Reduce memory consumption while servicing queries and compactions.
• Search: Make several performance improvements for sparse queries.
• Storage: Validate Azure Blob Storage SAS URLs on startup for accuracy. Invalid URLs now trigger the Storage service to exit.
• Storage: Free system resources for indexes after ten minutes of inactivity.
• Storage: Make the POST /_bulk API endpoint return HTTP 429 errors when the system exceeds the Storage service's treasurer settings. That change informs clients to retry requests.

Breaking changes¶

In EraSearch RBAC, actors no longer need the monitor permission to access GET /_cat/indices, GET /_license, and GET /_xpack. Those API endpoints are now available for all authenticated requests. For GET /_cat/indices, EraSearch only returns indexes for which the actor has read access.

All other breaking changes are related to metrics:

• quarry_bulk_request_indexed_bytes no longer uses per-index labels to reduce label cardinality.

• quarry_compaction_current_count now has labels from and to for compaction levels.

• quarry_compaction_io_failure and quarry_compaction_upload_failure have been replaced with the single quarry_compaction_failure metric. The new metric includes a cause specifying the reason for the failure.
• quarry_eviction_file_total_count no longer uses per-index labels to reduce label cardinality.
• quarry_usage_write_bytes no longer uses per-index tags to reduce label cardinality.

Bug fixes¶

• Compactions: Fixed an issue preventing original documents from being rehydrated.

• Compactions: Fixed issue where compactions would not be retried under certain circumstances.

• Deletion: Fixed issue where an infinite loop may be encountered while compacting documents during deletion.
• Deletion: Fixed issue where more indexes can be cleaned up at a given time.
• Ingest: Fixed issue causing excessive calls to the Coordinator and Cache nodes while servicing bulk ingest requests.
• Memory: Fixed a memory leak when fetching large numbers of documents at query time.
• Memory: Fixed issue causing excessive memory consumption for serving calls to the _mapping API.
• Metrics: Fixed issue where summary metrics were not expired, which could cause integer overflow errors.
• Metrics: Fixed gauge-based metrics for concurrent operations -- they're accurate now.
• Query: Restricted exists queries to string field types.
• Query: Restricted percentile query aggregations to numeric fields.
• Shutdown: Added a maximum timeout for graceful HTTP shutdown, improving shutdown behavior when running in Kubernetes.
• Shutdown: Made the liveness API endpoint active during shutdown, ensuring the Cache nodes shut down cleanly. Previously, the Kubernetes scheduler may have forcefully shut down the Cache nodes, requiring recovery on startup.
• Shutdown: Made the system more aggressively interrupt queries and GC during shutdown.
• Storage: Fixed an issue where EraSearch returned HTTP 500 Internal Server Error when an Azure Blob Storage range lookup returned no results.
• Storage: Fixed an issue where EraSearch returned HTTP 500 (instead of HTTP 404) if an Azure Blob Storage lookup contained no results.
• Storage: Fixed an issue where Azure Blob Storage objects weren't accurately labeled, and couldn't be retrieved at query time.