Writing data with Fluent Bit
Estimated time to read: 4 minutes
This page shows how to use Fluent Bit to write data to EraSearch. In this guide, you'll:
- Use Fluent Bit to generate dummy log events
- Configure Fluent Bit to write the data to EraSearch
- Query the data using the EraSearch REST API
While the steps below use dummy log data, you can customize the setup to use any Fluent Bit input plugin, including StatsD, Docker, Windows Event Logs, and more.
Before you begin¶
This content is intended for engineers and developers using self-hosted EraSearch without RBAC. The steps below don't work with EraSearch on EraCloud or self-hosted EraSearch with RBAC.
Before you begin, install Fluent Bit, and make sure you have your EraSearch URL and authentication credentials.
Step 1: Configure the Fluent Bit dummy input¶
Fluent Bit's dummy input plugin generates fake/dummy log events, which can be useful for getting started with and testing Fluent Bit. Follow these steps to set it up:
- Open or create your Fluent Bit configuration (typically called
- Paste in this content:
Step 2: Configure the EraSearch output plugin¶
To configure Fluent Bit to send data to EraSearch, paste the content below into your Fluent Bit configuration file, replacing:
YOUR_ERASEARCH_HOSTNAMEwith your EraSearch hostname
YOUR_ERASEARCH_PORTwith your EraSearch port
YOUR_PASSWORDwith your EraSearch credentials
YOUR_INDEX_NAMEwith the target EraSearch index -- EraSearch creates the index for you
This step uses the Elasticsearch output plugin to let Fluent Bit work with EraSearch. That workflow is possible because the EraSearch REST API supports much of the Elasticsearch API.
Step 3: Start Fluent Bit and confirm your configuration¶
Start Fluent Bit with the relevant command. When Fluent Bit starts up, it should show output like this:
Fluent Bit v1.8.11 * Copyright (C) 2019-2021 The Fluent Bit Authors * Copyright (C) 2015-2018 Treasure Data * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io [ info] [engine] started (pid=1) [ info] [storage] version=1.1.5, initializing... [ info] [storage] in-memory [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128 [ info] [cmetrics] version=0.2.2 [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020 [ info] [sp] stream processor started
With Fluent Bit started, you should start to see log data from the file start to be persisted in EraSearch.
Step 4: Query your data in EraSearch¶
Use the EraSearch REST API to query the Fluent Bit data in EraSearch. Paste this cURL command in your terminal, replacing:
YOUR_ERASEARCH_URLwith your EraSearch URL
YOUR_INDEX_NAMEwith the EraSearch index you specified above
The response shows information about your data and API request, including:
took- The time, in milliseconds, EraSearch took to serve the query request
_id- A unique, auto-generated numerical identifier for documents
count- The number of files in the specified directory
You're all set! You're now using Fluent Bit to send log data to your EraSearch instance. For more information about Fluent Bit, visit these pages:
For other ways to get data into your database, visit the write-integrations reference. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages:
- Alerting with PagerDuty (for EraCloud users only)
- Alerting with Slack (for EraCloud users only)
- Connecting EraSearch to Grafana
- Exploring data in EraSearch's UI (for EraCloud users only)
- Explore-integrations reference