Skip to content

Writing data with Fluent Bit

Estimated time to read: 4 minutes

Acquisition notice

In October 2022, ServiceNow acquired Era Software. The documentation on this site is no longer maintained and is intended for existing Era Software users only.

To get the latest information about ServiceNow's observability solutions, visit their website and documentation.

This page shows how to use Fluent Bit to write data to EraSearch. In this guide, you'll:

  • Use Fluent Bit to generate dummy log events
  • Configure Fluent Bit to write the data to EraSearch
  • Query the data using the EraSearch REST API

While the steps below use dummy log data, you can customize the setup to use any Fluent Bit input plugin, including StatsD, Docker, Windows Event Logs, and more.

Before you begin

This content is intended for engineers and developers using self-hosted EraSearch without RBAC. The steps below don't work with EraSearch on EraCloud or self-hosted EraSearch with RBAC.

Before you begin, install Fluent Bit, and make sure you have your EraSearch URL and authentication credentials.

Instructions

Step 1: Configure the Fluent Bit dummy input

Fluent Bit's dummy input plugin generates fake/dummy log events, which can be useful for getting started with and testing Fluent Bit. Follow these steps to set it up:

  1. Open or create your Fluent Bit configuration (typically called fluent-bit.conf)
  2. Paste in this content: 
    [INPUT]
    Name   dummy
    Tag    dummy.log

Step 2: Configure the EraSearch output plugin

To configure Fluent Bit to send data to EraSearch, paste the content below into your Fluent Bit configuration file, replacing:

  • YOUR_ERASEARCH_HOSTNAME with your EraSearch hostname

    Examples: localhost and erasearch.example.com

  • YOUR_ERASEARCH_PORT with your EraSearch port

    Examples: 9200 and 443

  • YOUR_USERNAME and YOUR_PASSWORD with your EraSearch credentials

  • YOUR_INDEX_NAME with the target EraSearch index -- EraSearch creates the index for you
[OUTPUT]
    Name  es
    Match *
    Host  YOUR_ERASEARCH_HOSTNAME
    Port  YOUR_ERASEARCH_PORT
    # if TLS/SSL is needed
    tls On

    Index YOUR_INDEX_NAME
    Time_Key _ts

    # for HTTP authentication
    HTTP_User YOUR_USERNAME
    HTTP_Password YOUR_PASSWORD

Note

This step uses the Elasticsearch output plugin to let Fluent Bit work with EraSearch. That workflow is possible because the EraSearch REST API supports much of the Elasticsearch API.

Step 3: Start Fluent Bit and confirm your configuration

Start Fluent Bit with the relevant command. When Fluent Bit starts up, it should show output like this:

Fluent Bit v1.8.11
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[ info] [engine] started (pid=1)
[ info] [storage] version=1.1.5, initializing...
[ info] [storage] in-memory
[ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[ info] [cmetrics] version=0.2.2
[ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[ info] [sp] stream processor started

With Fluent Bit started, you should start to see log data from the file start to be persisted in EraSearch.

Step 4: Query your data in EraSearch

Use the EraSearch REST API to query the Fluent Bit data in EraSearch. Paste this cURL command in your terminal, replacing:

  • YOUR_ERASEARCH_URL with your EraSearch URL

    Example: http://localhost:9200 - YOUR_INDEX_NAME with the EraSearch index you specified above

$ curl 'YOUR_ERASEARCH_URL/YOUR_INDEX_NAME/_search?q=_lid:*' | jq

The response shows information about your data and API request, including:

  • took - The time, in milliseconds, EraSearch took to serve the query request
  • _id - A unique, auto-generated numerical identifier for documents
  • count - The number of files in the specified directory
{
  "took": 33,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "YOUR_INDEX_NAME",
        "_id": "2528005658872578048",
        "_score": 1,
        "_source": {
          "_ts": "2022-01-20T16:28:14.225Z",
          "message": "dummy",
          "_lid": 2528005658872578000
        }
      }
    ]
  }
}

Next steps

You're all set! You're now using Fluent Bit to send log data to your EraSearch instance. For more information about Fluent Bit, visit these pages:

For other ways to get data into your database, visit the write-integrations reference. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages:

Last update: August 7, 2023