Skip to content

Writing data with Fluentd

Estimated time to read: 5 minutes

Acquisition notice

In October 2022, ServiceNow acquired Era Software. The documentation on this site is no longer maintained and is intended for existing Era Software users only.

To get the latest information about ServiceNow's observability solutions, visit their website and documentation.

This page shows how to use Fluentd to write real-time data to EraSearch. In this guide, you'll do the following:

  • Use Fluentd to generate sample logs.
  • Configure Fluentd to write logs to EraSearch.
  • View the logs in EraSearch.

While the steps below use Fluentd's sample input plugin, you can customize the setup to use any Fluentd input, including windows_eventlog, http, monitor_agent, and more.

Before you begin

This content is intended for engineers and developers using EraSearch on EraCloud or self-hosted EraSearch:

This guide also assumes you've installed Fluentd and are using versions above 1.11.1.

Instructions

Step 1: Configure the sample input

Fluentd's sample input plugin generates sample logs for starting with and testing Fluentd. Follow these steps to set it up:

  1. Open or create your Fluentd configuration (typically called fluentd.conf).
  2. Paste in this content: 
    <source>
  @type sample
  sample {"hello":"world"}
  tag sample
</source>

Step 2: Configure the EraSearch output

Paste the content below in your Fluentd configuration file, replacing:

  • YOUR_SERVICE_URI with your EraSearch service URI. Don't include https://.

    Example: db-abcdefghi1234567.abc.eradb.com.

  • YOUR_API_KEY with your EraSearch API key.

  • YOUR_INDEX_NAME with the target EraSearch index -- EraSearch creates the index for you.
<match sample>
  @type elasticsearch
  host YOUR_SERVICE_URI
  port 443
  scheme https
  custom_headers "Authorization: Bearer YOUR_API_KEY"
  index_name YOUR_INDEX_NAME
</match>

Paste the content below in your Fluentd configuration file, replacing:

  • YOUR_HOSTNAME with your EraSearch hostname. Don't include http:// or https://.

    Example: localhost.

  • YOUR_INDEX_NAME with the target EraSearch index -- EraSearch creates the index for you.

  • YOUR_API_KEY with your EraSearch RBAC API key. If you're not using RBAC, remove custom_headers "Authorization: Bearer YOUR_API_KEY" from the file.
<match sample>
  @type elasticsearch
  host YOUR_HOSTNAME
  port 9200
  index_name YOUR_INDEX_NAME
  custom_headers "Authorization: Bearer YOUR_API_KEY"
</match>

Note

The configuration above uses the Elasticsearch output to let Fluentd work with EraSearch. That workflow is possible because the EraSearch REST API supports much of the Elasticsearch API.

Step 3: Start Fluentd and check your configuration

Start Fluentd with the relevant command. For information on starting Fluentd, visit their installation guides.

Next, run this command to make sure Fluentd is running with your new configuration. The output is similar to the example below.

$ tail /var/log/td-agent/td-agent.log
[...]
2022-01-21 18:04:51 +0000 [info]: adding source type="sample"
2021-12-03 14:38:40 -0500 [info]: #0 fluentd worker is now running worker=0

Step 4: View your data in EraSearch

Access EraSearch's UI by visiting your EraCloud account and clicking search icon. Your data is in the index you specified above. You may need to refresh the UI if the index is new.

Use the EraSearch REST API to query the data in EraSearch. Paste the command below in your terminal, replacing:

  • YOUR_ERASEARCH_URL with your EraSearch URL.

    Example: http://localhost:9200.

  • YOUR_INDEX_NAME with the EraSearch index you specified above.

  • YOUR_API_KEY with your EraSearch RBAC API key. If you're not using RBAC, remove \ -H 'Authorization: Bearer YOUR_API_KEY' from the command.
$ curl 'YOUR_ERASEARCH_URL/YOUR_INDEX_NAME/_search?q=_lid:*' \
  -H 'Authorization: Bearer YOUR_API_KEY'

The response shows information about your data and API request, including:

  • took - The time, in milliseconds, EraSearch took to serve the query request.
  • _id - A unique, auto-generated numerical identifier for documents.

Next steps

You're all set! You're now using Fluentd to send real-time log data to your EraSearch instance. For more information about Fluentd, visit these pages:

For other ways to get data into your database, visit the write-integrations reference. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages:

Last update: August 7, 2023