Writing data with Logstash

Estimated time to read: 5 minutes

This page shows how to use Logstash to write data to EraSearch. In this guide, you'll:

• Use Logstash to collect data about specific files.
• Configure Logstash to write the data to EraSearch.
• View the data in EraSearch.

While the steps below use log data stored in files, you can customize the setup to use any Logstash input plugin, including Elastic Beats, AWS Kinesis, Kafka, and more.

Before you begin¶

This content is intended for engineers and developers using EraSearch on EraCloud or self-hosted EraSearch:

• If you're using EraSearch on EraCloud, you need your service URI and API key. To get started with EraSearch on EraCloud, set up an account.

• If you're using self-hosted EraSearch, you need your EraSearch URL. If you also set up EraSearch RBAC, you need an API key for writing and reading data.

  To get started with self-hosted EraSearch, contact us at Era Software.

This guide also assumes you've installed Logstash.

Instructions¶

Step 1: Configure the Logstash file input¶

Logstash's file input plugin tails files, emitting one log message per file line. Follow these steps to set it up:

1. Open or create your Logstash configuration file, for example, logstash.conf.
2. Paste this content in the file:

   input {
     file {
       path => "YOUR_FILE_PATH"
     }
   }
   

3. Replace YOUR_FILE_PATH with the path to the file you want to monitor.

Step 2: Configure the EraSearch output plugin¶

output {
  elasticsearch {
    hosts => "YOUR_SERVICE_URI:443"
    index => "YOUR_INDEX_NAME"

    custom_headers => {
        "Authorization" => "Bearer YOUR_API_KEY"
    }
  }
}

output {
  elasticsearch {
    hosts => "YOUR_ERASEARCH_URL"
    index => "YOUR_INDEX_NAME"

    custom_headers => {
        "Authorization" => "Bearer YOUR_API_KEY"
    }

    # Uncomment the following line if you're using SSL.
    # ssl => true
  }
}

Step 3: Start Logstash and confirm your configuration¶

Start Logstash with the relevant command. When Logstash loads the file input and elasticsearch output, it shows output like this:

[INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-aarch64]"}
[...]
[INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["YOUR_ERASEARCH_URL"]}
[INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ][filewatch.observingtail  ][main][XXXX] START, creating Discoverer, Watch with file and sincedb collections

With Logstash started, you'll begin to see lines from the file in EraSearch.

$ echo "Hello, EraSearch!" >> YOUR_FILE_PATH

Step 4: View your data EraSearch¶

$ curl 'YOUR_ERASEARCH_URL/YOUR_INDEX_NAME/_search?q=_lid:*' \
  -H 'Authorization: Bearer YOUR_API_KEY'

Next steps¶

You're all set! You're now using Logstash to send log data to your EraSearch instance. For more information about Logstash, visit these pages:

• Logstash documentation
• Logstash's Input reference
• Logstash's Elasticsearch Output reference

For other ways to get data into your database, visit the write-integrations reference. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages:

• Alerting with Slack (for EraCloud users only)
• Connecting EraSearch to Grafana
• Exploring data in EraSearch's UI (for EraCloud users only)
• Explore-integrations reference