Skip to content

Writing data with Vector

Estimated time to read: 4 minutes

Acquisition notice

In October 2022, ServiceNow acquired Era Software. The documentation on this site is no longer maintained and is intended for existing Era Software users only.

To get the latest information about ServiceNow's observability solutions, visit their website and documentation.

This page shows how to use Vector to write real-time data to EraSearch. In this guide, you'll:

  • Use Vector to generate sample log data.
  • Configure Vector to collect, transform, and write the logs to EraSearch.
  • View the logs in EraSearch.

While the steps below use log data stored in files, you can customize the setup to use any Vector source, including Fluent, AWS Kinesis Firehose, and Kubernetes logs.

Before you begin

This content is intended for engineers and developers using EraSearch on EraCloud or self-hosted EraSearch:

This page also assumes you've installed Vector.

Instructions

Step 1: Configure the demo_log source

Vector's demo_logs source generates sample log events for getting started with and testing Vector. Follow these steps to set it up:

  1. Open or create your Vector configuration file, for example, vector.toml.

  2. Paste this content in the file:

    # Generate sample logs
[sources.sample_logs]
type = "demo_logs"
format = "apache_common"

# Parse the log body before sending to any sinks
[transforms.parse_logs]
type = "remap"
inputs = ["sample_logs"]
source = '''
. = parse_apache_log!(string!(.message), "common")
'''

Step 2: Configure the EraSearch output sink

To configure Vector to send data to EraSearch, add the content below to your Vector configuration file, replacing:

  • YOUR_SERVICE_URI with your EraCloud service URI.
  • YOUR_INDEX_NAME with the target EraSearch index -- EraSearch creates the index for you.
  • YOUR_API_KEY with your EraCloud API key.
[sinks.erasearch]
type="elasticsearch"
inputs=["parse_logs"]
endpoint="YOUR_SERVICE_URI"
healthcheck.enabled = false
request.concurrency = "adaptive"
bulk.index = "YOUR_INDEX_NAME"

request.headers.Authorization = "Bearer YOUR_API_KEY"

To configure Vector to send data to EraSearch, add the content below into your Vector configuration file, replacing:

  • YOUR_ERASEARCH_URL with your EraSearch URL.

    Example: http://localhost:9200.

  • YOUR_INDEX_NAME with the target EraSearch index -- EraSearch creates the index for you.

  • YOUR_API_KEY with your EraSearch RBAC API key. If you're not using RBAC, remove request.headers.Authorization = "Bearer YOUR_API_KEY" from the file.
[sinks.erasearch]
type="elasticsearch"
inputs=["parse_logs"]
endpoint="YOUR_ERASEARCH_URL"
healthcheck.enabled = false
bulk.index = "YOUR_INDEX_NAME"

request.headers.Authorization = "Bearer YOUR_API_KEY"

Note

The configuration above uses the Elasticsearch sink to let Vector work with EraSearch. That workflow is possible because the EraSearch REST API supports much of the Elasticsearch API.

Step 3: Start Vector

In the same directory, enter this command to start Vector:

$ vector --config ./vector.toml

When successful, your terminal outputs several INFO logs about Vector.

Step 4: View your data in EraSearch's UI

Access EraSearch's UI by visiting your EraCloud account and clicking search icon. Your logs are in the index you specified above.

Use the EraSearch REST API to query the data in EraSearch. Paste the command below in your terminal, replacing:

  • YOUR_ERASEARCH_URL with your EraSearch URL.

    Example: http://localhost:9200.

  • YOUR_INDEX_NAME with the EraSearch index you specified above.

  • YOUR_API_KEY with your EraSearch RBAC API key. If you're not using RBAC, remove \ -H 'Authorization: Bearer YOUR_API_KEY' from the command.
$ curl 'YOUR_ERASEARCH_URL/YOUR_INDEX_NAME/_search?q=_lid:*' \
  -H 'Authorization: Bearer YOUR_API_KEY'

The response shows information about your data and API request, including:

  • took - The time, in milliseconds, EraSearch took to serve the query request.
  • _id - A unique, auto-generated numerical identifier for documents.

Next steps

You're all set. Your EraSearch instance is now receiving real-time log data. For more information about Vector, including what logs you can collect and how to configure the Elasticsearch sink, visit these pages:

For other ways to get data into your database, visit the write-integrations reference. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages:

Last update: August 7, 2023