Skip to content

Writing bulk data

Estimated time to read: 5 minutes

Acquisition notice

In October 2022, ServiceNow acquired Era Software. The documentation on this site is no longer maintained and is intended for existing Era Software users only.

To get the latest information about ServiceNow's observability solutions, visit their website and documentation.

To start managing and learning from your logs, you need to get them into your database. With EraSearch, you can write data using the EraSearch REST API or by integrating with other tools.

This page shows how to write data from a file to your database using the EraSearch REST API.

Before you begin

This content is intended for engineers and developers using EraSearch on EraCloud or self-hosted EraSearch:

The steps below also assume you've installed jq, a JSON parser for the command line.

Writing data in bulk

Step 1: Prepare your data

EraSearch accepts bulk data in JSON Lines text file format. When formatting your JSON Lines file, separate lines with newlines and specify the index before every document.

For example, the bulkdata.json file below has three documents for the my_era_logs index. Each document includes these field keys: _line, response, and _ts.

{"index":{"_index":"my_era_logs"}}
{"_line":"health check","response":404,"_ts":1633632261000}
{"index":{"_index":"my_era_logs"}}
{"_line":"health check","response":503,"_ts":1633628661000}
{"index":{"_index":"my_era_logs"}}
{"_line":"access","response":200,"_ts":1634060854000}

Note

_line and _ts are recognized field keys in EraSearch.

_line
EraSearch auto-parses _line values and stores them as distinct strings for future queries. For example, EraSearch stores the field value "health check" as ["health","check"]. By default, EraSearch doesn't auto-parse the values of other field keys such as line or _logline.

_ts
EraSearch recognizes _ts as the document's timestamp, where the field value is in epoch time in milliseconds. If you don't include _ts in your file, EraSearch generates the field for you. In those cases, the value is the time EraSearch writes the data to your database.

Step 2: Write your file to EraSearch

To write bulk data to your database, enter the command below in your terminal, replacing YOUR_SERVICE_URI and YOUR_API_KEY with your own information.

This example assumes the data is in bulkdata.json and the current directory.

$ curl -XPOST 'YOUR_SERVICE_URI/_bulk' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  --data-binary "@bulkdata.json"

To write bulk data to your database, enter the command below in your terminal, replacing YOUR_ERASEARCH_URL and YOUR_API_KEY with your own information. If you're not using RBAC, remove -H 'Authorization: Bearer YOUR_API_KEY' \ from the command.

This example assumes the data is in bulkdata.json and the current directory.

$ curl -XPOST 'YOUR_ERASEARCH_URL/_bulk' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  --data-binary "@bulkdata.json"

EraSearch's response has information about the bulk write. It includes the following information:

  • "took" : 490 - The time, in milliseconds, it took EraSearch to complete the request.
  • "errors" : false - The boolean showing all writes succeeded.
  • "_id" : "XXX" - The unique numerical identifier EraSearch generated for each document.
  • "status" : 201 - The HTTP status code for each write.
{
  "took": 490,
  "errors": false,
  "items": [
    {
      "index": {
        "_index": "my_era_logs",
        "_type": "_doc",
        "_id": "6383901618637176832",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 1,
          "successful": 1,
          "failed": 0
        },
        "status": 201
      }
    },
    {
      "index": {
        "_index": "my_era_logs",
        "_type": "_doc",
        "_id": "6383901618637176833",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 1,
          "successful": 1,
          "failed": 0
        },
        "status": 201
      }
    },
    {
      "index": {
        "_index": "my_era_logs",
        "_type": "_doc",
        "_id": "6383901618637176834",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 1,
          "successful": 1,
          "failed": 0
        },
        "status": 201
      }
    }
  ]
}

Step 3: Query your data in EraSearch

To view the data, go to your EraCloud account and click search icon. The three documents are in the my_era_logs index between October 7, 2021, and October 12, 2021.

To view your data in EraSearch, enter the command below in your terminal, replacing YOUR_ERASEARCH_URL and YOUR_API_KEY with your own information. If you're not using RBAC, remove -H 'Authorization: Bearer YOUR_API_KEY' from the command.

This request targets only the my_era_logs index, and the query uses Elasticsearch's query-string syntax.

$ curl 'YOUR_ERASEARCH_URL/my_era_logs/_search?q=_line:*' \
  -H 'Authorization: Bearer YOUR_API_KEY' | jq

EraSearch's response shows three documents in the my_era_logs index:

{
  "took": 4,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "my_era_logs",
        "_id": "6383901618637176834",
        "_score": 1,
        "_source": {
          "_line": "access",
          "response": 200,
          "_ts": 1634060854000,
          "_lid": 6383901618637177000
        }
      },
      {
        "_index": "my_era_logs",
        "_id": "6383901618637176832",
        "_score": 1,
        "_source": {
          "_line": "health check",
          "response": 404,
          "_ts": 1633632261000,
          "_lid": 6383901618637177000
        }
      },
      {
        "_index": "my_era_logs",
        "_id": "6383901618637176833",
        "_score": 1,
        "_source": {
          "_line": "health check",
          "response": 503,
          "_ts": 1633628661000,
          "_lid": 6383901618637177000
        }
      }
    ]
  }
}

Next steps

You're all set. You wrote bulk data to EraSearch and viewed it in the database. Next, visit the write-integrations reference to see how to get real-time data into your database. To learn more about exploring, querying, and visualizing your data in EraSearch, visit these pages:

Last update: August 7, 2023