Splunk Search
Highlighted

Real time event tracing

I have a simple setup of forwarder->indexer and I want to display real time events coming from the forwarder. The data coming from the forwarder is very small, every event produces about 20 log lines and there are about 1-3 events per minute. Almost all log lines are recorded at the same second.

I need to show the events as fast as possible, the ideal would be that they appear as they happen and in correct order.

Currently I have two problems:
1. Events are appearing with at least 20 seconds delay after they really happen, they also come in chunks with big delay between them, meaning I can get 4 lines and other 16 after another delay.
2. Log lines are coming in wrong order (I sort by _indextime), if they happen on the same second.

When I tried same search with local files without the forwarder everything worked as expected.

How can I improve forwarder->indexer communication and why there is a problem with sorting?

Update: I was able to pin down the problem with the delay, the forwarder had about +40 seconds offset and events coming from it were displayed in chunks. This happens if you set specific real time window only.

Now the question is how to sort the events that happen on the same second in correct order, I tried time,indextime and their combinations.

Tags (2)
Highlighted

Re: Real time event tracing

Champion

Welcome.
Firstly there will be some unavoidable delay with the forwarder to indexer process.
20 seconds actually sounds like a fantastic time. If you imagine that the device has an event happen that causes it to write to its log, the log is then checked by the forwarder and updated data is found. This data is then pulled from the log and a connection made to the indexer, data is forwarded.

The indexer then receives the data and does its initial parsing to check for field extractions, timestamps etc. This data is then written to disc (into a hot bucket). Given both systems could be busy doing other things, there could be network traffic and packets could fail and require a re-send (TCP, also a reason why events may be the wrong way round at indextime as well as slightly larger events to parse).

Anyway, although others may have some suggestions firstly you can adjust how fast the forwarder forwards data. It has a rate limiter to prevent it flooding the network / indexer, have a look here. You want the part about MAXKBPS.

Also, best practice is generally to snap your search to at least -1m@m to allow for time delay (although on a busier setup/network -5m@m is best)

For your interest;

How indexing works

Highlighted

Re: Real time event tracing

While I generally agree with you, there was very significant delay which I was finally able to solve (I updated the question) Do you have any ideas regarding the second part?

0 Karma
Highlighted

Re: Real time event tracing

Splunk Employee
Splunk Employee

A forwarder sends data to an indexer in blocks and a block of data is approximately 64KB. In a small volume implementation, like yours, this could be one of the reasons that events are not being displayed in "real-time" or as fast as they're being written to.

0 Karma
Highlighted

Re: Real time event tracing

I did some tests with bigger data amounts and it still comes in chunks of various size. For example I got chunks of 4, 183, 435 events with few seconds delay between them.
Are there config options to handle values of blocks etc. on forwarder? I can see only limits.conf file with only maxKBps parameter which is currently set to 0.

0 Karma