Getting Data In
Highlighted

What is the frequency with which logs are read in Splunk?

New Member

What is the frequency with which logs are read in Splunk? Does delay in seeing recent log details in Splunk related to poor indexing?
And if indexing is perfectly fine, then what should be the ideal time after which details of a new request must be visible in Splunk?

Tags (2)
0 Karma
Highlighted

Re: What is the frequency with which logs are read in Splunk?

SplunkTrust
SplunkTrust

From my understanding, there is no "frequency." If the file changes, it starts forwarding the new data. There are technically several ways to do this, I don't know which one Splunk uses, but I have used a few throughout my programming career.

How soon should you see indexed data in Splunk? That depends on the processing of the data at index time. If not much is being looked at, you can have data visible sub-second after it is placed in a file on a forwarder, if all the stars align properly. We index about 350GB/day and see most of our data within 10 seconds with most of that under 5 seconds. I have had "Last 5 seconds" searches that fill up even the last 1 second slot with almost as many events as the other 4 slots.

If your indexing is putting structure on the data, or the indexing has to search hard to find the timestamp, or your network is slow, or the indexing hardware is slow (especially the disks), these things can lead to slow indexing of the data.

0 Karma
Highlighted

Re: What is the frequency with which logs are read in Splunk?

New Member

Thanks Cary 🙂

0 Karma
Highlighted

Re: What is the frequency with which logs are read in Splunk?

SplunkTrust
SplunkTrust

Hi garimayadav,

an universal forwarder will poll / check the files once a second.

cheers, MuS

0 Karma
Highlighted

Re: What is the frequency with which logs are read in Splunk?

Builder
0 Karma