What is the frequency with which logs are read in Splunk?

[garimayadav]

What is the frequency with which logs are read in Splunk? Does delay in seeing recent log details in Splunk related to poor indexing?
And if indexing is perfectly fine, then what should be the ideal time after which details of a new request must be visible in Splunk?

[satishsdange]

Hi garimayadav -

Questions asked by you purely depends upon data volume/day, underlying hardware, deployment architecture, network latency. Following links will address your queries. Please have a look.

http://docs.splunk.com/Documentation/Splunk/6.2.2/Capacity/Forwarder-to-indexerratios
http://docs.splunk.com/Documentation/Splunk/6.2.2/Capacity/Referencehardware
http://docs.splunk.com/Documentation/Splunk/6.2.2/Capacity/Performancechecklist
http://docs.splunk.com/Documentation/Splunk/6.2.2/Capacity/Summaryofperformancerecommendations

[MuS]

Hi garimayadav,

an universal forwarder will poll / check the files once a second.

cheers, MuS

[cpetterborg]

From my understanding, there is no "frequency." If the file changes, it starts forwarding the new data. There are technically several ways to do this, I don't know which one Splunk uses, but I have used a few throughout my programming career.

How soon should you see indexed data in Splunk? That depends on the processing of the data at index time. If not much is being looked at, you can have data visible sub-second after it is placed in a file on a forwarder, if all the stars align properly. We index about 350GB/day and see most of our data within 10 seconds with most of that under 5 seconds. I have had "Last 5 seconds" searches that fill up even the last 1 second slot with almost as many events as the other 4 slots.

If your indexing is putting structure on the data, or the indexing has to search hard to find the timestamp, or your network is slow, or the indexing hardware is slow (especially the disks), these things can lead to slow indexing of the data.

[garimayadav]

Thanks Cary 🙂