Splunk 3.4.4 LWF doesn´t process data until logrotate happens. Why?

[tpaulsen]

We have on four Linux SLES10_64 Servers Splunk 3.4.4. Forwarders installed. Usually our production logs produce a constant stream of at least 30 events/minute during the night time. Due to performance issues with these production boxes, we recently switched those Forwarders into LWF mode, to reduce their footprint. Now we found out that during the night time, when the data stream on the logs drop down to app. 30 events/minute, the LWFs don´t forward any data to our Indexer until at 8am the logrotate sets in. During the day time the LWFs work fine until app. 11pm plus minus 2 hours.

Is this a bug in the Splunk 3.4.4. LWF? Or could it be a licensing issue? The logs of these LWFs don´t show anything.

[Mick]

It's not a licensing issue, as forwarders don't require a license unless they are indexing data locally.

It sounds like a bug to me, if it was working ok with the regular Forwarder, and then switching to the LWF broke it, that suggests that the LWF isn't checking your files correctly.

It could also be something to do with the way the file is updated by your logging app, maybe it's not updating the modtime during those hours?

I recommend using the troubleshooting instructions here to enable DEBUG settings and figure out what Splunk is seeing when it checks these files.

A last resort would be to add the 'alwaysOpenFile' setting in inputs.conf, but seeing as you want to reduce resource footprint that may not be for you

By the way, 3.4.4 was a short-lived build, if you're planning an upgrade, look into 4.1.1 as that has a much improved file-monitoring capability