topic Re: Why does data stop getting indexed after a log rotation? in Getting Data In

Why does data stop getting indexed after a log rotation?

Arkon — Sat, 30 Jul 2016 00:52:27 GMT

Hi,

I noticed that, right after a log rotation, the data is not being indexed anymore.
Data is still going through /var/log/myapp.log and /var/log/messages (rsyslog UDP), so it all arrives on the machine (at 100%), but it is not being indexed.

On Splunk, I am monitoring logs arriving with real-time searches. Before log-rotate, everything is fine and logs are arriving on a regular basis. After logrotate, I do not get anything anymore.

Here is my inputs.conf:

[monitor:///var/log/myapp.log]
sourcetype = myappsourcetype
crcSalt = <SOURCE>
crcSalt = 2048
disabled = 0

My log rotate conf:

"/var/log/myapp.log" {
  monthly
  size 100M
  rotate 30
  compress
  postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2>/dev/null` &> /dev/null || true
  endscript
}

My sourcetype shouldn't be the problem as it only contains some side fields extractions.

Thank you very much in advance

Re: Why does data stop getting indexed after a log rotation?

Arkon — Sat, 30 Jul 2016 01:01:27 GMT

It looks like that deleting the file and restarting rsyslog fix the issue.
Does that mean the problem is coming from Rsyslog or Splunk "loosing his pointer" to the file?

Re: Why does data stop getting indexed after a log rotation?

ddrillic — Sat, 30 Jul 2016 01:04:08 GMT

For one thing -

crcSalt = <SOURCE>
crcSalt = 2048

Should probably be -

crcSalt = <SOURCE>
initCrcLength = 2048

Re: Why does data stop getting indexed after a log rotation?

twinspop — Sat, 30 Jul 2016 01:07:38 GMT

You very very very very likely do not want to use crcSalt. It's the truly rare log file that requires it.

Re: Why does data stop getting indexed after a log rotation?

twinspop — Sat, 30 Jul 2016 01:29:53 GMT

Can you explain why you think you need crcSalt? And I'm guessing you meant initCrcLength for the second crcSalt entry.

There are very particular reasons for using crcSalt, and I'm going to take a guess that UDP delivered syslog will not fit that criteria 99% of the time. You would only need initCrcLength if the beginning of your file always had a long header that was identical each time.

Try removing both crcSalt entries and see what happens.

Re: Why does data stop getting indexed after a log rotation?

ddrillic — Sun, 31 Jul 2016 00:52:48 GMT

In our place we normally add crcSalt and initCrcLength when things don't work as expected. However, since we already speak about it, it's fun to look further.

A very thorough explanation about How Splunk software handles log file rotation

It says at the end of the page -

-- Do not use crcSalt = with rolling log files, or any other scenario in which logfiles get renamed or moved to another monitored location. Doing so prevents Splunk software from recognizing log files across the roll or rename, which results in the data being reindexed.

Re: Why does data stop getting indexed after a log rotation?

twinspop — Sun, 31 Jul 2016 02:08:53 GMT

ddrillic: Ding!

Re: Why does data stop getting indexed after a log rotation?

hardikJsheth — Mon, 01 Aug 2016 06:22:42 GMT

I agree with twinspop, you don't need crcSalt and initCrcLength parameters. What problem are you facing? Is your file not getting monitored?

Why are you using postrotate? Do you need to kill any process on log rotation? Try removing this parameter.

Re: Why does data stop getting indexed after a log rotation?

Arkon — Mon, 01 Aug 2016 23:12:13 GMT

Thanks! So I removed crcSalt but it is still the same issue: I receive the logs but right after logrotate, they are not arriving anymore.

Re: Why does data stop getting indexed after a log rotation?

Arkon — Mon, 01 Aug 2016 23:14:46 GMT

I supposed that Splunk was not reading enough data in each event so I increased initCrcLength.
As for CrcSalt, it was just a bet, I removed it.
My file is getting monitoring but the data is not getting to the dashboard, I do not see them after my logrotate. But the file is still getting filled with incoming data.

As for the -HUP, it is simply to close the file and reload rsyslog, to make the process cleaner. I tried removing it and it does not help at all.

Re: Why does data stop getting indexed after a log rotation?

Arkon — Mon, 01 Aug 2016 23:16:11 GMT

Thank you very much, I removed crcSalt but I am still encoutering the same issue.

Re: Why does data stop getting indexed after a log rotation?

twinspop — Mon, 01 Aug 2016 23:21:55 GMT

Just to be clear... you restarted the forwarder after making this change?

Re: Why does data stop getting indexed after a log rotation?

Arkon — Mon, 01 Aug 2016 23:45:01 GMT

Yes I removed crcSalt and restarting splunk right after.
I then did a logrotate -f /var/log/myapp.log and then I could not see the new events anymore.

Re: Why does data stop getting indexed after a log rotation?

twinspop — Tue, 29 Sep 2020 10:27:15 GMT

What do the _internal logs say? Check with index=_internal host=yourhost myapp.log. You should see a series of logs there at the time you forced the log rotation.

Re: Why does data stop getting indexed after a log rotation?

sjohnson_splunk — Tue, 02 Aug 2016 01:14:44 GMT

I have encountered the problem. Upon rotation the current logfile is moved and compressed. Then a new file is created with the same name. That is a problem since Splunk has the log file opened at the time of rotation, it seems to follow the old file (nothing else is written to it so you don't see any more data).

The solution is with your log rotate script. You either need to use the copytruncate option or else the postrotate/script option to restart the Splunk forwarder.

Re: Why does data stop getting indexed after a log rotation?

Arkon — Tue, 02 Aug 2016 02:12:30 GMT

You, Sir, made my day (and probably a few more).

Re: Why does data stop getting indexed after a log rotation?

twinspop — Tue, 02 Aug 2016 02:18:52 GMT

Wow, good one to know! Almost seems like a bug, or at least a potential feature enhancement.

Re: Why does data stop getting indexed after a log rotation?

witski — Mon, 16 Apr 2018 16:12:00 GMT

I have the same issue with my suse forwarder 6.4.3.
The log file will be renamed when its size reaches 20MB. And a new log file with the original name will be created to log data. For example, after abc.123.log reaches 20MB, it will be renamed as abc.123.1501.log. And a new abc.123.log will be created.
After the log was rotated, there's a chance that the new data in the log will not be collected. And if you use lsof command to check splunkd's pid, you may find that splunkd opens the renamed history log(abc.123.1501.log), not the right file(abc.123.log).
Is this a bug?

Re: Why does data stop getting indexed after a log rotation?

thirusama — Tue, 29 Sep 2020 22:59:57 GMT

We are on forwarder 7.0.3. We seem to have similar issue. What should be the ideal fix from Splunk monitoring side?
Below is the monitoring stanza

[monitor:///opt/mapr/hadoop/hadoop/logs/nodemanager]
sourcetype = my_st
index = my_index
disabled = 0
ignoreOlderThan = 2h

The latest file will be with yarn-mapr-nodemanager-host_name.log and the latest archived file be with yarn-mapr-nodemanager-host_name.log.1.

What is interesting is intermittently on certain servers, the current file gets indexed only at the time of its roll/archival i.e. lets say after 6 hours. And the issue of live/current file not getting indexed on time does not happen all the time. The next live file might get indexed on time. To me, it sounds like what @sjohnson_splunk has mentioned. But there should be ideal settings to avoid this. Any insights on this will be helpful.

Re: Why does data stop getting indexed after a log rotation?

witski — Tue, 29 Jan 2019 03:16:32 GMT

Hi, @thirusama , I believe you can resolve your problem by changing your monitored path.

There is no need to collect the archived file again since you have already collected the current file (if it functions normally).
Hence, there is no need to monitor the whole path. You can add .log to the end of your current path, which will be "/opt/mapr/hadoop/hadoop/logs/nodemanager/.log".
Doing this will let your splunk forwarder to focus on current files.

And my issue has been confirmed by the official support team as a bug solved since 6.6.4. It's recorded as SPL-142334.