Why does data stop getting indexed after a log rotation?

[Arkon]

Hi,

I noticed that, right after a log rotation, the data is not being indexed anymore.
Data is still going through /var/log/myapp.log and /var/log/messages (rsyslog UDP), so it all arrives on the machine (at 100%), but it is not being indexed.

On Splunk, I am monitoring logs arriving with real-time searches. Before log-rotate, everything is fine and logs are arriving on a regular basis. After logrotate, I do not get anything anymore.

Here is my inputs.conf:

[monitor:///var/log/myapp.log]
sourcetype = myappsourcetype
crcSalt = <SOURCE>
crcSalt = 2048
disabled = 0

My log rotate conf:

"/var/log/myapp.log" {
  monthly
  size 100M
  rotate 30
  compress
  postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2>/dev/null` &> /dev/null || true
  endscript
}

My sourcetype shouldn't be the problem as it only contains some side fields extractions.

Thank you very much in advance

[Arkon]

Thanks! So I removed crcSalt but it is still the same issue: I receive the logs but right after logrotate, they are not arriving anymore.

[twinspop]

Just to be clear... you restarted the forwarder after making this change?

[Arkon]

Yes I removed crcSalt and restarting splunk right after.
I then did a logrotate -f /var/log/myapp.log and then I could not see the new events anymore.

[twinspop]

What do the _internal logs say? Check with index=_internal host=yourhost myapp.log. You should see a series of logs there at the time you forced the log rotation.

[twinspop]

ddrillic: Ding!

[twinspop]

You very very very very likely do not want to use crcSalt. It's the truly rare log file that requires it.

[Arkon]

It looks like that deleting the file and restarting rsyslog fix the issue.
Does that mean the problem is coming from Rsyslog or Splunk "loosing his pointer" to the file?