topic Re: Splunk Stopped Pulling in Getting Data In

Splunk Stopped Pulling

reverse — Wed, 03 Jul 2019 19:10:47 GMT

So here is the issue -

the file (abc.log) which was being pulled into splunk got rolled over.
abc.log became abc.log.1
and a new abc.log got created.the issue was that timestamp of new file(abc.log) didn't change at all for some unknown reasons ..

And ever since that time it stopped pulling .. does this make any sense ?

File will not be read, is too small to match seekptr checksum

Re: Splunk Stopped Pulling

reverse — Wed, 03 Jul 2019 19:11:49 GMT

hey @Vijeta.. hey @jnudell_2 ever faced this issue ?

Re: Splunk Stopped Pulling

reverse — Thu, 04 Jul 2019 00:44:54 GMT

@martin_mueller .. please guide on this. thanks.

Re: Splunk Stopped Pulling

skalliger — Thu, 04 Jul 2019 06:00:43 GMT

People will respond to you when they have time. 🙂

What is your issue exactly? After the file is rolled to *.1, the newly created file does not get indexed, correct? Have you checked other threads? There about a dozen threads about this problem, like increasing initCrcLength or setting crcSalt if initCrcLength isn't doing the job alone (which it should).

Skalli

Re: Splunk Stopped Pulling

reverse — Thu, 04 Jul 2019 17:16:07 GMT

Lets say abc.log has a creation of jan 1 1am .. and it got rolled over ... to abc.log.1

The new abc.log is still having the same create timestamp..

Can this issue alone stop splunk pulling behavior.. are splunk agents timestamp sensitive when it comes to t
Rolling logs ?

Re: Splunk Stopped Pulling

codebuilder — Thu, 04 Jul 2019 17:33:52 GMT

My first guess is that you have logrotate configured on the given filesystem, and that it is set to create a new file at the same time that it rotates the existing log file.

Note, the use of logrotate is indicated by the filename in your output (abc.log became abc.log.1).

If this process happens quickly enough, you can potentially get a new log file with a nearly identical timestamp.
Example:

cat /etc/logrotate.d/yum
        /var/log/yum.log {
            missingok
            notifempty
            size 30k
            yearly
            create 0600 root root
        }

Most applications are intuitive enough to create their own log file if it does not exist, so the use of 'create' is generally not necessary.
Try commenting out that line in your logrotate conf file, cycle the daemon, and

     /var/log/yum.log {
         missingok
         notifempty
         size 30k
         yearly
         # create 0600 root root
     }

systemctl restart rsyslog.service
systemctl status rsyslog.service

The above assumes systemd, and always test in a non-prod environment first obviously.

Re: Splunk Stopped Pulling

skalliger — Fri, 05 Jul 2019 09:41:58 GMT

Yes, they are. You probably need to adjust your inputs.conf when you're running into timestamping issues. You can try using crcSalt = <SOURCE> for this.

Skalli

Re: Splunk Stopped Pulling

DavidHourani — Fri, 05 Jul 2019 10:17:01 GMT

Hi @reverse,

Try setting initCrcLength, if should the trick. It defaults to 256 in input.conf so change it to 512 and see if it solved your issue.

This is explained here :
https://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled

Let me know if that helps.

Cheers,
David