topic Re: logs being cutoff in Getting Data In

logs being cutoff

ralphw_SAIC — Tue, 08 Mar 2016 14:58:51 GMT

Running Splunk Enterprise and Splunkforwarder, both on RHEL, and we are having issues with the front portion of some logs being cutoff while the back half remains and gets indexed. The datetime stamp and server name remains, but then the front half is removed. This occurs randomly for different events.

This is an example from the same server and timestamp:
From localhost
audispd: node=localhost type=SYSCALL msg=audit(1457382989.281:3703928): arch=c000003e syscall=91 success=yes exit=0 a0=3 a1=100 a2=0 a3=7fffdedde310 items=1 ppid=2866 pid=2881 auid=4094 uid=4094 gid=518
8 euid=4094 suid=4094 fsuid=4094 egid=5188 sgid=5188 fsgid=5188 tty=(none) ses=11541 comm="betaGraph.ksh" exe="/bin/ksh93" key=(null)

From Splunk
=0 a3=7fffdeddec90 items=1 ppid=2866 pid=2881 auid=4094 uid=4094 gid=5188 euid=4094 suid=4094 fsuid=4094 egid=5188 sgid=5188 fsgid=5188 tty=(none) ses=11541 comm="betaGraph.ksh" exe="/bin/ksh93" key=(null)

Re: logs being cutoff

skoelpin — Tue, 08 Mar 2016 16:20:39 GMT

So these are log files from the same server, some of the events are being cutoff while other events are correct? Can you see if they have different sourcetypes? If so then you will need to edit your inputs.conf and change the sourcetype or edit your props.conf and add the linebreaking for that other sourcetype

Re: logs being cutoff

ralphw_SAIC — Wed, 09 Mar 2016 12:45:33 GMT

There are multiples of these type logs in /var/log/messages. The only difference is the timestamp on them. Some come through ok and some get the leading portion cutoff.

Re: logs being cutoff

skoelpin — Wed, 09 Mar 2016 15:17:16 GMT

Go into Splunk and compare the events which are being cutoff vs the events that are not being cutoff. When doing this comparison, look at the sourcetypes (There should be a pre-extracted field called sourcetype). If the sourcetypes are different then its getting cutoff when being indexed. You can fix this by modifying your props.conf

Re: logs being cutoff

ralphw_SAIC — Tue, 29 Sep 2020 09:04:16 GMT

They are both the same sourcetype, linux_messages_syslog.

Re: logs being cutoff

skoelpin — Wed, 09 Mar 2016 21:23:02 GMT

Can you post your props.conf stanza?

Re: logs being cutoff

ralphw_SAIC — Thu, 10 Mar 2016 13:51:29 GMT

I do not have a local props.conf file, just the default props.conf.

Re: logs being cutoff

rosplunk07 — Wed, 29 Jan 2020 06:57:37 GMT

Hey @ralphw_SAIC ... You got any solution on this? I am facing the same issue, some random logs are being cutoff intermittently from the start.
Thanks.