logs being cutoff

[ralphw_SAIC]

Running Splunk Enterprise and Splunkforwarder, both on RHEL, and we are having issues with the front portion of some logs being cutoff while the back half remains and gets indexed. The datetime stamp and server name remains, but then the front half is removed. This occurs randomly for different events.

This is an example from the same server and timestamp:
From localhost
audispd: node=localhost type=SYSCALL msg=audit(1457382989.281:3703928): arch=c000003e syscall=91 success=yes exit=0 a0=3 a1=100 a2=0 a3=7fffdedde310 items=1 ppid=2866 pid=2881 auid=4094 uid=4094 gid=518
8 euid=4094 suid=4094 fsuid=4094 egid=5188 sgid=5188 fsgid=5188 tty=(none) ses=11541 comm="betaGraph.ksh" exe="/bin/ksh93" key=(null)

From Splunk
=0 a3=7fffdeddec90 items=1 ppid=2866 pid=2881 auid=4094 uid=4094 gid=5188 euid=4094 suid=4094 fsuid=4094 egid=5188 sgid=5188 fsgid=5188 tty=(none) ses=11541 comm="betaGraph.ksh" exe="/bin/ksh93" key=(null)

[skoelpin]

So these are log files from the same server, some of the events are being cutoff while other events are correct? Can you see if they have different sourcetypes? If so then you will need to edit your inputs.conf and change the sourcetype or edit your props.conf and add the linebreaking for that other sourcetype

[ralphw_SAIC]

There are multiples of these type logs in /var/log/messages. The only difference is the timestamp on them. Some come through ok and some get the leading portion cutoff.

[skoelpin]

Go into Splunk and compare the events which are being cutoff vs the events that are not being cutoff. When doing this comparison, look at the sourcetypes (There should be a pre-extracted field called sourcetype). If the sourcetypes are different then its getting cutoff when being indexed. You can fix this by modifying your props.conf

[ralphw_SAIC]

They are both the same sourcetype, linux_messages_syslog.

[skoelpin]

Can you post your props.conf stanza?

[ralphw_SAIC]

I do not have a local props.conf file, just the default props.conf.

[rosplunk07]

Hey @ralphw_SAIC ... You got any solution on this? I am facing the same issue, some random logs are being cutoff intermittently from the start.
Thanks.