Getting Data In

logs being cutoff

ralphw_SAIC
Path Finder

Running Splunk Enterprise and Splunkforwarder, both on RHEL, and we are having issues with the front portion of some logs being cutoff while the back half remains and gets indexed. The datetime stamp and server name remains, but then the front half is removed. This occurs randomly for different events.

This is an example from the same server and timestamp:
From localhost
audispd: node=localhost type=SYSCALL msg=audit(1457382989.281:3703928): arch=c000003e syscall=91 success=yes exit=0 a0=3 a1=100 a2=0 a3=7fffdedde310 items=1 ppid=2866 pid=2881 auid=4094 uid=4094 gid=518
8 euid=4094 suid=4094 fsuid=4094 egid=5188 sgid=5188 fsgid=5188 tty=(none) ses=11541 comm="betaGraph.ksh" exe="/bin/ksh93" key=(null)

From Splunk
=0 a3=7fffdeddec90 items=1 ppid=2866 pid=2881 auid=4094 uid=4094 gid=5188 euid=4094 suid=4094 fsuid=4094 egid=5188 sgid=5188 fsgid=5188 tty=(none) ses=11541 comm="betaGraph.ksh" exe="/bin/ksh93" key=(null)

0 Karma

skoelpin
SplunkTrust
SplunkTrust

So these are log files from the same server, some of the events are being cutoff while other events are correct? Can you see if they have different sourcetypes? If so then you will need to edit your inputs.conf and change the sourcetype or edit your props.conf and add the linebreaking for that other sourcetype

0 Karma

ralphw_SAIC
Path Finder

There are multiples of these type logs in /var/log/messages. The only difference is the timestamp on them. Some come through ok and some get the leading portion cutoff.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Go into Splunk and compare the events which are being cutoff vs the events that are not being cutoff. When doing this comparison, look at the sourcetypes (There should be a pre-extracted field called sourcetype). If the sourcetypes are different then its getting cutoff when being indexed. You can fix this by modifying your props.conf

0 Karma

ralphw_SAIC
Path Finder

They are both the same sourcetype, linux_messages_syslog.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Can you post your props.conf stanza?

0 Karma

ralphw_SAIC
Path Finder

I do not have a local props.conf file, just the default props.conf.

0 Karma

rosplunk07
Observer

Hey @ralphw_SAIC ... You got any solution on this? I am facing the same issue, some random logs are being cutoff intermittently from the start.
Thanks.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...