Getting Data In

How to edit my log4j sourcetype configuration on my Splunk forwarder for proper line breaking?

marshallmat
New Member

I have a java app that writes to a log file... I have configured a Splunk forwarder to forward this log (using source type 'log4j' ) to our Splunk indexer (central server located in different location)... things are fine except for searching, which displays multiple events for, what I consider one event. Example of the one 'event' in the log:

2016-04-29 21:46:26,760 INFO  AwsSqsReader:215 - .doTheRead()... MESSAGE BREADCRUMB... TYPE=XXXXX    MessageId:     mmmmm    ReceiptHandle: ddddddd    MD5OfBody:     3893393d93835e123d243903b84cfb34    Body:          {
  "Type" : "Notification",
  "MessageId" : "dddddd",
  "TopicArn" : "arn:aws:sns:us-east-1:dasfdasfdsafdasf:asdfasdfasdf",
  "Subject" : "ddddddddddd",
  "Message" : "{ \"EventCode\": 12345,         \"Summary\": \"llllll\",         \"Node\": \"uuuuuuu\",         \"Severity\": 3,         \"EMSInstance\": \"ddddddd\",         \"Agent\": \"aaaaaaa\",         \"AlertGroup\": \"ddd\",         \"AlertKey\": \"ddddd\",         \"Location\": \"ssssssssss\",         \"EpochTime\": \"1461980781\",         \"GenericString1\": \"abbbb\",         \"GenericString2\": \"12345\"         } ",
  "Timestamp" : "2016-04-30T01:46:22.278Z",
  "SignatureVersion" : "1",
  "Signature" : "xxxxxxxx",
  "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-ddddddd",
  "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=xxxxxxxxx"
}

There are two events shown on the search results page, with the split at the '"Timestamp":...' line.

On the splunk-forwarder host, I have tried to create my own sourcetype, reconfiguring the splunk-forwarder to use 'mylog4j' for this file... and then in $SPLUNKHOME/system/local/props.conf:

[mylog4j]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2},\d{3})
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 25

I restarted the splunk-forwarder, sent a new event thru to the log, tried to re-search. Alas, I still see two events on search results page. I think the sourcetype change worked, since I had to change 'sourcetype=mylog4j' on the search.

How can I remedy this?

thx in advance
marshall

0 Karma

Jeremiah
Motivator

If you are running a universal forwarder, you need to put this props.conf setting on the indexer.

Get Updates on the Splunk Community!

Splunk Mobile: Your Brand-New Home Screen

Meet Your New Mobile Hub  Hello Splunk Community!  Staying connected to your data—no matter where you are—is ...

Introducing Value Insights (Beta): Understand the Business Impact your organization ...

Real progress on your strategic priorities starts with knowing the business outcomes your teams are delivering ...

Enterprise Security (ES) Essentials 8.3 is Now GA — Smarter Detections, Faster ...

As of today, Enterprise Security (ES) Essentials 8.3 is now generally available, helping SOC teams simplify ...