I have a java app that writes to a log file... I have configured a Splunk forwarder to forward this log (using source type 'log4j' ) to our Splunk indexer (central server located in different location)... things are fine except for searching, which displays multiple events for, what I consider one event. Example of the one 'event' in the log:
2016-04-29 21:46:26,760 INFO AwsSqsReader:215 - .doTheRead()... MESSAGE BREADCRUMB... TYPE=XXXXX MessageId: mmmmm ReceiptHandle: ddddddd MD5OfBody: 3893393d93835e123d243903b84cfb34 Body: {
"Type" : "Notification",
"MessageId" : "dddddd",
"TopicArn" : "arn:aws:sns:us-east-1:dasfdasfdsafdasf:asdfasdfasdf",
"Subject" : "ddddddddddd",
"Message" : "{ \"EventCode\": 12345, \"Summary\": \"llllll\", \"Node\": \"uuuuuuu\", \"Severity\": 3, \"EMSInstance\": \"ddddddd\", \"Agent\": \"aaaaaaa\", \"AlertGroup\": \"ddd\", \"AlertKey\": \"ddddd\", \"Location\": \"ssssssssss\", \"EpochTime\": \"1461980781\", \"GenericString1\": \"abbbb\", \"GenericString2\": \"12345\" } ",
"Timestamp" : "2016-04-30T01:46:22.278Z",
"SignatureVersion" : "1",
"Signature" : "xxxxxxxx",
"SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-ddddddd",
"UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=xxxxxxxxx"
}
There are two events shown on the search results page, with the split at the '"Timestamp":...' line.
On the splunk-forwarder host, I have tried to create my own sourcetype, reconfiguring the splunk-forwarder to use 'mylog4j' for this file... and then in $SPLUNKHOME/system/local/props.conf:
[mylog4j]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2},\d{3})
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 25
I restarted the splunk-forwarder, sent a new event thru to the log, tried to re-search. Alas, I still see two events on search results page. I think the sourcetype change worked, since I had to change 'sourcetype=mylog4j' on the search.
How can I remedy this?
thx in advance
marshall
If you are running a universal forwarder, you need to put this props.conf setting on the indexer.