Getting Data In

How to configure Splunk to ignore a timestamp to count two lines of data as one event?

MikeBertelsen
Communicator

Here is a sampling of the log data. Lines 1-3 had no issues, but line 4 issues a warning and dumps the problem onto line 5. Splunk needs to treat line 5 as part of Line 4 and not index the timestamp for line 5. Suggestions please.

2015-09-11 04:24:05,568 INFO  [nexj.model.class.LDAPGRP] (WorkManager(2)-91:dispatcher ) End getSnapshot for LDAPGRP
2015-09-11 04:24:05,568 INFO  [nexj.model.class.LDAPUSR] (WorkManager(2)-91:dispatcher ) Start getSnapshot for LDAPUSR.
2015-09-11 04:24:05,977 INFO  [nexj.model.class.LDAPUSR] (WorkManager(2)-91:dispatcher ) End getSnapshot for LDAPUSR
2015-09-11 04:42:47,652 WARN  [nexj.core.rpc.http.session.SessionManager] (servername%2F-IPAdress:/project-8009-7:/apex) Acquring lock exceeded max wait time of 30720s. Diagnostic dump:
2015-09-10 11:39:47.312: (Thread: servername%2F-IPAdress:/project-8009-15 Owner: Thread[servername%2F-IPAdress:/project-8009-15,5,jboss] Lock Count: 1 Ref Count: 0 Session: 488E74B020DAC9F33CD503F058947C6A.server) Acqured lock
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Give this a try in your sourcetype definition in props.conf (on Indexer/Heavy Forwarder)

[YourSourceType]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
LINE_BREAKER=([\r\n]+)(?=\d+-\d+-\d+ \d+:\d+:\d+,\d+ \w+)
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%Q
MAX_TIMESTAMP_LOOKAHEAD=30

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

Give this a try in your sourcetype definition in props.conf (on Indexer/Heavy Forwarder)

[YourSourceType]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
LINE_BREAKER=([\r\n]+)(?=\d+-\d+-\d+ \d+:\d+:\d+,\d+ \w+)
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%Q
MAX_TIMESTAMP_LOOKAHEAD=30

MikeBertelsen
Communicator

This did work. Can i get a breakdown of the various lines and what they do?
I get Time_format but the others are unclear.

Thanks,
Mike

0 Karma

somesoni2
SplunkTrust
SplunkTrust

LINE_BREAKER - provides a delimiter which will break input stream into separate events, default is new line.
I've set it to match new line char + string with this format 2015-09-11 04:24:05,568 INFO. Thus excluding intermediate timestamp which doesn't have log level.
SHOULD_LINEMERGE=provide where multiple lines should be merged into one events. Since I'm using custom LINE_BREAKER, it should be false
MAX_TIMESTAMP_LOOKAHEAD=After Timestamp is location, for how many characters the timestamp value should be looked upon. Based on your date (23 char), I just set to near by whole figure of 30.

0 Karma

edrivera3
Builder

Use a regex in props.conf.

0 Karma

edrivera3
Builder

Hope this helps:
2015-09-11 04:24:05,977 INFO : \d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\sINFO|WARN

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...