Getting Data In

Please assist in LINE_BREAKER stanza in `props.conf` for custom app

Neur0mencer
Explorer

Dears,
I have an app which generates logs in following pattern:

----------------------------------------
Timestamp: 2019-08-23 14:00:01.545 
UserLogin: ascache 
UserId: -1 
Severity: Information 
Message: REQUEST


----------------------------------------
----------------------------------------
Timestamp: 2019-08-23 14:00:02.949 
UserLogin: Gistt 
UserId: -1 
Severity: Information 
Message: REQUEST


----------------------------------------
----------------------------------------
Timestamp: 2019-08-23 14:00:24.634 
UserLogin: rerwer 
UserId: 3254 
Severity: Information 
Message: xxxx


----------------------------------------

Every event is encapsulated betwen ---------------------------------------- characters. I'm trying to have Splunk break this accordingly, but I'm failing.
I've tried the following stanzas (commented out are ones that failed, uncommented also fail...):

# [APP-Trace.log]
# LINE_BREAKER = \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
# 
# [APP-PerformanceLogs.log]
# LINE_BREAKER = \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

# # 13:51 2019-08-29
# [APP-Trace.log]
# LINE_BREAKER = \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\s+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
# [APP-PerformanceLogs.log]
# LINE_BREAKER = \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\s+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

[APP-Trace.log]
LINE_BREAKER = \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
SHOULD_LINEMERGE = false

[APP-PerformanceLogs.log]
LINE_BREAKER = \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
SHOULD_LINEMERGE = false

Can you please assist how to properly capture this?

Thank you,

0 Karma

woodcock
Esteemed Legend

You must use a capture group; try this in props.conf:

SHOULD_LINEMERGE = false
LINE_BREAKER = ((?:[\s\r\n]+-{40})+[\r\n\s]*)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Neur0mencer,
at first you don't need to repeat 40 times the char, you can use {40} .
Anyway, i'd use a different approach:

[APP-Trace.log]
SHOULD_LINEMERGE = True
TIME_PREFIX = Timestamp:\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N

This give a break every time finds a Timestamp.

Bye.
Giuseppe

Neur0mencer
Explorer

Thank you kindly for the swift reply, this does look way better.
Is there a way to cut the unnecesary line at the end? This method of event breaking still leaves in the ---------------------------------------- in the event, which is unnecessary.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Neur0mencer,
you can see at https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/Anonymizedata
anyway in you props.conf insert

SEDCMD-lines = s/\-{40}/\s/g

This replaces your lines with a space.

Bye.
Giuseppe

P.S.: if this answer solves your problem, please accept and/ot upvote it.

0 Karma

Neur0mencer
Explorer

This is "almost" working. Some events come "clean" (just lines with values), but others still contain the 40 dash'es for some reason.
I wanted to upload a screenshot, but there's no such option without uploading the image to a web server (or I'm missing that too).

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi Neur0mencer,
check (in search) if the regex that you used for replacing chars runs in all you logs, if not (maybe you have a different number of chars) so you can change your replacing regex.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...