Splunk Search

Why is line breaking not working as expected ?

romainbouajila
Path Finder

Hi,
I would like to break my logs at every time + log level but it is not working as expected.
Here's my props.conf :

[log_name]
TIME_FORMAT = %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 12
NO_BINARY_CHECK = true
disabled = false
MAX_EVENTS = 10240 
TRUNCATE = 0

[other_name]
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
MAX_TIMESTAMP_LOOKAHEAD = 32
disabled = false
TRUNCATE = 0
MAX_EVENTS = 10240

My log file :

09:31:51,359 | INFO  | 640512999-933058 | someinformation                  | 72 - text.texttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttext
09:31:51,359 | WARN  | 640512999-933058 | someinformation                  | 204 - texttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttext
09:31:51,359 | INFO  | 640512999-933058 | someinformation                  | 72 - texttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttext
09:31:51,359 | INFO  | 640512999-933058 | someinformation                  | 72 - texttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttext
09:31:51,359 | INFO  | 640512999-933058 | someinformation                  | 243 - texttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttext
09:31:51,360 | INFO  | 640512999-933058 | someinformation                  | 243 - texttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttext
09:31:51,361 | INFO  | 640512999-933058 | someinformation                  | 240 - 

I tried to use BREAK_ONLY_BEFORE option in props.conf with the following regexp : (?:[01]\d|2[0-3]):(?:[0-5]\d):(?:[0-5]\d),[0-9]{3}.\|.*?\|
Same behavior with LINE_BREAKER option.
Can someone give me a hint on how to configure my props.conf ? What am I missing ?

Tags (1)
0 Karma

whrg
Motivator

Hello @romainbouajila,

This article Configure event line breaking goes into detail on how line breaking works.

The dates in your logs (a space character at the beginning of each line? and no year/month/day) look peculiar. You will need to set the timestamp extraction manually. The rest should work fine. By default, SHOULD_LINEMERGE and BREAK_ONLY_BEFORE_DATE are set to true.

Screenshot: https://ibb.co/JCtK7hw

This is the props.conf:

[log_name]
category = Custom
pulldown_type = true
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
TIME_PREFIX = \s*
TIME_FORMAT = %H:%M:%S,%3N
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
0 Karma

romainbouajila
Path Finder

Hello, thank you for your help !
You generated this props.conf from my log file example ?

0 Karma

whrg
Motivator

Yes.
%H:%M:%S,%3N will match your time format "09:31:51,359".

0 Karma

romainbouajila
Path Finder

Hello,
Basically, my logs looks like the following.
Sometimes it is a 1 line event, sometimes it is several lines. I would like to break every time there's a timestamp at the beginning of the line (cf picture)
In addition, my logs are not starting with a whitespace, it might be due to a bad copy/paste from me. So I changed the TIME_PREFIX from "\s*" to "^"




texttexttext
text text text




11:35:11,715 | INFO | 12345678-12345 | texttexttexttext | 107 - texttexttexttexttexttext | Outbound Message

ID: xxxxxxx
Address: http://url
Encoding: UTF-8
Http-Method: POST
Content-Type: text/xml
Headers: {Accept=[/], SOAPAction=[""]}
Payload:

<texttexttexttexttexttexttexttexttexttexttexttext"texttexttexttexttexttexttexttexttexttext">
  <texttext>
    <text>text</text>
    <text>text</text>
    <text/>
    <text/>
    <text>text</text>
    <text>001</text>
    <text/>
    <text>texttext</text>
    <OPERATION>QueryCardDtlsLst</OPERATION>
    <SOURCE_OPERATION/>
    <SOURCE_USERID/>

[...]

08:20:22,972 | INFO | 12345678-234567890 | texttexttexttext | texttexttexttexttexttexttexttexttext | Outbound Message

ID: 1234
Response-Code: 200
Content-Type: application/json
Headers: {Content-Type=[application/json], Date=[Mon, 06 Jan 2020 01:20:22 GMT]}

Payload: {"texttexttext":{"texttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttexttext}

08:20:24,862 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,862 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | WARN | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,866 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,867 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,867 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla


ID: 123456
Address: texttexttexttexttexttexttexttext
Encoding: texttext
Http-Method: POST
Content-Type: application/json
Headers: {Accept=[application/json], texttexttexttexttexttexttexttexttexttexttexttexttexttexttext}

Payload: {"startDate":"2020-01-01T00:00:00.000+0700","endDate":"2020-01-06T23:59:59.000+0700","pageNumber":1,"pageSize":300}

08:20:24,862 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,862 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | INFO | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla
08:20:24,865 | WARN | 12345678-234567890 | texttexttexttexttexttext | blablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablablabla

alt text

0 Karma

whrg
Motivator

@romainbouajila I edited my answer. Have a look at the screenshot and the new props.conf.

0 Karma

romainbouajila
Path Finder

Hi dear whrg,

I tried your props.conf and I am still having the same issue. For instance, I keep having timestamps at the end of some events like in the screenshot below (sorry, if not readable I can send another one)
alt text

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @romainbouajila,
try to add to your props.conf:

TIME_PREFIX = ^

Ciao.
Giuseppe

0 Karma

romainbouajila
Path Finder

Should I add this option with LINE_BREAKER or BREAK_ONLY_BEFORE ?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @romainbouajila,
if each of your events is in one row (in other words, if it has LF CR) you don't need of LINE BREAK, it's useful when you have multilines events.
Splunk divide events using the date, in this way you say to Splunk that the date is at the start of the row.
Add this oprtion to your props anche check the results, it should be sufficient.
Don't add also the other options.

Ciao.
Giuseppe

0 Karma

romainbouajila
Path Finder

Hi @gcusello !
Thank you for your help. I tried your solution but it doesn' t work.
Maybe I applied it wrong. What do you mean by "Don't add also the other options." ? I should use only TIME_PREFIX = ^ and not TIME_FORMAT etc ?

Thanks in advance

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @romainbouajila,
Sorry, I wasn't clear, try this:

[log_name]
 TIME_FORMAT = %H:%M:%S,%3N
 MAX_TIMESTAMP_LOOKAHEAD = 12
 NO_BINARY_CHECK = true
 disabled = false
 MAX_EVENTS = 10240 
 TRUNCATE = 0
 TIME_PREFIX = ^

Ciao.
Giuseppe

0 Karma

romainbouajila
Path Finder

I also tried this props.conf as suggested by @whrg :

[log]
pulldown_type = true
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
TIME_PREFIX = ^
TIME_FORMAT = %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 12
SHOULD_LINEMERGE = false (& true)
BREAK_ONLY_BEFORE_DATE = true

but I had the same result as before.
How long after restarting Splunk service should I check my logs ?

0 Karma

romainbouajila
Path Finder

I tried your solution, I see improvement but I still have some weird behavior.
For example, it still breaks before the "Header" line, and I can't explain or understand why.
The following picture is where I would like logs to be cut FYI.
Do you have any idea ?
Thanks a lot for your help !

alt text

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...