Getting Data In

Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb
Explorer

My Event logs in splunk are getting truncated in the front part.

Is it possible to spllit lines based on below condition.

[logLevel=ERROR] - 2019-03-22 08:00:04,697 +0000 --

log level can be ERROR OR INFO, either one will come in logs.

how to use LINE_BREAKER for this. I tired couple of examples from the posts, but it is not working.

Currently using as before, but not working, event are chopped of from the front.

SHOULD_LINEMERGE=false
TRUNCATE=5000000

Any help please!!

0 Karma
1 Solution

nickhills
Ultra Champion

Use this in your props.conf

SHOULD_LINEMERGE=false
LINE_BREAKER=(^)\[logLevel
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%N3
TRUNCATE=10000 #(or remove this as it's the default)
If my comment helps, please give it a thumbs up!

View solution in original post

0 Karma

nickhills
Ultra Champion

Use this in your props.conf

SHOULD_LINEMERGE=false
LINE_BREAKER=(^)\[logLevel
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%N3
TRUNCATE=10000 #(or remove this as it's the default)
If my comment helps, please give it a thumbs up!
0 Karma

vijayakumarkb
Explorer

Thanks for the Answer Nick..
I added the solun, but it got all the events clubbed into one event.

for Eg :

[logLevel=ERROR] - 2019-03-25 01:39:24,980 +0000 -- "ConsumerService-9-ConnectionPool-Thread-24" com.DeliveryHandler -- message="INFO: First Time" timestamp="1553477964923"
[logLevel=ERROR] - 2019-03-25 01:39:52,094 +0000 -- "ConsumerService-9-ConnectionPool-Thread-25" com.DeliveryHandler -- message="INFO: First Time" timestamp="1553477992048"

both above events got merged and came as single event.

0 Karma

vijayakumarkb
Explorer

should i be using %Y-%m-%d %H:%M:%S:%3N

0 Karma

nickhills
Ultra Champion

Tbh, I’m not sure how , is handled vs . in the time format. Worth a try, I can’t see what else is wrong.
I presume the events are split by line in the actual source file?

If my comment helps, please give it a thumbs up!
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Use below props.conf, its almost same as provided by @nickhillscpl but timestamp config is fixed.

[yoursourcetype]
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3N %z
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
MAX_TIMESTAMP_LOOKAHEAD=29
LINE_BREAKER=(^)\[logLevel
0 Karma

yadavmalay
Observer

Logs are truncated at beginning

e WorkDay CW Data" transaction_start_epoch="1597257375.0102034" execution_id="87e92c54-dcca-11ea-8c01-0050568d9e34" browser="HeadlessChrome" browser_version="84.0.4147" os="Windows" os_version="10" ip="10.24.85.121" title="Horizon ACM - Custom Task: Synchronize  Data" app_name="XYZ

Below are the props.conf file I am using at universal forwarder side.

SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3N %z
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
MAX_TIMESTAMP_LOOKAHEAD=29
LINE_BREAKER=(^)\[logLevel

 

Any suggestion please.

0 Karma

vijayakumarkb
Explorer

Thanks Harsh for the help. let me test it

0 Karma

vijayakumarkb
Explorer

let me try with %Y-%m-%d %H:%M:%S,%N3

0 Karma

vijayakumarkb
Explorer

sorry %Y-%m-%d %H:%M:%S,%3N

0 Karma

vijayakumarkb
Explorer

I presume the events are split by line in the actual source file?

Yes events are split

0 Karma

nickhills
Ultra Champion

Yikes, you don't want events that are 5MB each!

If my comment helps, please give it a thumbs up!
0 Karma

vijayakumarkb
Explorer

Eg :

[logLevel=ERROR] - 2019-03-22 11:30:04,100 +0000 -- "ConsumerService-10-Thread-9" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."
[logLevel=ERROR] - 2019-03-22 11:30:04,201 +0000 -- "ConsumerService-10-Thread-7" MasterService -- message="INFO SEND"

it is not coming as two separate events in Splunk always. getting trimmed of like as below

erService-10-Thread-29" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."

0 Karma

nickhills
Ultra Champion

Can you post some example events, it sounds like you just need to configure your breaking settings correctly

If my comment helps, please give it a thumbs up!
0 Karma

vijayakumarkb
Explorer

[logLevel=ERROR] - 2019-03-22 11:30:04,100 +0000 -- "ConsumerService-10-Thread-9" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."
[logLevel=ERROR] - 2019-03-22 11:30:04,201 +0000 -- "ConsumerService-10-Thread-7" MasterService -- message="INFO SEND"

it is not coming as two separate events in Splunk always. getting trimmed of like as below

erService-10-Thread-29" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."

0 Karma

vijayakumarkb
Explorer

Any help for this

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...

SplunkTrust | 2024 SplunkTrust Application Period is Open!

It's that time again, folks! That's right, the application/nomination period for the 2024 SplunkTrust is ...