topic Re: Splunk Events getting truncated/choppedoff at the begining in Getting Data In

Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb — Fri, 22 Mar 2019 09:27:08 GMT

My Event logs in splunk are getting truncated in the front part.

Is it possible to spllit lines based on below condition.

[logLevel=ERROR] - 2019-03-22 08:00:04,697 +0000 --

log level can be ERROR OR INFO, either one will come in logs.

how to use LINE_BREAKER for this. I tired couple of examples from the posts, but it is not working.

Currently using as before, but not working, event are chopped of from the front.

SHOULD_LINEMERGE=false
TRUNCATE=5000000

Any help please!!

Re: Splunk Events getting truncated/choppedoff at the begining

nickhills — Fri, 22 Mar 2019 13:05:09 GMT

Yikes, you don't want events that are 5MB each!

Re: Splunk Events getting truncated/choppedoff at the begining

nickhills — Fri, 22 Mar 2019 13:06:06 GMT

Can you post some example events, it sounds like you just need to configure your breaking settings correctly

Re: Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb — Fri, 22 Mar 2019 17:27:15 GMT

Eg :

[logLevel=ERROR] - 2019-03-22 11:30:04,100 +0000 -- "ConsumerService-10-Thread-9" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."
[logLevel=ERROR] - 2019-03-22 11:30:04,201 +0000 -- "ConsumerService-10-Thread-7" MasterService -- message="INFO SEND"

it is not coming as two separate events in Splunk always. getting trimmed of like as below

erService-10-Thread-29" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."

Re: Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb — Mon, 25 Mar 2019 09:11:05 GMT

Any help for this

Re: Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb — Mon, 25 Mar 2019 09:11:27 GMT

it is not coming as two separate events in Splunk always. getting trimmed of like as below

erService-10-Thread-29" MasterService -- requestId="abc234" requestType="Register" message="Reporting - Success."

Re: Splunk Events getting truncated/choppedoff at the begining

nickhills — Mon, 25 Mar 2019 09:41:05 GMT

Use this in your props.conf

SHOULD_LINEMERGE=false
LINE_BREAKER=(^)\[logLevel
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%N3
TRUNCATE=10000 #(or remove this as it's the default)

Re: Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb — Mon, 25 Mar 2019 12:53:26 GMT

Thanks for the Answer Nick..
I added the solun, but it got all the events clubbed into one event.

for Eg :

[logLevel=ERROR] - 2019-03-25 01:39:24,980 +0000 -- "ConsumerService-9-ConnectionPool-Thread-24" com.DeliveryHandler -- message="INFO: First Time" timestamp="1553477964923"
[logLevel=ERROR] - 2019-03-25 01:39:52,094 +0000 -- "ConsumerService-9-ConnectionPool-Thread-25" com.DeliveryHandler -- message="INFO: First Time" timestamp="1553477992048"

both above events got merged and came as single event.

Re: Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb — Mon, 25 Mar 2019 13:06:06 GMT

should i be using %Y-%m-%d %H:%M:%S:%3N

Re: Splunk Events getting truncated/choppedoff at the begining

nickhills — Mon, 25 Mar 2019 13:12:08 GMT

Tbh, I’m not sure how , is handled vs . in the time format. Worth a try, I can’t see what else is wrong.
I presume the events are split by line in the actual source file?

Re: Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb — Mon, 25 Mar 2019 13:17:41 GMT

let me try with %Y-%m-%d %H:%M:%S,%N3

Re: Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb — Mon, 25 Mar 2019 13:18:04 GMT

sorry %Y-%m-%d %H:%M:%S,%3N

Re: Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb — Mon, 25 Mar 2019 13:19:14 GMT

I presume the events are split by line in the actual source file?

Yes events are split

Re: Splunk Events getting truncated/choppedoff at the begining

harsmarvania57 — Mon, 25 Mar 2019 13:23:19 GMT

Use below props.conf, its almost same as provided by @nickhillscpl but timestamp config is fixed.

[yoursourcetype]
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3N %z
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
MAX_TIMESTAMP_LOOKAHEAD=29
LINE_BREAKER=(^)\[logLevel

Re: Splunk Events getting truncated/choppedoff at the begining

vijayakumarkb — Tue, 26 Mar 2019 05:11:14 GMT

Thanks Harsh for the help. let me test it

Re: Splunk Events getting truncated/choppedoff at the begining

yadavmalay — Wed, 12 Aug 2020 18:44:53 GMT

Logs are truncated at beginning

e WorkDay CW Data" transaction_start_epoch="1597257375.0102034" execution_id="87e92c54-dcca-11ea-8c01-0050568d9e34" browser="HeadlessChrome" browser_version="84.0.4147" os="Windows" os_version="10" ip="10.24.85.121" title="Horizon ACM - Custom Task: Synchronize Data" app_name="XYZ

Below are the props.conf file I am using at universal forwarder side.

SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3N %z
TIME_PREFIX=^\[logLevel=\w+\]\s-\s
MAX_TIMESTAMP_LOOKAHEAD=29
LINE_BREAKER=(^)\[logLevel

Any suggestion please.