topic Re: log4j truncating the log entry in Getting Data In

log4j truncating the log entry

bohrasaurabh — Mon, 16 Dec 2013 22:31:24 GMT

We are noticing some of the log entries which are getting truncated. we are using the log4j sourcetype.

actual log entry looks like below, however several times we will only see first two lines and line starting with Title: onwards will be truncated. Any ideas how to fix it.

Splunk and forwarder both are 5.0.3

2013-12-10 10:11:27,986 INFO [something.here] :-) Transfer successful! Bytes: 508,174,896, ET: 0:00:12.604 ID: 1f1496c2-cea5-4148-ade2-e625ef6a2e82 Title: ABCD - 11/23/12 EFGH - Something HERE - username (00:11:48;00 - 00:12:22;00) SRC: source.name:host=my.fqdn.hostname,path=/path/to/file.txt,port=21,type=TypeOfFile DEST: destination.name.1001:host=10.11.12.13,name=servername,path=/1111/,poolId=2222,port=21,type=Container,zoneId=1001

Re: log4j truncating the log entry

somesoni2 — Mon, 16 Dec 2013 22:42:54 GMT

Is the event breaking properly configured?? can you provide props.conf values for log4j sourcetype?

Re: log4j truncating the log entry

bohrasaurabh — Mon, 16 Dec 2013 23:21:05 GMT

I do not have any custom props and transforms. The only thing we have defined is local inputs.conf. we are using predefined sourcetype - http://docs.splunk.com/Documentation/Splunk/5.0.3/Data/Listofpretrainedsourcetypes

[monitor:///data/web/logs/jboss/.../server.log]
host_segment = 5
sourcetype = log4j
index = myIndex

ignoreOlderThan = 30d

disabled = false

Re: log4j truncating the log entry

somesoni2 — Mon, 16 Dec 2013 23:36:40 GMT

Ok...and do you see any specific pattern from which the events are truncated? May be after a timestamp like field? Would be great if you can provide some example of truncated event and corresponding full event.

Re: log4j truncating the log entry

bohrasaurabh — Wed, 18 Dec 2013 22:03:26 GMT

i dont have enough karmas to upload images. however i do not see any pattern in the logs which would lead to this situation.

Re: log4j truncating the log entry

yannK — Mon, 28 Sep 2020 15:31:02 GMT

if the events are truncated in the middle of the line, it can be that your application has a write buffer.
See the setting time_before_close in inputs.conf

and this answer
http://answers.splunk.com/answers/81385/events-from-my-universal-forwarder-are-getting-random-linefeeds-inserted-into-the-events-why-is-this-happening-and-how-do-i-fix-it

Re: log4j truncating the log entry

yannK — Thu, 19 Dec 2013 22:28:18 GMT

after testing with sample, the issue can be also an event line breaking
please try with this custom log4j sourcetype in props.conf
[customlog4j] BREAK_ONLY_BEFORE=^\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{3} MAX_TIMESTAMP_LOOKAHEAD=30 NO_BINARY_CHECK=1 maxDist=75 pulldown_type=true