log4j truncating the log entry

[bohrasaurabh]

We are noticing some of the log entries which are getting truncated. we are using the log4j sourcetype.

actual log entry looks like below, however several times we will only see first two lines and line starting with Title: onwards will be truncated. Any ideas how to fix it.

Splunk and forwarder both are 5.0.3

2013-12-10 10:11:27,986 INFO [something.here] 🙂 Transfer successful! Bytes: 508,174,896, ET: 0:00:12.604
ID: 1f1496c2-cea5-4148-ade2-e625ef6a2e82
Title: ABCD - 11/23/12 EFGH - Something HERE - username (00:11:48;00 - 00:12:22;00)
SRC: source.name:host=my.fqdn.hostname,path=/path/to/file.txt,port=21,type=TypeOfFile
DEST: destination.name.1001:host=10.11.12.13,name=servername,path=/1111/,poolId=2222,port=21,type=Container,zoneId=1001

[yannK]

if the events are truncated in the middle of the line, it can be that your application has a write buffer.
See the setting time_before_close in inputs.conf

and this answer
http://answers.splunk.com/answers/81385/events-from-my-universal-forwarder-are-getting-random-linefe...

[yannK]

after testing with sample, the issue can be also an event line breaking
please try with this custom log4j sourcetype in props.conf

[customlog4j]
BREAK_ONLY_BEFORE=^\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{3}
MAX_TIMESTAMP_LOOKAHEAD=30
NO_BINARY_CHECK=1
maxDist=75
pulldown_type=true


[bohrasaurabh]

i dont have enough karmas to upload images. however i do not see any pattern in the logs which would lead to this situation.

[somesoni2]

Ok...and do you see any specific pattern from which the events are truncated? May be after a timestamp like field? Would be great if you can provide some example of truncated event and corresponding full event.

[bohrasaurabh]

I do not have any custom props and transforms. The only thing we have defined is local inputs.conf. we are using predefined sourcetype - http://docs.splunk.com/Documentation/Splunk/5.0.3/Data/Listofpretrainedsourcetypes

[monitor:///data/web/logs/jboss/.../server.log]
host_segment = 5
sourcetype = log4j
index = myIndex

ignoreOlderThan = 30d

disabled = false

[somesoni2]

Is the event breaking properly configured?? can you provide props.conf values for log4j sourcetype?