Getting Data In

log4j truncating the log entry

bohrasaurabh
Communicator

We are noticing some of the log entries which are getting truncated. we are using the log4j sourcetype.

actual log entry looks like below, however several times we will only see first two lines and line starting with Title: onwards will be truncated. Any ideas how to fix it.

Splunk and forwarder both are 5.0.3

2013-12-10 10:11:27,986 INFO [something.here] 🙂 Transfer successful! Bytes: 508,174,896, ET: 0:00:12.604
ID: 1f1496c2-cea5-4148-ade2-e625ef6a2e82
Title: ABCD - 11/23/12 EFGH - Something HERE - username (00:11:48;00 - 00:12:22;00)
SRC: source.name:host=my.fqdn.hostname,path=/path/to/file.txt,port=21,type=TypeOfFile
DEST: destination.name.1001:host=10.11.12.13,name=servername,path=/1111/,poolId=2222,port=21,type=Container,zoneId=1001

Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

if the events are truncated in the middle of the line, it can be that your application has a write buffer.
See the setting time_before_close in inputs.conf

and this answer
http://answers.splunk.com/answers/81385/events-from-my-universal-forwarder-are-getting-random-linefe...

yannK
Splunk Employee
Splunk Employee

after testing with sample, the issue can be also an event line breaking
please try with this custom log4j sourcetype in props.conf

[customlog4j]
BREAK_ONLY_BEFORE=^\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{3}
MAX_TIMESTAMP_LOOKAHEAD=30
NO_BINARY_CHECK=1
maxDist=75
pulldown_type=true

0 Karma

bohrasaurabh
Communicator

i dont have enough karmas to upload images. however i do not see any pattern in the logs which would lead to this situation.

0 Karma

somesoni2
Revered Legend

Ok...and do you see any specific pattern from which the events are truncated? May be after a timestamp like field? Would be great if you can provide some example of truncated event and corresponding full event.

0 Karma

bohrasaurabh
Communicator

I do not have any custom props and transforms. The only thing we have defined is local inputs.conf. we are using predefined sourcetype - http://docs.splunk.com/Documentation/Splunk/5.0.3/Data/Listofpretrainedsourcetypes

[monitor:///data/web/logs/jboss/.../server.log]
host_segment = 5
sourcetype = log4j
index = myIndex

ignoreOlderThan = 30d

disabled = false

0 Karma

somesoni2
Revered Legend

Is the event breaking properly configured?? can you provide props.conf values for log4j sourcetype?

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...