Getting Data In

How to to troubleshoot why my logs are getting truncated?

usha_nittala
New Member

Hi,

I am facing an issue where logs are getting truncated, even though I have set TRUNCATE and MAX_EVENTS to very high values.

Sample log:

TRUNCATED LOG

2016-02-02 02:48:57,511 -0500 | [sacxsadcdsfvfdvdvsdcsc] | [dscsavcweqfsdcdscdscdscs] | [ascdsasa] | dsfwscdwscew | 745522813 | 123.456.41.101 | asfcdsafvgfvfdvcdscdsaI= | my_ANALYSIS | dscsacdscscdce | ewfwefewcw:3543_TRX | SESSION_SIGNIN | [&sdcscdsc_EVENT_TYPE=SESSION_SIGNIN&dcscs csa_SCORE=49&my_OUTCOME=A

ACTUAL LOG
2016-02-02 02:48:57,511 -0500 | [sacxsadcdsfvfdvdvsdcsc] | [dscsavcweqfsdcdscdscdscs] | [ascdsasa] | dsfwscdwscew | 745522813 | 123.456.41.101 | asfcdsafvgfvfdvcdscdsaI= | my_ANALYSIS | dscsacdscscdce | ewfwefewcw:3543_TRX | SESSION_SIGNIN | [&sdcscdsc_EVENT_TYPE=SESSION_SIGNIN&dcscs csa_SCORE=49&my_OUTCOME=ALLOW

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Debugging tip: You should be able to take a sample of the logs from the server and use the add data feature of the UI to validate and play with the props.conf. That way you can validate settings before implementing.

0 Karma

mtranchita
Communicator

I've seen a similar issue where the host the logs were being forwarded from had ntp out of sync (and the host was undersized). We had to adjust the monitor stanza in the inputs.conf and add a time_before_close value. Once we increased the value over the default we stopped truncating.

time_before_close =
* Modtime delta required before Splunk can close a file on EOF.
* Tells the system not to close files that have been updated in past
seconds.
* Defaults to 3.

0 Karma

usha_nittala
New Member

some logs are coming up without getting truncated while only few are getting truncated.
Not able to understand why is it working on some logs while it is not working for other.

0 Karma

Jeremiah
Motivator

What is the props.conf configuration you have in place for this source/sourcetype?

0 Karma

usha_nittala
New Member

[prod1_app_log]
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
TRUNCATE = 500000
MAX_EVENTS = 2048

0 Karma

Jeremiah
Motivator

The first thing that jumps out is that you have set SHOULD_LINEMERGE = false. Since you set that, BREAK_ONLY_BEFORE_DATE has no effect. Are these events single-line or multi-line? Can you show us a sanitized snippet from one of your files?

0 Karma

usha_nittala
New Member

Here are the logs that are getting generated right now.

If you see the first snippet, It should be complete, but it got truncated at A where as it should have ALLOW like other log snippets.

2016-02-02 23:52:04,692 -0500 | [mylab] | [1675:dbab655a251:b9974475] | [9240B7000CCAB7B5C1710A3C09D1D43DB5B53B04 | REOC | 753693454 | 123.232.112.213 | q4/XXp0ao9MBGwKNS55aBNgX7lE= | mytest | 2675:dbab655a251:b9974475_MET | 2675:dbab655a251:b9974475_MET | MYSIGN | [&FORENSIC_EVENT_TYPE=MYSIGN&SCORE_TEST=432&RES_OUTCOME=A

2016-02-02 23:52:36,474 -0500 | [[B@4d8d49] | [96a5:dbab655a251:b9974475] | [7A25390C2F03048402648F5CCACFA0860F65ADB9 | REOC | 753695926 | 213.132.70.9 | 5R8voERCMTUR38J9/lwonUvZ3cE= | mytest | b6a5:dbab655a251:b9974475_MET | b6a5:dbab655a251:b9974475_MET | MYSIGN | [&FORENSIC_EVENT_TYPE=MYSIGN&SCORE_TEST=215&RES_OUTCOME=ALLOW]]

2016-02-02 23:52:36,461 -0500 | [[B@1c2da1e] | [86a5:dbab655a251:b9974475] | [580E890CCEB3AB6A6C10AC16F90C59090BA04DF6 | REOC | 753695923 | 48.104.47.203 | HsGmOmkZQcnUtjP2k9QnqlPwgvQ= | mytest | a6a5:dbab655a251:b9974475_MET | a6a5:dbab655a251:b9974475_MET | MYSIGN | [&FORENSIC_EVENT_TYPE=MYSIGN&SCORE_TEST=66&RES_OUTCOME=ALLOW]]

2016-02-02 23:52:36,446 -0500 | [[B@4d8d49] | [96a5:dbab655a251:b9974475] | [7A25390C2F03048402648F5CCACFA0860F65ADB9 | REOC | 753695926 | 123.32.116.19 | 5R8voERCMTUR38J9/lwonUvZ3cE= | USER_SIGNIN | b6a5:dbab655a251:b9974475_MET | b6a5:dbab655a251:b9974475_MET | MYSIGN | []]
0 Karma

Jeremiah
Motivator

Where are the newline characters in the original log? From this it looks like

TEST=432&RES_OUTCOME=A 2016-02-02 23:52:36,474 -0500 | [[B@4d8d49] | 

Are all on a single line.

0 Karma

usha_nittala
New Member

The post was gobbled, my bad.
The new line starts with the date i.e.

2016-02-02 23:52:36,474 -0500 , I posted logs again.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Silly question, but can you validate that the exact same log entry on the source server is not truncating the event? When I've seen this issue in the past it turned out that the truncating was occurring before Splunk and it was actually a problem with the application writing the logs.

0 Karma