How to to troubleshoot why my logs are getting truncated?

[usha_nittala]

Hi,

I am facing an issue where logs are getting truncated, even though I have set TRUNCATE and MAX_EVENTS to very high values.

Sample log:

TRUNCATED LOG

2016-02-02 02:48:57,511 -0500 | [sacxsadcdsfvfdvdvsdcsc] | [dscsavcweqfsdcdscdscdscs] | [ascdsasa] | dsfwscdwscew | 745522813 | 123.456.41.101 | asfcdsafvgfvfdvcdscdsaI= | my_ANALYSIS | dscsacdscscdce | ewfwefewcw:3543_TRX | SESSION_SIGNIN | [&sdcscdsc_EVENT_TYPE=SESSION_SIGNIN&dcscs csa_SCORE=49&my_OUTCOME=A

ACTUAL LOG
2016-02-02 02:48:57,511 -0500 | [sacxsadcdsfvfdvdvsdcsc] | [dscsavcweqfsdcdscdscdscs] | [ascdsasa] | dsfwscdwscew | 745522813 | 123.456.41.101 | asfcdsafvgfvfdvcdscdsaI= | my_ANALYSIS | dscsacdscscdce | ewfwefewcw:3543_TRX | SESSION_SIGNIN | [&sdcscdsc_EVENT_TYPE=SESSION_SIGNIN&dcscs csa_SCORE=49&my_OUTCOME=ALLOW

[sloshburch]

Debugging tip: You should be able to take a sample of the logs from the server and use the add data feature of the UI to validate and play with the props.conf. That way you can validate settings before implementing.

[mtranchita]

I've seen a similar issue where the host the logs were being forwarded from had ntp out of sync (and the host was undersized). We had to adjust the monitor stanza in the inputs.conf and add a time_before_close value. Once we increased the value over the default we stopped truncating.

time_before_close =
* Modtime delta required before Splunk can close a file on EOF.
* Tells the system not to close files that have been updated in past
seconds.
* Defaults to 3.

[usha_nittala]

some logs are coming up without getting truncated while only few are getting truncated.
Not able to understand why is it working on some logs while it is not working for other.

[Jeremiah]

What is the props.conf configuration you have in place for this source/sourcetype?

[usha_nittala]

[prod1_app_log]
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE_DATE = true
TRUNCATE = 500000
MAX_EVENTS = 2048

[Jeremiah]

The first thing that jumps out is that you have set SHOULD_LINEMERGE = false. Since you set that, BREAK_ONLY_BEFORE_DATE has no effect. Are these events single-line or multi-line? Can you show us a sanitized snippet from one of your files?

[usha_nittala]

Here are the logs that are getting generated right now.

If you see the first snippet, It should be complete, but it got truncated at A where as it should have ALLOW like other log snippets.

2016-02-02 23:52:04,692 -0500 | [mylab] | [1675:dbab655a251:b9974475] | [9240B7000CCAB7B5C1710A3C09D1D43DB5B53B04 | REOC | 753693454 | 123.232.112.213 | q4/XXp0ao9MBGwKNS55aBNgX7lE= | mytest | 2675:dbab655a251:b9974475_MET | 2675:dbab655a251:b9974475_MET | MYSIGN | [&FORENSIC_EVENT_TYPE=MYSIGN&SCORE_TEST=432&RES_OUTCOME=A

2016-02-02 23:52:36,474 -0500 | [[B@4d8d49] | [96a5:dbab655a251:b9974475] | [7A25390C2F03048402648F5CCACFA0860F65ADB9 | REOC | 753695926 | 213.132.70.9 | 5R8voERCMTUR38J9/lwonUvZ3cE= | mytest | b6a5:dbab655a251:b9974475_MET | b6a5:dbab655a251:b9974475_MET | MYSIGN | [&FORENSIC_EVENT_TYPE=MYSIGN&SCORE_TEST=215&RES_OUTCOME=ALLOW]]

2016-02-02 23:52:36,461 -0500 | [[B@1c2da1e] | [86a5:dbab655a251:b9974475] | [580E890CCEB3AB6A6C10AC16F90C59090BA04DF6 | REOC | 753695923 | 48.104.47.203 | HsGmOmkZQcnUtjP2k9QnqlPwgvQ= | mytest | a6a5:dbab655a251:b9974475_MET | a6a5:dbab655a251:b9974475_MET | MYSIGN | [&FORENSIC_EVENT_TYPE=MYSIGN&SCORE_TEST=66&RES_OUTCOME=ALLOW]]

2016-02-02 23:52:36,446 -0500 | [[B@4d8d49] | [96a5:dbab655a251:b9974475] | [7A25390C2F03048402648F5CCACFA0860F65ADB9 | REOC | 753695926 | 123.32.116.19 | 5R8voERCMTUR38J9/lwonUvZ3cE= | USER_SIGNIN | b6a5:dbab655a251:b9974475_MET | b6a5:dbab655a251:b9974475_MET | MYSIGN | []]

[Jeremiah]

Where are the newline characters in the original log? From this it looks like

TEST=432&RES_OUTCOME=A 2016-02-02 23:52:36,474 -0500 | [[B@4d8d49] | 

Are all on a single line.

[usha_nittala]

The post was gobbled, my bad.
The new line starts with the date i.e.

2016-02-02 23:52:36,474 -0500 , I posted logs again.

[sloshburch]

Silly question, but can you validate that the exact same log entry on the source server is not truncating the event? When I've seen this issue in the past it turned out that the truncating was occurring before Splunk and it was actually a problem with the application writing the logs.