topic Re: Splunk data are cut of randomly in Getting Data In

Splunk data are cut of randomly

jadengoho — Fri, 06 Sep 2019 01:34:01 GMT

I am having problem with UF data ingestion.
There are 36 servers (18 server are prod and 18 are test-prod)
I have deployment server who deploy configuration files to the 36 servers. But the logs i get are incomplete from different server.
"incomplete" means most server send complete logs but other servers logs are being cut.

Example logs:
(Complete)
+ server-service applicationservice-engine1:status
server-service: applicationservice-engine1: OK
+ server-service applicationservice-engine2:status
server-service: applicationservice-engine2: OK

(incomplete sample 1)
+ server-service applicationservice-engine2:status
server-service: applicationservice-engine2: OK

(incomplete sample 2)
server-service: applicationservice-engine1: OK
+ server-service applicationservice-engine2:status
server-service: applicationservice-engine2: OK

my inputs.conf :
[monitor:///var/log/scripts/service-status.log]
disabled = 0
index = operations
sourcetype = service:status

and there are NO Props or Transforms applied.
What may be the reason of this occurrence?

Re: Splunk data are cut of randomly

woodcock — Fri, 06 Sep 2019 21:04:34 GMT

Rather than explain the details (that is @DalJeanis job), here are some configurations to be sure to set:

DATETIME_CONFIG = CURRENT
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)
useACK = false
time_before_close = <experiment>

Re: Splunk data are cut of randomly

jadengoho — Sat, 07 Sep 2019 04:45:46 GMT

Thank you very much this really help me a lot.

@DalJeanis can you tell me why does this happens?

Re: Splunk data are cut of randomly

woodcock — Mon, 09 Sep 2019 13:59:39 GMT

So did it work for you?

Re: Splunk data are cut of randomly

jadengoho — Wed, 30 Sep 2020 02:06:05 GMT

Yes it works, thank you very much

Props.conf
DATETIME_CONFIG = CURRENT
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)

Inputs.conf
useACK = false
time_before_close = 300

Still i want to know why does behavior happens ?

Re: Splunk data are cut of randomly

woodcock — Mon, 09 Sep 2019 22:28:02 GMT

The useACK is very buggy and causes as many problems as it might perhaps solve so only consider using it when you absolutely must and be prepared for many duplicated events. The on that did it for you was the time_before_close.