Splunk data are cut of randomly

[jadengoho]

I am having problem with UF data ingestion.
There are 36 servers (18 server are prod and 18 are test-prod)
I have deployment server who deploy configuration files to the 36 servers. But the logs i get are incomplete from different server.
"incomplete" means most server send complete logs but other servers logs are being cut.

Example logs:
(Complete)
+ server-service applicationservice-engine1:status
server-service: applicationservice-engine1: OK
+ server-service applicationservice-engine2:status
server-service: applicationservice-engine2: OK

(incomplete sample 1)
+ server-service applicationservice-engine2:status
server-service: applicationservice-engine2: OK

(incomplete sample 2)
server-service: applicationservice-engine1: OK
+ server-service applicationservice-engine2:status
server-service: applicationservice-engine2: OK

my inputs.conf :
[monitor:///var/log/scripts/service-status.log]
disabled = 0
index = operations
sourcetype = service:status

and there are NO Props or Transforms applied.
What may be the reason of this occurrence?

[woodcock · Accepted solution]

Rather than explain the details (that is @DalJeanis job), here are some configurations to be sure to set:

DATETIME_CONFIG = CURRENT
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)
useACK = false
time_before_close = <experiment>

[jadengoho]

Thank you very much this really help me a lot.

@DalJeanis can you tell me why does this happens?

[woodcock]

So did it work for you?

[jadengoho]

Yes it works, thank you very much

Props.conf
DATETIME_CONFIG = CURRENT
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)

Inputs.conf
useACK = false
time_before_close = 300

Still i want to know why does behavior happens ?

[woodcock]

The useACK is very buggy and causes as many problems as it might perhaps solve so only consider using it when you absolutely must and be prepared for many duplicated events. The on that did it for you was the time_before_close.