Scripted input is done many times regardless of interval setting.

[yutaka1005]

My environment :
Splunk Indexer : 7.2.3 on Linux7
Splunk Deployment Server : 7.2.3 on Linux7
Universal Forwarder : 7.2.3 on Linux7

I configured that Deployment Server deploys below inputs.conf to universal forwarder(UF).

[script//./bin/sample.sh]
interval = 14 12 * * *
index = sample_index
source = sample.sh
sourcetype = sample
disabled = 0

Everyday, the UF kicks this script that runs "cat" to file(* about 7MB), and forwards result to Indexer.

However, sometimes the UF ignores the setting of "interval" and tries to do this script input many times (* dozens times etc.), and is causing duplicates on the Indexer.

Why does this happen?

If anyone knows a similar event, please tell me.

[yutaka1005 · Accepted solution]

After all, scripted input was not done many times.

Because parsing and aggregation of data takes time and Indexer can not return ACK, the result of scripted input and the internal log to the effect that scripted input were executed have been repeated many times by the useACK retransmission function.

[burwell]

Some ideas/questions. We use scripted input and I haven't heard of such issues.

1) how often is the script running exactly? If you have the above exact cron specification, exactly what hour/minutes is it running? [maybe you can cat the date to the file]

2) could it possibly be the UF is restarting?

3) could the date on your UF be getting reset by NTP or something?

4) Do you have this on more than one UF and is happening on more than one?

[yutaka1005]

Me too...

1)
This script takes about 5 minutes until end.
Normally, it starts at 12:14 every day, and data is captured about 5 minutes later, but when this issue suddenly starts, it seems that this script retries many times(* sometimes for days) immediately after completion of script execution.

Then, the following execution completion message is output many times.
* Although "reschedule_ms" shows 24 hours ...

INFO  ExecProcessor - setting reschedule_ms=86399993, for command=/opt/splunkforwarder/etc/apps/sample_app/bin/sample.sh

2)
UF restarts when deployment server deploys new apps.
But this issue suddenly begins regardless of restarting. And rather by restarting, I can fix this issue temporarily.

3)
Although the system time fine modification is done by NTP, I think that
the more NTP need to change date, the time will not be shifted... probably.

4)
There are systems A and B, and in "system A", Indexer have received this from a UF. In "system B" Indexer has received this from two UFs.

But strangely, this issue only happens in "system A".