Getting Data In

Scripted input is done many times regardless of interval setting.

yutaka1005
Builder

My environment :
Splunk Indexer : 7.2.3 on Linux7
Splunk Deployment Server : 7.2.3 on Linux7
Universal Forwarder : 7.2.3 on Linux7

I configured that Deployment Server deploys below inputs.conf to universal forwarder(UF).

[script//./bin/sample.sh]
interval = 14 12 * * *
index = sample_index
source = sample.sh
sourcetype = sample
disabled = 0

Everyday, the UF kicks this script that runs "cat" to file(* about 7MB), and forwards result to Indexer.

However, sometimes the UF ignores the setting of "interval" and tries to do this script input many times (* dozens times etc.), and is causing duplicates on the Indexer.

Why does this happen?

If anyone knows a similar event, please tell me.

0 Karma
1 Solution

yutaka1005
Builder

After all, scripted input was not done many times.

Because parsing and aggregation of data takes time and Indexer can not return ACK, the result of scripted input and the internal log to the effect that scripted input were executed have been repeated many times by the useACK retransmission function.

View solution in original post

0 Karma

yutaka1005
Builder

After all, scripted input was not done many times.

Because parsing and aggregation of data takes time and Indexer can not return ACK, the result of scripted input and the internal log to the effect that scripted input were executed have been repeated many times by the useACK retransmission function.

0 Karma

burwell
SplunkTrust
SplunkTrust

Some ideas/questions. We use scripted input and I haven't heard of such issues.

1) how often is the script running exactly? If you have the above exact cron specification, exactly what hour/minutes is it running? [maybe you can cat the date to the file]

2) could it possibly be the UF is restarting?

3) could the date on your UF be getting reset by NTP or something?

4) Do you have this on more than one UF and is happening on more than one?

0 Karma

yutaka1005
Builder

Me too...

1)
This script takes about 5 minutes until end.
Normally, it starts at 12:14 every day, and data is captured about 5 minutes later, but when this issue suddenly starts, it seems that this script retries many times(* sometimes for days) immediately after completion of script execution.

Then, the following execution completion message is output many times.
* Although "reschedule_ms" shows 24 hours ...

INFO  ExecProcessor - setting reschedule_ms=86399993, for command=/opt/splunkforwarder/etc/apps/sample_app/bin/sample.sh

2)
UF restarts when deployment server deploys new apps.
But this issue suddenly begins regardless of restarting. And rather by restarting, I can fix this issue temporarily.

3)
Although the system time fine modification is done by NTP, I think that
the more NTP need to change date, the time will not be shifted... probably.

4)
There are systems A and B, and in "system A", Indexer have received this from a UF. In "system B" Indexer has received this from two UFs.

But strangely, this issue only happens in "system A".

0 Karma
Get Updates on the Splunk Community!

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...

Splunk With AppDynamics - Meet the New IT (And Engineering) Couple

Wednesday, November 20, 2024  |  10AM PT / 1PM ET Register Now Join us in this session to learn all about ...