topic Re: Script set up via universal forwarder stops and starts working on its own again in Splunk Enterprise

Script set up via universal forwarder stops and starts working on its own again

_pravin — Tue, 05 Apr 2022 09:42:41 GMT

Hi Community,

I am having a weird issue with Splunk Enterprise. I had set up a universal internal forwarder to execute a script that gives me the list of all different processes within the Linux environment.

All of a sudden the script stopped producing results from 12 am and the panel didn't work. But again it starts working after 3 days by itself. This happened in both the test and production setup. Is there something that should be taken care of when using scripts in Universal forwarder or is there some reason for this unusual behaviour?

Regards,

Pravin

Re: Script set up via universal forwarder stops and starts working on its own again

VatsalJagani — Tue, 05 Apr 2022 13:54:24 GMT

Please provide more information on how the script is being managed. (inputs.conf, script logic, etc)

Also, please make sure your forwarder was running during the time when you did not see data. You can run the below search for that.

| tstats count where index=_internal host="<your forwarder host>" by _indextime | eval _time=_indextime | timechart span=1h sum(count)

(You should see gap in this timechart if forwarder was down.)

Re: Script set up via universal forwarder stops and starts working on its own again

_pravin — Thu, 07 Apr 2022 08:52:02 GMT

Hi @VatsalJagani ,

The script is being managed by input.conf from the internal forwarder. There are a few more scripts and files being managed by the same forwarder which are working as usual but only this particular script doesn't work.

Also, the command doesn't produce any results and shows 0 results found.

| tstats count where index=_internal host="<your forwarder host>" by _indextime
| eval _time=_indextime
| timechart span=1h sum(count)

Thanks,

Pravin

Re: Script set up via universal forwarder stops and starts working on its own again

VatsalJagani — Thu, 07 Apr 2022 13:05:00 GMT

Please try this search query to check if server was running all the time or not.

index=_internal host="<your forwarder host>" | timechart span=15m count

If the forwarder was up all the time
- If the server stopped sending all kinds of data during that time as you mentioned
  - then there could be a network bandwidth issue.
  - Also, note that UF has a bandwidth limit of 256Kbps by default
  - If your server is producing a lot of data then your network bandwidth could create that problem.
- If the server stopped sending particular input data
  - then look at the logs related to that input, and see if you see any errors/warnings.
- Check for splunkd logs to see if you see any warnings/errors around that time. It should give answer to your question.
If above query give you gap in the timechart then that means your forwarder was down during that time.