topic Applying Quarantine .... Removing quarantine in Deployment Architecture

Applying Quarantine .... Removing quarantine

abhayneilam — Mon, 28 Sep 2020 18:46:32 GMT

Hi,
I have a UF in which Splunk_TA_nix application is installed and it was working fine but suddenly it started giving these errors in splunkd.log which causes discountinuty of sending the data to the Indexers.

02-03-2015 12:05:39.119 +0100 INFO  TailingProcessor - Could not send data to output queue (parsingQueue), retrying...
02-03-2015 12:05:50.632 +0100 WARN  TcpOutputProc - Cooked connection to ip=XXXXXXXXXXX:9997 timed out
02-03-2015 12:05:56.872 +0100 INFO  ExecProcessor - Ran script: /opt/SP/apps/splunkforwarder/Splunkforwarder-5.0/etc/apps/Splunk_TA_nix/bin/ps.sh, took 74.28 milliseconds to run, 11510 bytes read
02-03-2015 12:05:57.594 +0100 INFO  ExecProcessor - Ran script: /opt/SP/apps/splunkforwarder/Splunkforwarder-5.0/etc/apps/Splunk_TA_nix/bin/cpu.sh, took 1043.8 milliseconds to run, 1003 bytes read
02-03-2015 12:06:00.636 +0100 WARN  TcpOutputFd - Connect to XXXXXX:9997 failed. Connection refused
02-03-2015 12:06:00.636 +0100 ERROR TcpOutputFd - Connection to host=XXXXXXXXXXX:9997 failed
02-03-2015 12:06:00.636 +0100 WARN  TcpOutputProc - Applying quarantine to ip=XXXXXXXX port=9997 _numberOfFailures=2

I have an outputs.conf which is as follows :

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = AABBCCDD:9997
useACK=true
sendCookedData = true


[tcpout-server://AABBCCDD:9997]

Note : AABBCCDD is the load balancer server ip

Data is appearing in the dashboard but not in the continous manner, it is missing for last 3 hours , sometimes it is missing for last 30 mins. Please HELP !!

Cheers,

Re: Applying Quarantine .... Removing quarantine

MuS — Tue, 03 Feb 2015 11:51:37 GMT

Hi abhayneilam,

usually there was a change somewhere, If something suddenly stops working.
Check this load-balancer or the Server OS, because Connection refused means that the target machine actively rejected the connection.
Check any fire wall in between, also consider routing settings.
Don't forget to check if splunkd is running on the indexers....

May I ask, why are you not using the universal forwarders internal load-balancing method?

cheers, MuS

Re: Applying Quarantine .... Removing quarantine

abhayneilam — Tue, 03 Feb 2015 12:05:33 GMT

We have this configured from the past 1 year, so all fine , no issues until yesterday, suddenly I dont know, Quantine issue appears :

02-03-2015 09:47:19.246 +0100 INFO TcpOutputProc - Removing quarantine from idx=XXXXX:9997
02-03-2015 09:47:39.247 +0100 WARN TcpOutputProc - Cooked connection to ip=XXXXX:9997 timed out
02-03-2015 09:47:39.248 +0100 WARN TcpOutputProc - Cooked connection to ip=XXXXX:9997 timed out
02-03-2015 09:47:39.248 +0100 WARN TcpOutputProc - Cooked connection to ip=XXXXX:9997 timed out
02-03-2015 09:47:39.248 +0100 WARN TcpOutputProc - Applying quarantine to ip=XXXXX port=9997 _numberOfFailures=2

Re: Applying Quarantine .... Removing quarantine

MuS — Tue, 03 Feb 2015 12:10:38 GMT

so what did change yesterday? You're obviously no longer able to connect to port 9997 on IP XXXXX

Re: Applying Quarantine .... Removing quarantine

abhayneilam — Tue, 03 Feb 2015 13:23:08 GMT

Nothing was changed !! that's why it is strange , suddenly it happened.

Re: Applying Quarantine .... Removing quarantine

mmensch — Wed, 04 May 2016 16:44:48 GMT

Did you ever resolve this issue? If so, how?