Splunk universal forwarder active status suddenly changed to disappear and active again?

[aldi_mukti]

Hi all,

we are having a little trouble finding the cause of the active universal forwarder status changing to disappear and become active again. We have also checked from the network side and there are no problems related to the network. this doesn't always happen but is very annoying operationally. architecturally we have one search head, one indexer, 1 heavy forwarder DC1, and 1 heavy forwarder DC2. for universal forwarders we have thirty five DC1 to heavy forwarder DC1 and thirty five universal forwarders DC2 to heavy forwarder DC2. We really ask for help regarding this problem and thank you for your attention

Thanks. 

[aldi_mukti]

Yes, that's right, we saw from the Forwarder Management screen of the Management Console, from the forwarder management we correlated it as an alert to email
Do we need to add thruput settings (default 256) ? how to prevent MC from displaying wrong information.

[PickleRick]

The information as such is not "wrong". It's just how the forwarder monitoring works. If you verify that you hit the default thruput limit of 256kBps, you can raise it in limits.conf.

[aldi_mukti]

If we raise the thruput does the incident not happen again. is there a way to ensure that the thruput should indeed be increased?
Because we have to make sure before increasing thruput settings.

[PickleRick]

As with any technical solution - there is no 100% proof solution that will tell you that always everything will be ok. Anything can break, it's just that you lower the probability of such situation. If you raise the thruput limit over an expected data intake, you should be safe from such hiccups but in case of sudden spikes of data ingestion rate, you can get the same problem again. If you remove the limit altogether you might - in extreme cases - clog pipelines in another processing point so it's again not 100% sure.

Sorry, but that's just how life is - nothing is 100% certain.

And if you want to tweak the limit, be cautious to the case of the letters - splunk is case-sensitive about the parameter names.

[aldi_mukti]

Well noted.. thanks a lot for your advice.

Thank you and have a great day

Happy Splunking...

[PickleRick · Accepted solution]

And where is this status shown? Are you talking about the Forwarder Management screen of Management Console? Forwarders are not directly monitored but their status and performance is pulled from the logs they send. So if there are problems with forwarding the internal logs of the forwarder, the MC might show incorrect information.

Typically problems with internal logs can arise from low thruput settings on the UF. A bit counterintuitively, internal logs are just another input and are not prioritized in any way so you might end up starving yourself from them. If your "main" forwarder activity is spiky, you might end up with a periodically "disappearing" forwarder.