topic Re: SplunkCloud - Heavy Forwarder Communication in Getting Data In

SplunkCloud - Heavy Forwarder Communication: Why is my heavy forwarder is now skipping data?

herguzav — Tue, 15 Aug 2023 19:45:15 GMT

Hi friends.

I've followed de path to use UniversarForwarder app from my splunk cloud enviromen. But i have the next message:

The TCP output processor has paused the data flow. Forwarding to host_dest=inputs1.XXXX.splunkcloud.com inside output group splunkcloud_ from host_src=YYYYYY has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. Learn more.

I've tested the communications to splunk cloud

splunkcloud.com:9997

splunkcloud.com:8000

splunkcloud.com:8089

And all are OK.

My heavy forwarder is now skipping data. Is there something else I clould check out?

Re: SplunkCloud - Heavy Forwarder Communication

burwell — Mon, 14 Aug 2023 21:08:42 GMT

Hi. Isn't this saying indexer inputs1.XXXX.splunkcloud.com has full queues?

Can you look at the queues there? Messages?

Re: SplunkCloud - Heavy Forwarder Communication

herguzav — Tue, 15 Aug 2023 00:56:47 GMT

Hi!!

All splunkcloud queues are ok. Even my other heavy forwarder is working fine.

Do you have another test to do?

Re: SplunkCloud - Heavy Forwarder Communication

isoutamo — Tue, 15 Aug 2023 09:41:58 GMT

Usually it's like @burwell said. This error/warning means that for some reason there are some queues full. Earlier You could try this

index=_internal host=*.<your stack name>.splunkcloud.com source=*metrics.log sourcetype=splunkd TERM(group=queue) (TERM(name=parsingQueue) OR TERM(name=indexqueue) OR TERM(name=tcpin_queue) OR TERM(name=aggqueue)) | eval is_blocked=if(blocked=="true",1,0), host_queue=host." - ".name | stats sparkline sum(is_blocked) as blocked,count by host_queue | eval blocked_ratio=round(blocked/count*100,2) | where blocked_ratio > 0 | sort 50 -blocked_ratio | eval requires_attention=case(blocked_ratio>50.0,"fix highly recommended!",blocked_ratio>40.0,"you better check..",blocked_ratio>20.0,"usually no need to worry but keep an eye on it",1=1,"not unusual")

to check what is status of those input forwarders. But not I cannot see that information on SC, probably it's forwarder to somewhere else?

I suppose that you will create a ticket to SC support and ask situation about those input forwarders.

r. Ismo

Re: SplunkCloud - Heavy Forwarder Communication

herguzav — Wed, 16 Aug 2023 00:17:56 GMT

Hi .

I've run your search and no results were displayed, I've retired the condition and all results said

"not unusual"

The problem persist. 😞

Regards

Re: SplunkCloud - Heavy Forwarder Communication

isoutamo — Fri, 18 Aug 2023 08:01:35 GMT

As I said, I cannot see that information (inputs1-15) on SC side anymore. Earlier this was stored to customer cloud stack, but now I expecting that this information is currently forwarder to Splunk Clouds' admin stack or somewhere else where Customers haven't access?

I think that you haven't any other option than create a support ticket to Splunk and ask that they solve this issue.