Can't see data coming in from UF to Indexer

[MAvasthi]

Hi Guys,

I am very new to Splunk Cloud and how things work here. Our current setup is:

1. UF(Linux) -> Heavy Forwarder(On Prem) -> Indexer/Search Head(Splunk Cloud)

2. Created a new index quifapp on Splunk Cloud.

2. UF is already connected to HF (just dummy connection and verified that its sending _internal logs to Splunk Cloud) as can be seen from the logs:

02-20-2024 11:22:11.394 +1100 INFO AutoLoadBalancedConnectionStrategy [566068 TcpOutEloop] - Found currently active indexer. Connected to idx=192.168.23.32:9997:0, reuse=1

3. New request is to forward logs from location /app/quif/quif.log to Splunk Cloud.

4. I have put the required config under below location /opt/splunkforwarder/etc/apps/quif/local and it has two files:

#cat inputs.conf

[monitor:///app/quif/quif.log*]

sourcetype=quif_requests

disabled=0

index=quifapp

# cat props.conf

[quif_requests]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true
EXTRACT-AgentId = ^\w+:\s+(?P<AgentId>\w+)

####

4. I restarted SplunkForwarder but can't see any logs coming in the Cloud.

Is there any additional config that's required at any level. How can I troubleshoot?

[MAvasthi]

Also I can see below logs in metrics.log:

/opt/splunkforwarder/var/log/splunk# grep -Ri blocked metrics.log*
metrics.log:02-20-2024 02:18:21.653 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=102, largest_size=105, smallest_size=35
metrics.log:02-20-2024 02:27:30.654 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=100, largest_size=107, smallest_size=36
metrics.log:02-20-2024 02:28:31.653 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=102, largest_size=107, smallest_size=40
metrics.log:02-20-2024 03:01:03.654 +1100 INFO Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=512, current_size_kb=511, current_size=120, largest_size=125, smallest_size=41
metrics.log:02-20-2024 03:13:15.656 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=102, largest_size=105, smallest_size=32
metrics.log:02-20-2024 03:21:23.654 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=102, largest_size=107, smallest_size=36
metrics.log:02-20-2024 03:27:29.653 +1100 INFO Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=512, current_size_kb=511, current_size=121, largest_size=123, smallest_size=38
metrics.log:02-20-2024 03:31:33.653 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=100, largest_size=105, smallest_size=35
metrics.log:02-20-2024 03:57:59.653 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=100, largest_size=105, smallest_size=35
metrics.log.1:02-19-2024 21:45:53.652 +1100 INFO Metrics - group=knowledgebundle_replication, name=blocked_search_metrics, app=none, user=none, elapsed_ms=18446744073709551615
metrics.log.1:02-19-2024 22:07:14.652 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=100, largest_size=106, smallest_size=33
metrics.log.1:02-19-2024 22:27:34.653 +1100 INFO Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=512, current_size_kb=511, current_size=100, largest_size=112, smallest_size=35
metrics.log.1:02-19-2024 22:56:02.653 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=102, largest_size=111, smallest_size=32
metrics.log.1:02-19-2024 22:57:03.653 +1100 INFO Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=512, current_size_kb=511, current_size=120, largest_size=125, smallest_size=42
metrics.log.1:02-19-2024 23:18:24.654 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=101, largest_size=106, smallest_size=33
metrics.log.1:02-20-2024 00:08:13.652 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=100, largest_size=111, smallest_size=37
metrics.log.1:02-20-2024 00:21:26.652 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=102, largest_size=111, smallest_size=37
metrics.log.1:02-20-2024 00:44:49.653 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=100, largest_size=105, smallest_size=39
metrics.log.1:02-20-2024 00:49:54.655 +1100 INFO Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=512, current_size_kb=511, current_size=119, largest_size=129, smallest_size=40
metrics.log.1:02-20-2024 01:25:29.654 +1100 INFO Metrics - group=queue, name=parsingqueue, blocked=true, max_size_kb=512, current_size_kb=511, current_size=121, largest_size=131, smallest_size=37
metrics.log.1:02-20-2024 01:27:31.654 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=102, largest_size=111, smallest_size=29
metrics.log.1:02-20-2024 01:33:37.653 +1100 INFO Metrics - group=queue, name=aeq, blocked=true, max_size_kb=500, current_size_kb=499, current_size=101, largest_size=111, smallest_size=33