Why is UF not forwarding to indexers?

[NAVEEN_CTS]

I have a very peculiar situation where my UF is sending data only to Lab Indexers where as it not been forwarded to Prod Indexers.

Here are the details:

1. UF monitors a syslog file in the same linux box .
2. Inputs.conf is part of an app in the etc/apps directory.
3. In inputs TCP routing is configured to send it to both lab and prod indexer groups as below

[monitor:///xxx/xxx/xxx/xxx/.../*.log]
_TCP_ROUTING=lab-indexers,prod-indexers
host_segment=5
sourcetype=syslog
blacklist = .log.1
disabled = false
index = X

1. outputs in etc/local which is common for all apps as below

[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
forwardedindex.filter.disable = false

defaultGroup = lab-indexers, prod-indexers

[tcpout:lab-indexers]
server = xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997

[tcpout:prod-indexers]
server = xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997

Note : there many other inputs app which is routing data to both lab and prod indexers. so there is no problem with network related issues. Index is also created in prod

So help me with troubleshooting steps to narrow down this issue

Thanks in advance

[NAVEEN_CTS · Accepted solution]

Issue is fixed now. Problem is with the TA - Splunk Add-on for VMware.

internal index gave the hint with the following error:

ERROR AggregatorMiningProcessor - Uncaught Exception in Aggregator, skipping an event:
Can't open DateParser XML configuration file
"/opt/splunk/etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml": No such file or
directory - data_source="/data/log_files/syslog/.log", data_host="",
data_sourcetype="vmw-syslog"

Check this below URL for more details:
https://docs.splunk.com/Documentation/AddOns/released/VMW/Troubleshoot

[nickhills]

There is nothing obviously wrong with you config (although please use the code tool 101010 to post config snippets because its possible something has not been shown correctly in the text view)

As suggested, btool is a good test to make sure there is nothing wrong in your configuration.
You noted that:

there many other inputs app which is routing data to both lab and prod indexers
Is that true for this host? Is it possible that firewall rules/ACLs are preventing this host from communicating with the prod-indexers?

[13tsavage]

Have you run a btool check and see if that outputs any stdout messages?

Run $SPLUNK_HOME/bin/splunk btool check on your UF and it should show any errors in your outputs.conf and inputs.conf.

If there are no errors reported, then we know it is not the UF's fault that it cannot send to the prod indexers. Then you need to look at those prod indexers and check if their inputs.conf is configured to receive the correct input parameters.