Getting Data In

Why is UF not forwarding to indexers?

NAVEEN_CTS
Path Finder

I have a very peculiar situation where my UF is sending data only to Lab Indexers where as it not been forwarded to Prod Indexers.

Here are the details:

  1. UF monitors a syslog file in the same linux box .
  2. Inputs.conf is part of an app in the etc/apps directory.
  3. In inputs TCP routing is configured to send it to both lab and prod indexer groups as below

[monitor:///xxx/xxx/xxx/xxx/.../*.log]
_TCP_ROUTING=lab-indexers,prod-indexers
host_segment=5
sourcetype=syslog
blacklist = .log.1
disabled = false
index = X

  1. outputs in etc/local which is common for all apps as below

[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
forwardedindex.filter.disable = false

defaultGroup = lab-indexers, prod-indexers

[tcpout:lab-indexers]
server = xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997

[tcpout:prod-indexers]
server = xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997

Note : there many other inputs app which is routing data to both lab and prod indexers. so there is no problem with network related issues. Index is also created in prod

So help me with troubleshooting steps to narrow down this issue

Thanks in advance

0 Karma
1 Solution

NAVEEN_CTS
Path Finder

Issue is fixed now. Problem is with the TA - Splunk Add-on for VMware.

internal index gave the hint with the following error:

ERROR AggregatorMiningProcessor - Uncaught Exception in Aggregator, skipping an event:
Can't open DateParser XML configuration file
"/opt/splunk/etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml": No such file or
directory - data_source="/data/log_files/syslog/.log", data_host="",
data_sourcetype="vmw-syslog"

Check this below URL for more details:
https://docs.splunk.com/Documentation/AddOns/released/VMW/Troubleshoot

View solution in original post

0 Karma

NAVEEN_CTS
Path Finder

Issue is fixed now. Problem is with the TA - Splunk Add-on for VMware.

internal index gave the hint with the following error:

ERROR AggregatorMiningProcessor - Uncaught Exception in Aggregator, skipping an event:
Can't open DateParser XML configuration file
"/opt/splunk/etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml": No such file or
directory - data_source="/data/log_files/syslog/.log", data_host="",
data_sourcetype="vmw-syslog"

Check this below URL for more details:
https://docs.splunk.com/Documentation/AddOns/released/VMW/Troubleshoot

0 Karma

nickhills
Ultra Champion

There is nothing obviously wrong with you config (although please use the code tool 101010 to post config snippets because its possible something has not been shown correctly in the text view)

As suggested, btool is a good test to make sure there is nothing wrong in your configuration.
You noted that:

there many other inputs app which is routing data to both lab and prod indexers
Is that true for this host? Is it possible that firewall rules/ACLs are preventing this host from communicating with the prod-indexers?

If my comment helps, please give it a thumbs up!
0 Karma

13tsavage
Communicator

Have you run a btool check and see if that outputs any stdout messages?

Run $SPLUNK_HOME/bin/splunk btool check on your UF and it should show any errors in your outputs.conf and inputs.conf.

If there are no errors reported, then we know it is not the UF's fault that it cannot send to the prod indexers. Then you need to look at those prod indexers and check if their inputs.conf is configured to receive the correct input parameters.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...