Getting Data In

Why is UF not forwarding to indexers?

NAVEEN_CTS
Path Finder

I have a very peculiar situation where my UF is sending data only to Lab Indexers where as it not been forwarded to Prod Indexers.

Here are the details:

  1. UF monitors a syslog file in the same linux box .
  2. Inputs.conf is part of an app in the etc/apps directory.
  3. In inputs TCP routing is configured to send it to both lab and prod indexer groups as below

[monitor:///xxx/xxx/xxx/xxx/.../*.log]
_TCP_ROUTING=lab-indexers,prod-indexers
host_segment=5
sourcetype=syslog
blacklist = .log.1
disabled = false
index = X

  1. outputs in etc/local which is common for all apps as below

[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
forwardedindex.filter.disable = false

defaultGroup = lab-indexers, prod-indexers

[tcpout:lab-indexers]
server = xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997

[tcpout:prod-indexers]
server = xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997, xx.xxx.x.xxx:9997

Note : there many other inputs app which is routing data to both lab and prod indexers. so there is no problem with network related issues. Index is also created in prod

So help me with troubleshooting steps to narrow down this issue

Thanks in advance

0 Karma
1 Solution

NAVEEN_CTS
Path Finder

Issue is fixed now. Problem is with the TA - Splunk Add-on for VMware.

internal index gave the hint with the following error:

ERROR AggregatorMiningProcessor - Uncaught Exception in Aggregator, skipping an event:
Can't open DateParser XML configuration file
"/opt/splunk/etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml": No such file or
directory - data_source="/data/log_files/syslog/.log", data_host="",
data_sourcetype="vmw-syslog"

Check this below URL for more details:
https://docs.splunk.com/Documentation/AddOns/released/VMW/Troubleshoot

View solution in original post

0 Karma

NAVEEN_CTS
Path Finder

Issue is fixed now. Problem is with the TA - Splunk Add-on for VMware.

internal index gave the hint with the following error:

ERROR AggregatorMiningProcessor - Uncaught Exception in Aggregator, skipping an event:
Can't open DateParser XML configuration file
"/opt/splunk/etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml": No such file or
directory - data_source="/data/log_files/syslog/.log", data_host="",
data_sourcetype="vmw-syslog"

Check this below URL for more details:
https://docs.splunk.com/Documentation/AddOns/released/VMW/Troubleshoot

0 Karma

nickhills
Ultra Champion

There is nothing obviously wrong with you config (although please use the code tool 101010 to post config snippets because its possible something has not been shown correctly in the text view)

As suggested, btool is a good test to make sure there is nothing wrong in your configuration.
You noted that:

there many other inputs app which is routing data to both lab and prod indexers
Is that true for this host? Is it possible that firewall rules/ACLs are preventing this host from communicating with the prod-indexers?

If my comment helps, please give it a thumbs up!
0 Karma

13tsavage
Communicator

Have you run a btool check and see if that outputs any stdout messages?

Run $SPLUNK_HOME/bin/splunk btool check on your UF and it should show any errors in your outputs.conf and inputs.conf.

If there are no errors reported, then we know it is not the UF's fault that it cannot send to the prod indexers. Then you need to look at those prod indexers and check if their inputs.conf is configured to receive the correct input parameters.

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...