How to troubleshoot configuration mismatch in inputs.conf and outputs.conf?

[antifreke]

Background, I am not an engineer and have little engineering experience. In setting up my instance, I have a question about the .Conf files.

Search Head - x.x.x.25
Syslog Server - x.x.x.24
Indexer 1- x.x.x.23

if I'm forwarding syslog data on udp 514, I have the following:

inputs.conf

[udp://514]
connection_host=dns
index=syslog
sourcetype=syslog

outputs.conf

[syslog:syslogGroup]
server = x.x.x.23:9997

[tcpout:indexer1]
server:x.x.x.23:9997

When I run list forward-server, I get the following:

Active forwards: none
configured but inactive: x.x.x.23:9997

Any ideas how I got this mismatch and what I need to do do make them active? I currently have no issues with networking, no firewalls, and can openly ping between devices. Thoughts?

[lguinn2 · Accepted solution]

Your outputs.conf is misconfigured. It should be

[tcpout:group1]
server=x.x.x.23:9997

The tcpout specifies Splunk-forwarder-to-Splunk-indexer communication. group1 can be any unique name; it is needed only if there are multiple tcpout stanzas. (So "indexer1" is actually okay here, but it doesn't really mean anything.)
The server specification of x.x.x.23:9997 should be fine - as long as the indexer running on x.x.x.23 is actually listening on port 9997. If it is not, that would be one reason that it would show up as "inactive."

The [syslog:syslogGroup] stanza should be removed. It is specifying that the forwarder should send data in syslog format to the server - which it should not do. This also could be the reason that the indexer shows as inactive.

For more information, you should look at $SPLUNK_HOME/var/log/splunkd.log

[antifreke]

Alright, I have that up.. but I think I figured out part of my problem in rsyslog.conf. Does the following look like I did this correctly?

if $fromhost-ip startswith 'x.x.x.23; then /var/log/rsyslog/devices.log
&~