Getting Data In

Why am I unable to route syslog data to an index other than main?

timmy13
Communicator

I almost hesitate to ask this because I know the answer must be simple.

I have a small indexer clustering environment with a cluster master and two indexers. I am successfully receiving UDP:514 data, but it is being placed into the main index.

I have created an app, $splunkhome/etc/master_apps/syslogapp

Inside that, in the local directory, I have created the following inputs.conf:

[udp://514]
connection_host = ip
sourcetype = syslog
disabled = 0
index = poc

I pushed the configuration bundle successfully, however, syslog data is still being sent to the main index, not poc.

What am I missing?

0 Karma

woodcock
Esteemed Legend

Your directory path is wrong.

This is wrong:

$splunkhome/etc/master_apps/syslogapp

It should be this:

$SPLUNK_HOME/etc/syslogapp/default/

The put your inputs.conf, etc. there.
There is probably another problem, too. There is likely some other input already listening on that port. You need to find that first and disable that input.

0 Karma

timmy13
Communicator

Firewalls are off

0 Karma

timmy13
Communicator

I just discovered something. Now, since I added this app and inputs, the UDP 514 data is not getting indexed into any index. Neither main nor poc.

Didn't find anything else with an associated transforms.

0 Karma

sjohnson_splunk
Splunk Employee
Splunk Employee

If this is running on Linux, check your iptables. You could be blocking incoming traffic on that port.

On windows it could be windows firewall or your endpoint protection blocking.

Finally could also be some kind of network firewall rule.

0 Karma

sjohnson_splunk
Splunk Employee
Splunk Employee

Last guess is file permissions in the app dir or inputs.conf file.

0 Karma

timmy13
Communicator

Firewalls have all been disabled

0 Karma

craigv_splunk
Splunk Employee
Splunk Employee

Hmm. There is one thing I'm not sure about. It might be benign but it definitely strikes me as strange port and dedicatedIothread are both confiugrations associated with the http input not, to my knowledge, the udp port listener. This doesn't explain why its not indexing properly but it might lead to something.
Try running:
$SPLUNK_HOME/bin/splunk cmd btool inputs list --debug

This will show which file is creating that configuration entry. It might be helpful to see where that config is coming from

0 Karma

timmy13
Communicator

I see nothing in the btool for those entries besides splunk_httpinput

0 Karma

sjohnson_splunk
Splunk Employee
Splunk Employee

Your inputs looks fine.

The other possibility is that there is a transform somewhere that is over-riding the index setting for either the source (udp://514) or the sourcetype (syslog). You should run the btool command on the props and look for a TRANSFORMS- statement associated with either source or sourcetype.

If you one, you will need to locate the app and edit the transforms.conf to fix

0 Karma

timmy13
Communicator

[udp://514]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
connection_host = dns
dedicatedIoThreads = 2
disabled = 0
enableSSL = 1
host = ln-mcl-vm-000-02
index = poc
maxSockets = 0
maxThreads = 0
port = 8088
sourcetype = syslog
sslVersions = *,-ssl2
useDeploymentServer = 0

Looks ok to me, see anything wrong?

0 Karma

timmy13
Communicator

The index does exist on all indexers and is receiving data from UF's. Syslog data in main index does have the sourcetype of syslog.

0 Karma

craigv_splunk
Splunk Employee
Splunk Employee

First I would try to ensure that the configuration is recognized on the indexer correctly. On the indexer. Run:
$SPLUNK_HOME/bin/splunk cmd btool inputs list.

In the resulting printout, do all of those configurations parameters show up in the result

If you have a ton of configuration you may want to run:
$SPLUNK_HOME/bin/splunk cmd btool --app=[your-app-name] inputs list.

0 Karma

timmy13
Communicator

Looks good to me...

[udp://514]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
connection_host = dns
dedicatedIoThreads = 2
disabled = 0
enableSSL = 1
host = ln-mcl-vm-000-02
index = poc
maxSockets = 0
maxThreads = 0
port = 8088
sourcetype = syslog
sslVersions = *,-ssl2
useDeploymentServer = 0

See anything I'm missing?

0 Karma

timmy13
Communicator

Looks good to me... [udp://514]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
connection_host = dns
dedicatedIoThreads = 2
disabled = 0
enableSSL = 1
host = ln-mcl-vm-000-02
index = poc
maxSockets = 0
maxThreads = 0
port = 8088
sourcetype = syslog
sslVersions = *,-ssl2
useDeploymentServer = 0

Do you see anything wrong?

0 Karma

sjohnson_splunk
Splunk Employee
Splunk Employee

Does the data in the main index have the sourcetype = syslog?

Did you create the index = poc on all your indexers?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!