I almost hesitate to ask this because I know the answer must be simple.
I have a small indexer clustering environment with a cluster master and two indexers. I am successfully receiving UDP:514 data, but it is being placed into the main index.
I have created an app, $splunkhome/etc/master_apps/syslogapp
Inside that, in the local directory, I have created the following inputs.conf:
[udp://514]
connection_host = ip
sourcetype = syslog
disabled = 0
index = poc
I pushed the configuration bundle successfully, however, syslog data is still being sent to the main index, not poc.
What am I missing?
Your directory path is wrong.
This is wrong:
$splunkhome/etc/master_apps/syslogapp
It should be this:
$SPLUNK_HOME/etc/syslogapp/default/
The put your inputs.conf, etc. there.
There is probably another problem, too. There is likely some other input already listening on that port. You need to find that first and disable that input.
Firewalls are off
I just discovered something. Now, since I added this app and inputs, the UDP 514 data is not getting indexed into any index. Neither main nor poc.
Didn't find anything else with an associated transforms.
If this is running on Linux, check your iptables. You could be blocking incoming traffic on that port.
On windows it could be windows firewall or your endpoint protection blocking.
Finally could also be some kind of network firewall rule.
Last guess is file permissions in the app dir or inputs.conf file.
Firewalls have all been disabled
Hmm. There is one thing I'm not sure about. It might be benign but it definitely strikes me as strange port and dedicatedIothread are both confiugrations associated with the http input not, to my knowledge, the udp port listener. This doesn't explain why its not indexing properly but it might lead to something.
Try running:
$SPLUNK_HOME/bin/splunk cmd btool inputs list --debug
This will show which file is creating that configuration entry. It might be helpful to see where that config is coming from
I see nothing in the btool for those entries besides splunk_httpinput
Your inputs looks fine.
The other possibility is that there is a transform somewhere that is over-riding the index setting for either the source (udp://514) or the sourcetype (syslog). You should run the btool command on the props and look for a TRANSFORMS- statement associated with either source or sourcetype.
If you one, you will need to locate the app and edit the transforms.conf to fix
[udp://514]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
connection_host = dns
dedicatedIoThreads = 2
disabled = 0
enableSSL = 1
host = ln-mcl-vm-000-02
index = poc
maxSockets = 0
maxThreads = 0
port = 8088
sourcetype = syslog
sslVersions = *,-ssl2
useDeploymentServer = 0
Looks ok to me, see anything wrong?
The index does exist on all indexers and is receiving data from UF's. Syslog data in main index does have the sourcetype of syslog.
First I would try to ensure that the configuration is recognized on the indexer correctly. On the indexer. Run:
$SPLUNK_HOME/bin/splunk cmd btool inputs list.
In the resulting printout, do all of those configurations parameters show up in the result
If you have a ton of configuration you may want to run:
$SPLUNK_HOME/bin/splunk cmd btool --app=[your-app-name] inputs list.
Looks good to me...
[udp://514]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
connection_host = dns
dedicatedIoThreads = 2
disabled = 0
enableSSL = 1
host = ln-mcl-vm-000-02
index = poc
maxSockets = 0
maxThreads = 0
port = 8088
sourcetype = syslog
sslVersions = *,-ssl2
useDeploymentServer = 0
See anything I'm missing?
Looks good to me... [udp://514]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
connection_host = dns
dedicatedIoThreads = 2
disabled = 0
enableSSL = 1
host = ln-mcl-vm-000-02
index = poc
maxSockets = 0
maxThreads = 0
port = 8088
sourcetype = syslog
sslVersions = *,-ssl2
useDeploymentServer = 0
Do you see anything wrong?
Does the data in the main index have the sourcetype = syslog?
Did you create the index = poc on all your indexers?