Why are my Splunk UF events being sent to main index, even with a customized inputs.conf file?

[JMondares]

[Contents deleted per request]

[PickleRick]

Wait a second. Something is confusing me here. Do you have your inputs.conf defined on your splunk.server? It should be configured on the forwarder that's performing the ingestion.

[JMondares]

Hi @PickleRick ,

It is defined on the Splunk server. It's just a single instance that's all running on a single server (this is just a proof-of-concept that's going to be deleted once a sizing exercise is done).

[PickleRick]

Ok. But /opt/splunk/something paths are linux paths, so you have your server on linux, right?

But you want events from windows so you must have UF on windows. Or am I missing something?

The inputs must be defined on the windows component.

[JMondares]

[Deleted per request]

[PickleRick]

Remember that what you call "splunk server" which in your case is just an "all in one" installation can be deployed as a huge multilayered multisite installation in which you don't have a single component that you could call "splunk server" 🙂

Anyway, the config files have effect in the place they are deployed. So inputs.conf on a splunk component defines what happens on this particular splunk component.

There is some additional mechanics involved when you're using deployment server/deployer to manage remote nodes' configuration from a single place but even then you're just preparing a file "locally" and then distribute it to your forwarders, search heads and whatnot and the file gets applied there. But that's not the case as I understand you're not using your splunk server as the deployment server.

I suppose you installed your Splunk all-in-one on a Linux box then installed a Universal Forwarder on the Windows server and during the installation pointed it to your Splunk server and checked the "collect windows eventlogs" boxes. After that you installed your TA-windows to the Splunk _server_.

So now the situation looks like this:

1) You have your Universal Forwarder with inputs.conf created by the installer during the installation which contains default stanzas enabling event logs collection. Since they haven't been reconfigured in any way they simply pull events from eventlog, set proper sourcetypes and send them to your splunk server which by default places them into the main index.

2) You have your TA-windows installed on the Splunk server and you created inputs.conf there. Splunk is reading this file and probably tries to conform to the settings contained therein but since it's a Linux server it cannot run event log collection because it has no event log to work on (and doesn't have the exe files to perform the process anyway). But still the rest of the TA-windows app is in effect, so your events sent from the Universal Forwarder are properly parsed and displayed when you're doing a search and they're CIM-compliant. If your Splunk server was running on Windows box, the configuration that you created in your hand-made inputs.conf would be applied on this Splunk server and would regard local event log collection (in fact it would be merged with the default settings but that's a topic for another story).

In order to force specific destination index and event filtering on input you have to put the inputs.conf settings on the Universal Forwarder that's doing the actual Event Log collection.

[venky1544]

Hey @JMondares 

you are looking at the wrong location  as per your screenshot you have created the index in search app so check your \etc\apps\search\local you will find the indexes.conf  and probably could see the index you created and about the data going in main index seems this is pretty much a precedence issue you should use btool on the command line 

run the below command

/opt/splunk/bin/splunk cmd btool inputs list --debug  

on the splunk server. you would get all the configured inputs and you can check if there are other configurations that have the same monitor if still a confusion you could share the output of btool in the chat 

Happy Splunking

[JMondares]

Hi @venky1544 ,

Thanks for your help on this, that command is great. I'm still getting used to the more nitty-gritty parts of the Splunk config, so I'll chalk all this up to a newbie taking his lumps. 🙂

I've attached the output from that command you mentioned to this post. Do you see anything out of the ordinary?

[Attachment deleted per request]

[m_pham]

Don't know how I overlooked btool - that's also the fastest way to pin down the configs.

[m_pham]

Are you 100% positive that the "windowseventlog" index exist on your indexer(s) or your all-in-one Splunk server? I would double check your indexes.conf.

[JMondares]

[Deleted per request]

[m_pham]

That is kind of odd that the index isn't there when it's there in the GUI. Can you look for an indexes.conf file within the $SPLUNK_HOME/etc/apps directory? Sometimes users click into an app or TA, then go to the indexes settings and create an index there without realizing that the index.conf gets put into that TA/app context. If you find the indexes.conf, move it to etc/system/local and restart Splunk.