I'm troubleshooting an issue with my proof-of-concept Splunk instance (single instance of Splunk Enterprise hosted on-prem) and was wondering if someone could help.
I have a UF deployed on a single Windows server to test ingestion, sending to a custom index, and filtering events by the Splunk instance prior to indexing, and the UF isn't having any issues communicating or sending events to the Splunk instance. Unfortunately, all the events are being sent to and indexed by the "main" index, even with a customized inputs.conf file specifying that I only need certain Event IDs indexed instead, and sent to a custom index I created named "windowseventlog."
I do have the Splunk Add-On for Microsoft Windows installed, and I've configured my inputs.conf file on the Splunk server (located at /opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf and edited with vim). I can also confirm that the index named "windowseventlog" exists, is enabled, and is named correctly.
Here was a snippet from the customized inputs.conf file that I had set up in that directory:
Here's the list of all the directories and files that are in the /opt/splunk/etc/apps directory on the Splunk server:
alert_logevent alert_webhook appsbrowser force_directed_viz InfoSec_App_for_Splunk introspection_generator_addon journald_input launcher learned legacy lookup_editor punchcard_app python_upgrade_readiness_app sample_app sankey_diagram_app search splunk_archiver splunk-dashboard-studio splunk_essentials_8_2 SplunkForwarder splunk_gdi splunk_httpinput splunk_instrumentation splunk_internal_metrics SplunkLightForwarder splunk_metrics_workspace splunk_monitoring_console splunk_rapid_diag Splunk_SA_CIM splunk_secure_gateway Splunk_TA_cisco-asa Splunk_TA_citrix-netscaler Splunk_TA_Google_Workspace Splunk_TA_microsoft-iis Splunk_TA_microsoft-sqlserver Splunk_TA_windows TA-Okta_Identity_Cloud_for_Splunk user-prefs vmware_app_for_splunk
Is it possible that I'm editing the wrong inputs.conf file? If so, is there a "better" file I should be editing? Please let me know if there's any additional information that I can provide that can help better troubleshoot this issue.
Thanks for any assistance you can provide!
Wait a second. Something is confusing me here. Do you have your inputs.conf defined on your splunk.server? It should be configured on the forwarder that's performing the ingestion.
Hi @PickleRick ,
It is defined on the Splunk server. It's just a single instance that's all running on a single server (this is just a proof-of-concept that's going to be deleted once a sizing exercise is done).
Ok. But /opt/splunk/something paths are linux paths, so you have your server on linux, right?
But you want events from windows so you must have UF on windows. Or am I missing something?
The inputs must be defined on the windows component.
Yes, the server is running Linux, and the UF is deployed on a Windows server.
I need select events captured in the Security and System Event Viewer logs sent from the UF on the Windows server to a custom index named "windowseventlog" on the Linux Splunk server, so I can see just how much log data gets ingested in a given day so I can properly size our Splunk license.
So if I only wanted certain events sent from the UF to a specific index, would I have to define all that in the UF's settings? I was under the impression that I had to do all that on the Splunk server.
Thanks for checking.
Remember that what you call "splunk server" which in your case is just an "all in one" installation can be deployed as a huge multilayered multisite installation in which you don't have a single component that you could call "splunk server" 🙂
Anyway, the config files have effect in the place they are deployed. So inputs.conf on a splunk component defines what happens on this particular splunk component.
There is some additional mechanics involved when you're using deployment server/deployer to manage remote nodes' configuration from a single place but even then you're just preparing a file "locally" and then distribute it to your forwarders, search heads and whatnot and the file gets applied there. But that's not the case as I understand you're not using your splunk server as the deployment server.
I suppose you installed your Splunk all-in-one on a Linux box then installed a Universal Forwarder on the Windows server and during the installation pointed it to your Splunk server and checked the "collect windows eventlogs" boxes. After that you installed your TA-windows to the Splunk _server_.
So now the situation looks like this:
1) You have your Universal Forwarder with inputs.conf created by the installer during the installation which contains default stanzas enabling event logs collection. Since they haven't been reconfigured in any way they simply pull events from eventlog, set proper sourcetypes and send them to your splunk server which by default places them into the main index.
2) You have your TA-windows installed on the Splunk server and you created inputs.conf there. Splunk is reading this file and probably tries to conform to the settings contained therein but since it's a Linux server it cannot run event log collection because it has no event log to work on (and doesn't have the exe files to perform the process anyway). But still the rest of the TA-windows app is in effect, so your events sent from the Universal Forwarder are properly parsed and displayed when you're doing a search and they're CIM-compliant. If your Splunk server was running on Windows box, the configuration that you created in your hand-made inputs.conf would be applied on this Splunk server and would regard local event log collection (in fact it would be merged with the default settings but that's a topic for another story).
In order to force specific destination index and event filtering on input you have to put the inputs.conf settings on the Universal Forwarder that's doing the actual Event Log collection.
Hi @venky1544 ,
Thanks for your help on this, that command is great. I'm still getting used to the more nitty-gritty parts of the Splunk config, so I'll chalk all this up to a newbie taking his lumps. 🙂
I've attached the output from that command you mentioned to this post. Do you see anything out of the ordinary?
Thanks for responding and taking a look at this, I appreciate it.
I did look at the Indexes settings page, and it says that the index does exist in $SPLUNK_DB/windowseventlog/db. I've attached a screenshot for reference.
I did create the windowseventlog index through this page, though. Is it possible that that might have been the problem? I have a couple of other indexes that I created (e.g. carbonblackcloud and cbc_action), and I think I might have created that through the Terminal, but I can't remember off the top of my head if I did.
Also, I went into /opt/splunk/etc/system/local, and there isn't an indexes.conf file present:
Is it safe to say that I should create this file manually, and/or perhaps delete and re-create the windowseventlog index through the Terminal? This is just a proof-of-concept, so I don't mind wiping this index out and starting over.
That is kind of odd that the index isn't there when it's there in the GUI. Can you look for an indexes.conf file within the $SPLUNK_HOME/etc/apps directory? Sometimes users click into an app or TA, then go to the indexes settings and create an index there without realizing that the index.conf gets put into that TA/app context. If you find the indexes.conf, move it to etc/system/local and restart Splunk.