Getting Data In

Why are my Splunk UF events being sent to main index, even with a customized inputs.conf file?

JMondares
Explorer

Hello,

I'm troubleshooting an issue with my proof-of-concept Splunk instance (single instance of Splunk Enterprise hosted on-prem) and was wondering if someone could help.

I have a UF deployed on a single Windows server to test ingestion, sending to a custom index, and filtering events by the Splunk instance prior to indexing, and the UF isn't having any issues communicating or sending events to the Splunk instance. Unfortunately, all the events are being sent to and indexed by the "main" index, even with a customized inputs.conf file specifying that I only need certain Event IDs indexed instead, and sent to a custom index I created named "windowseventlog."

I do have the Splunk Add-On for Microsoft Windows installed, and I've configured my inputs.conf file on the Splunk server (located at /opt/splunk/etc/apps/Splunk_TA_windows/local/inputs.conf and edited with vim). I can also confirm that the index named "windowseventlog" exists, is enabled, and is named correctly.

Here was a snippet from the customized inputs.conf file that I had set up in that directory:

=======

 

 

[WinEventLog://System]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
index = windowseventlog
renderXml=true
whitelist= 4768,4769,4770,4771,4772,4773,4774,4775,4776,4777,4820,4720,4722,4723,4724,4725,4726,4727,4728,4729,4730,4731,4732,4733,4734,4735,4737,4738,4739,4740,4741,4742,4743,4744,4745,4746,4747,4748,4749,4750,4751,4752,4753,4754,4755,4756,4757,4758,4759,4760,4761,4762,4763,4764,4765,4766,4767,4780,4781,4782,4783,4784,4785,4786,4787,4788,4789,4790,4791,4792,4793,4794,4797,4798,4799,5376,5377,4661,4662,4928,4929,4930,4931,4932,4933,4934,4935,4936,4937,5136,5137,5138,5139,5141,5169,5170,4624,4625,4626,4627,4634,4646,4647,4648,4649,4650,4651,4652,4653,4654,4655,4672,4675,4778,4779,4800,4801,4802,4803,4964,4976,4977,4978,4979,4980,4981,4982,4983,4984,5378,5451,5452,5453,5632,5633,6272,6273,6274,6275,6276,6277,6278,6279,6280,1100,1101,1102,1104,1105,1108,4656,4657,4658,4659,4660,4661,4663,4664,4665,4666,4667,4668,4670,4671,4690,4691,4698,4699,4700,4701,4702,4818,4868,4869,4870,4871,4872,4873,4874,4875,4876,4877,4878,4879,4880,4881,4882,4883,4884,4885,4886,4887,4888,4889,4890,4891,4892,4893,4894,4895,4896,4897,4898,4899,4900,4985,5031,5120,5140,5142,5143,5144,5145,5148,5149,5150,5151,5152,5153,5154,5155,5156,5157,5158,5159,5168,5888,5889,4670,4703,4704,4705,4706,4707,4709,4710,4711,4712,4713,4714,4715,4716,4717,4718,4719,4817,4819,4826,4865,4866,4867,4902,4904,4905,4906,4907,4908,4911,4912,4913,4944,4945,4946,4947,4948,4949,4950,4951,4952,4954,4956,4957,4958,5063,5064,5065,5066,5067,5068,5069,5070,5440,5441,5442,5443,5444,5446,5447,5448,5449,5450,5456,5457,5458,5459,5460,5461,5462,5463,5464,5465,5466,5467,5468,5471,5472,5473,5474,5477,6144,6145,4673,4674,4688,4689,4692,4693,4694,4695,4696,4816,5712,6416,6419,6420,6421,6422,6423,6424,4864,4909,4910,4953,4960,4961,4962,4963,4965,5039,5040,5041,5042,5043,5044,5045,5046,5047,5048,5049,5050,5051,5057,5060,5062,5121,5122,5123,5124,5125,5126,5127

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
index = windowseventlog
renderXml=true
whitelist=4608,4609,4610,4611,4612,4614,4615,4616,4618,4621,4622,4697,4821,4822,4823,4824,4825,4830,5024,5025,5027,5028,5029,5030,5032,5033,5034,5035,5037,5038,5056,5058,5059,5061,5071,5146,5147,5379,5380,5381,5382,5478,5479,5480,5483,5484,5485,5890,6281,6400,6401,6402,6403,6404,6405,6406,6407,6408,6409,6410,6417,6418,8191
blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)"

###### Forwarded WinEventLogs (WEF) ######
[WinEventLog://ForwardedEvents]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5
## The addon supports only XML format for the collection of WinEventLogs using WEF, hence do not change the below renderXml parameter to false.
renderXml=true
host=WinEventLogForwardHost

 

 

=======

Here's the list of all the directories and files that are in the /opt/splunk/etc/apps directory on the Splunk server:

 

 

alert_logevent                 
alert_webhook                  
appsbrowser                    
force_directed_viz             
InfoSec_App_for_Splunk         
introspection_generator_addon  
journald_input                 
launcher                       
learned                        
legacy                         
lookup_editor                  
punchcard_app                  
python_upgrade_readiness_app   
sample_app                     
sankey_diagram_app             
search                         
splunk_archiver                
splunk-dashboard-studio        
splunk_essentials_8_2          
SplunkForwarder
splunk_gdi
splunk_httpinput
splunk_instrumentation
splunk_internal_metrics
SplunkLightForwarder
splunk_metrics_workspace
splunk_monitoring_console
splunk_rapid_diag
Splunk_SA_CIM
splunk_secure_gateway
Splunk_TA_cisco-asa
Splunk_TA_citrix-netscaler
Splunk_TA_Google_Workspace
Splunk_TA_microsoft-iis
Splunk_TA_microsoft-sqlserver
Splunk_TA_windows
TA-Okta_Identity_Cloud_for_Splunk
user-prefs
vmware_app_for_splunk

 

 

========

Is it possible that I'm editing the wrong inputs.conf file? If so, is there a "better" file I should be editing? Please let me know if there's any additional information that I can provide that can help better troubleshoot this issue.

Thanks for any assistance you can provide!

Jason

Labels (4)
0 Karma

PickleRick
Ultra Champion

Wait a second. Something is confusing me here. Do you have your inputs.conf defined on your splunk.server? It should be configured on the forwarder that's performing the ingestion.

0 Karma

JMondares
Explorer

Hi @PickleRick ,

It is defined on the Splunk server. It's just a single instance that's all running on a single server (this is just a proof-of-concept that's going to be deleted once a sizing exercise is done).

0 Karma

PickleRick
Ultra Champion

Ok. But /opt/splunk/something paths are linux paths, so you have your server on linux, right?

But you want events from windows so you must have UF on windows. Or am I missing something?

The inputs must be defined on the windows component.

0 Karma

JMondares
Explorer

Yes, the server is running Linux, and the UF is deployed on a Windows server.

I need select events captured in the Security and System Event Viewer logs sent from the UF on the Windows server to a custom index named "windowseventlog" on the Linux Splunk server, so I can see just how much log data gets ingested in a given day so I can properly size our Splunk license.

So if I only wanted certain events sent from the UF to a specific index, would I have to define all that in the UF's settings? I was under the impression that I had to do all that on the Splunk server.

Thanks for checking.

0 Karma

PickleRick
Ultra Champion

Remember that what you call "splunk server" which in your case is just an "all in one" installation can be deployed as a huge multilayered multisite installation in which you don't have a single component that you could call "splunk server" 🙂

Anyway, the config files have effect in the place they are deployed. So inputs.conf on a splunk component defines what happens on this particular splunk component.

There is some additional mechanics involved when you're using deployment server/deployer to manage remote nodes' configuration from a single place but even then you're just preparing a file "locally" and then distribute it to your forwarders, search heads and whatnot and the file gets applied there. But that's not the case as I understand you're not using your splunk server as the deployment server.

I suppose you installed your Splunk all-in-one on a Linux box then installed a Universal Forwarder on the Windows server and during the installation pointed it to your Splunk server and checked the "collect windows eventlogs" boxes. After that you installed your TA-windows to the Splunk _server_.

So now the situation looks like this:

1) You have your Universal Forwarder with inputs.conf created by the installer during the installation which contains default stanzas enabling event logs collection. Since they haven't been reconfigured in any way they simply pull events from eventlog, set proper sourcetypes and send them to your splunk server which by default places them into the main index.

2) You have your TA-windows installed on the Splunk server and you created inputs.conf there. Splunk is reading this file and probably tries to conform to the settings contained therein but since it's a Linux server it cannot run event log collection because it has no event log to work on (and doesn't have the exe files to perform the process anyway). But still the rest of the TA-windows app is in effect, so your events sent from the Universal Forwarder are properly parsed and displayed when you're doing a search and they're CIM-compliant. If your Splunk server was running on Windows box, the configuration that you created in your hand-made inputs.conf would be applied on this Splunk server and would regard local event log collection (in fact it would be merged with the default settings but that's a topic for another story).

In order to force specific destination index and event filtering on input you have to put the inputs.conf settings on the Universal Forwarder that's doing the actual Event Log collection.

0 Karma

venky1544
Contributor
Hey @JMondares 
you are looking at the wrong location  as per your screenshot you have created the index in search app so check your \etc\apps\search\local you will find the indexes.conf  and probably could see the index you created and about the data going in main index seems this is pretty much a precedence issue you should use btool on the command line 
run the below command
/opt/splunk/bin/splunk cmd btool inputs list --debug  
on the splunk server. you would get all the configured inputs and you can check if there are other configurations that have the same monitor if still a confusion you could share the output of btool in the chat 
 
Happy Splunking
 

JMondares
Explorer

Hi @venky1544 ,

Thanks for your help on this, that command is great. I'm still getting used to the more nitty-gritty parts of the Splunk config, so I'll chalk all this up to a newbie taking his lumps. 🙂

I've attached the output from that command you mentioned to this post. Do you see anything out of the ordinary?

0 Karma

m_pham
Splunk Employee
Splunk Employee

Don't know how I overlooked btool - that's also the fastest way to pin down the configs.

0 Karma

m_pham
Splunk Employee
Splunk Employee

Are you 100% positive that the "windowseventlog" index exist on your indexer(s) or your all-in-one Splunk server? I would double check your indexes.conf.

JMondares
Explorer

Hi m_pham,

Thanks for responding and taking a look at this, I appreciate it.

I did look at the Indexes settings page, and it says that the index does exist in $SPLUNK_DB/windowseventlog/db. I've attached a screenshot for reference.

indexes_listing.png

I did create the windowseventlog index through this page, though. Is it possible that that might have been the problem? I have a couple of other indexes that I created (e.g. carbonblackcloud and cbc_action), and I think I might have created that through the Terminal, but I can't remember off the top of my head if I did.

Also, I went into /opt/splunk/etc/system/local, and there isn't an indexes.conf file present:

etc_system_local_dir.png

Is it safe to say that I should create this file manually, and/or perhaps delete and re-create the windowseventlog index through the Terminal? This is just a proof-of-concept, so I don't mind wiping this index out and starting over.

0 Karma

m_pham
Splunk Employee
Splunk Employee

That is kind of odd that the index isn't there when it's there in the GUI. Can you look for an indexes.conf file within the $SPLUNK_HOME/etc/apps directory? Sometimes users click into an app or TA, then go to the indexes settings and create an index there without realizing that the index.conf gets put into that TA/app context. If you find the indexes.conf, move it to etc/system/local and restart Splunk.

Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! 👏 Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...