Getting Data In

How to limit a data sent to indexers to only with certain windows EventCodes ?

Hemnaath
Motivator

Hi , We have recently added a below stanza to monitor the windows power shell events and we have started getting the data from the remote windows machines, but data being ingested is a bit more than expected, so we wanted to limit the data sent to the indexers to only with the these windows Eventcodes=800,Eventcodes=4103 & Eventcodes=4104 apart from these, there are other events code are getting ingested into indexer but we want to restrict other EventCodes from being ingested into indexer instances.

Inputs.conf stanza details:

[WinEventLog://Windows PowerShell]
disabled = 0
ignoreOlderThan = 2d
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
index = windows

Kindly guide if there is a way to limit the unwanted Windows Event Codes being ingested in splunk.

0 Karma
1 Solution

FrankVl
Ultra Champion

The WinEventLog input type supports whitelisting and blacklisting through inputs.conf. I think that would be much simpler than going through the queue routing approach (and is also more efficient as it is dropped earlier on in the chain).

Docs.splunk.com seems to be down at the moment, so I can't lookup the exact config instructions for you, but just have a look at the WinEventLog section in the inputs.conf spec once docs is working again. It shows clear examples of how to enable only a few specific event codes 🙂

View solution in original post

0 Karma

FrankVl
Ultra Champion

The WinEventLog input type supports whitelisting and blacklisting through inputs.conf. I think that would be much simpler than going through the queue routing approach (and is also more efficient as it is dropped earlier on in the chain).

Docs.splunk.com seems to be down at the moment, so I can't lookup the exact config instructions for you, but just have a look at the WinEventLog section in the inputs.conf spec once docs is working again. It shows clear examples of how to enable only a few specific event codes 🙂

0 Karma

Hemnaath
Motivator

thanks FrankVI, for much need a help on this problem, I am working on this for more than five hours with the above stanza , but unable to limit the eventcodes. When test with the above stanza I got the below error detail from splunkd.log,so not sure whether above stanza are correct or not correct me if it is wrong.

ERROR Details:
04-26-2018 16:55:13.883 +0530 ERROR regexExtractionProcessor - REGEX field must be specified tranform_name=PowerShellnull
host = lab.thay.com source = /opt/splunk/var/log/splunk/splunkd.log sourcetype =splunkd

Anyway let me go through the inputs.conf document for whistling only few EventCodes.

thanks for your much needed help 🙂

0 Karma

Hemnaath
Motivator

Frank thanks for guiding me on this problem, Yes after updating the white list stanza in inputs.conf, we could limit/restrict only the particular events data getting ingested into splunk.

Steps:

1) Edited the inputs.conf with the below stanza details. 
[WinEventLog://Windows PowerShell]
  disabled = 0
  ignoreOlderThan = 2d
  current_only = 0
  evt_resolve_ad_obj = 1
  checkpointInterval = 5
  whitelist=800
  index = windows

 [WinEventLog://Microsoft-Windows-PowerShell/Operational]
  disabled = 0
  ignoreOlderThan = 2d
  current_only = 0
  evt_resolve_ad_obj = 1
  checkpointInterval = 5
  whitelist = 4103,4104
  index = windows

2) After updating the inputs.conf stanza, it was deployed to all the universal forwarder via deployment server, by executing the ./splunk reload deploy-server.

3) Validated by executing the simple splunk query.

index=windows sourcetype="WinEventLog:Microsoft-Windows-PowerShell/Operational" EventCode!=4103 AND EventCode!=4104 | dedup host EventCode | table host EventCode

index=windows sourcetype="WinEventLog:Windows PowerShell" EventCode!=800 | dedup host EventCode | table host EventCode

4) By executing the above query got zero result.

FrankVl
Ultra Champion

Should be as simple as adding the following to your inputs:

whitelist = 800,4103,4104

See: http://docs.splunk.com/Documentation/Splunk/latest/Admin/inputsconf#Event_Log_whitelist_and_blacklis...

0 Karma

Hemnaath
Motivator

Hi Frank, hey can I use the below syntax to include only these EventCode 800, 4103 & 4104 event data from these two sourcetype in the inputs.conf.

[WinEventLog://Windows PowerShell]
disabled = 0
ignoreOlderThan = 2d
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
whitelist= EventCode=800
index = windows

[WinEventLog://Microsoft-Windows-PowerShell/Operational]
disabled = 0
ignoreOlderThan = 2d
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
whitelist= Eventcode=4103
whitelist1= Eventcode=4104
index = windows

Got a doubt on this syntax as per the Splunk document the syntax provide for white listing the EventCode.

syntax
whitelist = key=

link to the document:
http://docs.splunk.com/Documentation/Splunk/7.1.0/Data/MonitorWindowseventlogdata#Create_advanced_fi...

But in the above syntax we have not provide any regex details, so will this work. I mean by pushing this syntax will I able to include only with the event data containing the Event Code 800,4103 & 4104 in to splunk, not all other EventCodes.

Kindly guide me on this.

0 Karma

FrankVl
Ultra Champion

Like I mentioned above, if you just want to filter for EventCodes, no need to specify the key, you can just do this: whitelist = 800 in the first stanza and whitelist = 4103,4104 in the second stanza.

0 Karma

Hemnaath
Motivator

thanks for your instant response on this, Frank we want only these EventCodes into Splunk not other EventCodes which currently getting ingested into splunk.

 [WinEventLog://Windows PowerShell]
 disabled = 0
 ignoreOlderThan = 2d
 current_only = 0
 evt_resolve_ad_obj = 1
 checkpointInterval = 5
 whitelist=800
 index = windows

[WinEventLog://Microsoft-Windows-PowerShell/Operational]
 disabled = 0
 ignoreOlderThan = 2d
 current_only = 0
 evt_resolve_ad_obj = 1
 checkpointInterval = 5
 whitelist = 4103,4104
 index = windows

Kindly guide me on this.

0 Karma

FrankVl
Ultra Champion

So is that latest config working? Or are you still receiving other event types as well?

0 Karma

Hemnaath
Motivator

Not yet pushed as we have almost 2000 + RM nodes configured in our environment, so need to get an approval to push this syntax and validate it.

0 Karma

FrankVl
Ultra Champion

No test environment for this? Ok, anyway, just give it a try and report back here with the results 🙂

0 Karma

Hemnaath
Motivator

Frank, the above syntax did not work, we are able to see other EventCodes are also being ingested into splunk indexer instance, so kindly guide me on this.

0 Karma

FrankVl
Ultra Champion

I only have experience with the blacklist variant of this approach and that worked perfectly fine for me with similar syntax. So I'm a bit puzzled why this is not working for you.

Have you double and triple checked that this config has really been deployed like this and properly activated on the forwarders? And are you sure that timestamping / timesync of your hosts is accurate, so you're not looking at events that were actually ingested from before the change was implemented?

In general, I would suggest you set up some test instance for trying out these configs, before rolling it out. That makes it easier to try some things and also makes it a lot less complex to troubleshoot than a 2000+ forwarder environment. Maybe a 100 forwarders did not pick up the change properly and cause you to think the change itself is bad.

0 Karma

Hemnaath
Motivator

Frank you are right, I had pushed the changes around 10:20 EDT and validate the same by executing the below query with time set from 10:40 EDT to Current time 13:20 EDT and found that its working perfectly, but could see only two of the host having an issue, when further investigated, noticed that app was not update, instead it was having an Error.

Query Details:

index=windows sourcetype="WinEventLog:Microsoft-Windows-PowerShell/Operational" EventCode!=4103 AND EventCode!=4104 | dedup host EventCode | table host EventCode

index=windows sourcetype="WinEventLog:Windows PowerShell" EventCode!=800 | dedup host EventCode | table host EventCode

Host having an issue in getting the update from DP instances.

05-02-2018 13:28:21.783 -0400 WARN ClientSessionsManager - ip=10.x.x.x name=9C691DDB-AF4A-4F10-90DB-94341B304F91 Updating record for sc=Test-SC-all_windows app=Test-IA-windows: action=Install result=Fail checksum=1927475316208885921

Anyway I will monitor for some more time and update you the result.

0 Karma

p_gurav
Champion

You can use route and filter at indexer end. Refer:
http://docs.splunk.com/Documentation/Splunk/7.0.3/Forwarding/Routeandfilterdatad

0 Karma

Hemnaath
Motivator

hey can anyone guide me on the below stanza. we have tested the regex and its working fine, we are able to filter only the EventCode=800, 4013, 4014 by using the respective sourcetype.

index=win sourcetype="WinEventLog:Microsoft-Windows-PowerShell/Operational" | regex "(?m)^EventCode=(4103|4104)"

index=win sourcetype="WinEventLog:Windows PowerShell" | regex "(?m)^EventCode=800"

kindly guide me whether the above props & transforms.conf stanza are good to implement.

0 Karma

Hemnaath
Motivator

Hey I had gone through referred splunk document link and created a stanza based on example provide in the document but It is not working as expected.

Could any one guide me on the stanza to limit data to the indexer instance.
thanks in advance.

0 Karma

ddrillic
Ultra Champion

The example I see, is slightly different -

[WinEventLog:System]
TRANSFORMS = null_queue_filter

[null_queue_filter]
REGEX = (?m)^EventCode=(592|593)
DEST_KEY = queue
FORMAT = nullQueue
0 Karma

p_gurav
Champion

Can you try in inputs.conf:

whitelist1= EventCode=800
whitelist2= Eventcodes=4103
whitelist3= Eventcodes=4104
0 Karma

Hemnaath
Motivator

thank gurav for sharing the link, but can you please correct me whether the below stanza can resolve my requirement.

Currently we need only the below EventCodes not the other eventcodes.

  sourcetype="WinEventLog:Microsoft-Windows-PowerShell/Operational"
    LogName=Microsoft-Windows-PowerShell/Operational
    SourceName=Microsoft-Windows-PowerShell
    EventCode=4103
    EventType=4

    sourcetype="WinEventLog:Microsoft-Windows-PowerShell/Operational"
    LogName=Microsoft-Windows-PowerShell/Operational
    SourceName=Microsoft-Windows-PowerShell
    EventCode=4104
    EventType=5

    sourcetype="WinEventLog:Windows
    LogName=Windows PowerShell
    SourceName=PowerShell
    EventCode=800
    EventType=4

Kindly guide me on this.

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...