How to filter Windows event logs on a Splunk 6.2.3 forwarder?

[vad34]

Hello

How do I filter events (Windows event log) on a forwarder? btw how do I install a heavy forwarder?
I have Splunk 6.2.3.

tnx in advance

[javiergn · Accepted solution]

Hi, I'm running out of ideas.
The last thing I would suggest is to run the diagnostic tool and upload the output to Dropbox so that I can take a look. Please make sure there's nothing sensitive there that I shouldn't have access to.

How to generate a diag: http://docs.splunk.com/Documentation/Splunk/6.3.2/Troubleshooting/Generateadiag

If I can't find anything in there I would recommend you to open a support call with Splunk as they will be in a much better position than me to debug this problem.

Thanks,
J

[vad34]

Hi , can you please reproduce this issue by yourself in lab?

[javiergn]

Hi, I couldn't reproduce the issue in my environment. All my white lists and blacklists are working fine for both Security and System event logs.

If you upload a diag file I might be able to replicate your environment but without that there's a lot of guessing about your unique deployment.

That's the reason Splunk Support is going to ask your for that diag file too as it is the only realistic way to reproduce an environment.

Thanks,
J

[vad34]

Hi
Here is the link to diag file:
https://www.dropbox.com/s/5eoa6k76ensc6iv/diag-splunk-102-2016-01-06_16-59-07.tar.gz?dl=0
Tnx in advance!

[javiergn]

Hi, the file seems corrupted:

tar -xvzf diag-splunk-102-2016-01-06_16-59-07.tar.gz 
tar: Error opening archive: gzip decompression failed

ls -lah diag-splunk-102-2016-01-06_16-59-07.tar.gz 
-rw-r-----@ 1 javier  1160039685    39M  7 Jan 15:26 diag-splunk-102-2016-01-06_16-59-07.tar.gz

[vad34]

Hello
I copied relevant folders & log and diag files,
Should be ok now
Here is the link below:
https://www.dropbox.com/s/y6mqx2s5b058jod/splunk%20diag.rar?dl=0

[javiergn]

Hi,

After reviewing your configuration I came up with following system/local/inputs.conf file that should work for you. It's a combination of Splunk_TA_Windows and your initial requirements. Replace your current file with the following content and restart Splunk afterwards:

[default]
host = splunk-102

[splunktcp://9997]

[WinEventLog://Application]
disabled = 0
index = wineventlog
renderXml=false

[WinEventLog://Security]
disabled = 0
current_only = 1
evt_resolve_ad_obj = 1
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist3 = EventCode="4726" 
index = wineventlog
renderXml=false

[WinEventLog://System]
disabled = 0
whitelist = 7036-7037
index = wineventlog
renderXml=false

Then run the following search:

index=wineventlog EventCode=4726

It should be empty.

NOTE I HAVE ASSUMED THERE IS AN INDEX CALLED wineventlog. IF THAT'S NOT THE CASE SIMPLY EDIT YOUR inputs.conf, MODIFY THE INDEX NAME THERE, RESTART AND THEN MODIFY THE SEARCH TO REFER TO THE RELEVANT INDEX.

Hope that helps.
If not I would suggest opening a support call with Splunk as they will be able to replicate your environment in a much more accurate way than what I did.

Thanks,
J

[vad34]

Hey,

We found the issue and here is the solution. On the Windows server, you need to configure this:

C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf

[WinEventLog://System]
disabled = 0
blacklist1 = EventCode="7036" 

and it works. I stopped getting events with code 7036.
Tnx for your help from the beginning !!!

[javiergn]

Take a look at the whitelist and blacklist attributes:

http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/MonitorWindowsdata

You can even use advanced filtering now (see advanced filtering section)

For question number 2, the way you install a HF is exactly the same as any other Splunk Entrerprise instance. Just configure inputs and outputs accordingly and it'll behave like one. Take a look at this:

http://docs.splunk.com/Documentation/Splunk/6.2.3/Forwarding/Deployaheavyforwarder

Hope that helps.

Thanks,
J

[vad34]

Ok i tried and still able to see the event id 4726 😞

[javiergn]

Hi, I can't see your latest comment but I got the email notification.
There's probably too many nested comments above so I'll answer here.

Let's try a whitelist approach for your Security stanza:

[WinEventLog://Security]
disabled = 0
current_only = 1
whitelist = 1100-1108,4608-4725,4727-6416
# More details here: https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/Default.aspx

[javiergn]

Hi, I can't see your latest message again because of the nested comment but I got the email notification.

What do you mean by?

i removed the server class-system & security and now unable to see system & security events.

Your system/local/inputs.conf file should be something like:

[default]
host = splunk-102
[splunktcp://9997]

[WinEventLog://System]
disabled = 0
whitelist = 7036-7037

[WinEventLog://Security]
disabled = 0
blacklist = 4726

Can you edit the system/local/inputs.conf file to include that, restart splunk and then paste the relevant btool output here to see what's going on?

Thanks,
J

[vad34]

Hello,
I mean server classes (define windows event logs source like : Security,System,Application)
did it ,
here is new output
https://www.dropbox.com/s/l4hd34w768ipqol/new%20btool%20output.txt?dl=0

[javiergn]

Hi, these are the relevant lines in the output file:

/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinEventLog://Security]
/opt/splunk/etc/system/local/inputs.conf blacklist = 4726
/opt/splunk/etc/system/local/inputs.conf disabled = 0

/opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf [WinEventLog://System]
/opt/splunk/etc/system/local/inputs.conf disabled = 0
/opt/splunk/etc/system/local/inputs.conf whitelist = 7036-7037

I can't see anything wrong in there so the last thing I would try is to disable (move) the whole "/opt/splunk/etc/apps/Splunk_TA_windows" directory to ensure your event log reading is only configured once.

You need to define your stanzas the way I mentioned above and your inputs.conf should look like:

[default]
host = splunk-102

[splunktcp://9997]

[WinEventLog://System]
disabled = 0
whitelist = 7036-7037

[WinEventLog://Security]
disabled = 0
blacklist = 4726

[vad34]

Hey, when i moved the TA directory i was unable to get any events so i moved it back
i tried to config the inputs.conf in TA dir but still not working.
what about set another splunk instance and configure heavy forwarder?

[javiergn]

Hi, installing a new forwarder from scratch is a good idea.

But install a Universal Forwarder. You don't need a heavy forwarder if you just want to collect logs. The UF is lighter and easier to manage.

[vad34]

Hello
I installed new splunk and new UF
here is the btool output:
[root@splunk-102 ~]# /opt/splunk/bin/splunk btool check --debug
Checking: /opt/splunk/etc/users/admin/search/local/ui-tour.conf
Checking: /opt/splunk/etc/users/admin/user-prefs/local/user-prefs.conf
Checking: /opt/splunk/etc/apps/learned/local/props.conf
Checking: /opt/splunk/etc/apps/search/local/indexes.conf
Checking: /opt/splunk/etc/apps/search/local/serverclass.conf
Checking: /opt/splunk/etc/apps/SplunkForwarder/default/app.conf
Checking: /opt/splunk/etc/apps/SplunkForwarder/default/default-mode.conf
Checking: /opt/splunk/etc/apps/SplunkForwarder/default/outputs.conf
Checking: /opt/splunk/etc/apps/SplunkForwarder/default/server.conf
Checking: /opt/splunk/etc/apps/SplunkLightForwarder/default/app.conf
Checking: /opt/splunk/etc/apps/SplunkLightForwarder/default/default-mode.conf
Checking: /opt/splunk/etc/apps/SplunkLightForwarder/default/indexes.conf
Checking: /opt/splunk/etc/apps/SplunkLightForwarder/default/inputs.conf
Checking: /opt/splunk/etc/apps/SplunkLightForwarder/default/limits.conf
Checking: /opt/splunk/etc/apps/SplunkLightForwarder/default/outputs.conf
Checking: /opt/splunk/etc/apps/SplunkLightForwarder/default/props.conf
Checking: /opt/splunk/etc/apps/SplunkLightForwarder/default/server.conf
Checking: /opt/splunk/etc/apps/SplunkLightForwarder/default/web.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/app.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/authorize.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/distsearch.conf
No spec file for: /opt/splunk/etc/apps/Splunk_TA_windows/default/eventgen.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/eventtypes.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/indexes.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/inputs.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/macros.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/props.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/tags.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/transforms.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/web.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/wmi.conf
Checking: /opt/splunk/etc/apps/Splunk_TA_windows/default/workflow_actions.conf
Checking: /opt/splunk/etc/apps/alert_webhook/default/alert_actions.conf
Checking: /opt/splunk/etc/apps/alert_webhook/default/app.conf
Checking: /opt/splunk/etc/apps/alert_webhook/default/restmap.conf
Checking: /opt/splunk/etc/apps/appsbrowser/default/app.conf
Checking: /opt/splunk/etc/apps/gettingstarted/default/app.conf
Checking: /opt/splunk/etc/apps/introspection_generator_addon/default/app.conf
Checking: /opt/splunk/etc/apps/introspection_generator_addon/default/inputs.conf
Checking: /opt/splunk/etc/apps/introspection_generator_addon/default/server.conf
Checking: /opt/splunk/etc/apps/launcher/default/app.conf
Checking: /opt/splunk/etc/apps/launcher/default/launcher.conf
Checking: /opt/splunk/etc/apps/legacy/default/app.conf
Checking: /opt/splunk/etc/apps/legacy/default/props.conf
Checking: /opt/splunk/etc/apps/sample_app/default/app.conf
Checking: /opt/splunk/etc/apps/sample_app/default/indexes.conf
Checking: /opt/splunk/etc/apps/sample_app/default/inputs.conf
Checking: /opt/splunk/etc/apps/sample_app/default/props.conf
Checking: /opt/splunk/etc/apps/search/default/app.conf
Checking: /opt/splunk/etc/apps/search/default/commands.conf
Checking: /opt/splunk/etc/apps/search/default/event_renderers.conf
Checking: /opt/splunk/etc/apps/search/default/macros.conf
Checking: /opt/splunk/etc/apps/search/default/props.conf
Checking: /opt/splunk/etc/apps/search/default/restmap.conf
Checking: /opt/splunk/etc/apps/search/default/savedsearches.conf
Checking: /opt/splunk/etc/apps/search/default/transforms.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/app.conf
Invalid key in stanza [ui] in /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/app.conf, line 14: attribution_link (value: app.attributions).
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/authorize.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/collections.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/distsearch.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/eventtypes.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/indexes.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/macros.conf
No spec file for: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/msftapps_winfra_setup.conf
No spec file for: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/paletteinputs.conf
No spec file for: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/palettepalettes.conf
No spec file for: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/palettepanels.conf
No spec file for: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/palettesearches.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/props.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/restmap.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/savedsearches.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/server.conf
No spec file for: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/splunk_msftapp.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/tags.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/transforms.conf
Checking: /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/web.conf
Checking: /opt/splunk/etc/apps/splunk_archiver/default/app.conf
Checking: /opt/splunk/etc/apps/splunk_archiver/default/commands.conf
Checking: /opt/splunk/etc/apps/splunk_archiver/default/distsearch.conf
Checking: /opt/splunk/etc/apps/splunk_archiver/default/props.conf
Checking: /opt/splunk/etc/apps/splunk_archiver/default/savedsearches.conf
Checking: /opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf
Checking: /opt/splunk/etc/apps/splunk_management_console/default/app.conf
Checking: /opt/splunk/etc/apps/splunk_management_console/default/distsearch.conf
Checking: /opt/splunk/etc/apps/splunk_management_console/default/macros.conf
Checking: /opt/splunk/etc/apps/splunk_management_console/default/props.conf
Checking: /opt/splunk/etc/apps/splunk_management_console/default/savedsearches.conf
Checking: /opt/splunk/etc/apps/splunk_management_console/default/splunk_management_console_assets.conf
Checking: /opt/splunk/etc/apps/splunk_management_console/default/transforms.conf
Checking: /opt/splunk/etc/apps/user-prefs/default/app.conf
Checking: /opt/splunk/etc/apps/user-prefs/default/user-prefs.conf
Checking: /opt/splunk/etc/system/default/alert_actions.conf
Checking: /opt/splunk/etc/system/default/app.conf
Checking: /opt/splunk/etc/system/default/audit.conf
Checking: /opt/splunk/etc/system/default/authentication.conf
Checking: /opt/splunk/etc/system/default/authorize.conf
Checking: /opt/splunk/etc/system/default/collections.conf
Checking: /opt/splunk/etc/system/default/commands.conf
No spec file for: /opt/splunk/etc/system/default/conf.conf
Checking: /opt/splunk/etc/system/default/crawl.conf
Checking: /opt/splunk/etc/system/default/datamodels.conf
Checking: /opt/splunk/etc/system/default/datatypesbnf.conf
Checking: /opt/splunk/etc/system/default/default-mode.conf
Checking: /opt/splunk/etc/system/default/distsearch.conf
Checking: /opt/splunk/etc/system/default/event_renderers.conf
Checking: /opt/splunk/etc/system/default/eventdiscoverer.conf
Checking: /opt/splunk/etc/system/default/eventtypes.conf
Checking: /opt/splunk/etc/system/default/fields.conf
Checking: /opt/splunk/etc/system/default/indexes.conf
Checking: /opt/splunk/etc/system/default/inputs.conf
Checking: /opt/splunk/etc/system/default/limits.conf
Checking: /opt/splunk/etc/system/default/multikv.conf
Checking: /opt/splunk/etc/system/default/outputs.conf
Checking: /opt/splunk/etc/system/default/pdf_server.conf
No spec file for: /opt/splunk/etc/system/default/prefs.conf
Checking: /opt/splunk/etc/system/default/procmon-filters.conf
Checking: /opt/splunk/etc/system/default/props.conf
Checking: /opt/splunk/etc/system/default/restmap.conf
Checking: /opt/splunk/etc/system/default/savedsearches.conf
Checking: /opt/splunk/etc/system/default/segmenters.conf
Checking: /opt/splunk/etc/system/default/server.conf
Checking: /opt/splunk/etc/system/default/serverclass.conf
Checking: /opt/splunk/etc/system/default/source-classifier.conf
Checking: /opt/splunk/etc/system/default/times.conf
Checking: /opt/splunk/etc/system/default/transactiontypes.conf
Checking: /opt/splunk/etc/system/default/transforms.conf
Checking: /opt/splunk/etc/system/default/ui-prefs.conf
Checking: /opt/splunk/etc/system/default/ui-tour.conf
Checking: /opt/splunk/etc/system/default/viewstates.conf
Checking: /opt/splunk/etc/system/default/web.conf
Checking: /opt/splunk/etc/system/default/workflow_actions.conf
Checking: /opt/splunk/etc/system/local/inputs.conf
No spec file for: /opt/splunk/etc/system/local/migration.conf
Checking: /opt/splunk/etc/system/local/server.conf

[vad34]

I still get this error not sure if it can affect the event log filtering
Invalid key in stanza [ui] in /opt/splunk/etc/apps/splunk_app_windows_infrastructure/default/app.conf, line 14: attribution_link (value: app.attributions).

[javiergn]

It shouldn't. Is the new UF working as expected? If not, can you paste your inputs.conf?

[vad34]

Hi
I tried your last suggestion with the white list, with no luck..