Getting Data In

How to apply filter to Windows logs using props.conf on HF vs indexer

cfloquet
Path Finder

Hello, 

Thanks for taking the time to read/consider my question! 

I'm working on reducing the overhead for Windows Event Logs that we are bringing in via UFs sitting on Windows workstations & servers by trimming some of the redundant text at the end of each log using a props.conf file located within /etc/system/local on each heavy forwarder. 

My understanding was that if you placed a props.conf on the heavy forwarder it would effectively filter out the messages being sent to Splunk cloud, but I'm starting to think that props.conf isn't read until the indexing tier. 

My question is this, if I need to keep indexAndForward=false on my heavy forwarders to avoid the licensing and overhead, how can I apply props.conf to filter events before Splunk cloud? Do I need to submit a support ticket for them to place the props.conf within the cloud-based indexers?

Many thanks in advance

 

0 Karma
1 Solution

PickleRick
Ultra Champion

Ok, first things first - you didn't touch the sourcetype for your eventlog input so you'll have the default one. If I remember correctly it might get overwritten depending on whether you're ingesting events as text or xml. To be honest I have no clue where you got this stanza definition from. For starters just try simple [WinEventLog]

And you theoretically you can put the props "anywhere" on your HF as long as the settings aren't overwritten by something else according to the config file precedence. https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Wheretofindtheconfigurationfiles

Personally, I'd put the sedcmds either into local subdir of windows ta or create a separate app for this. It's,just a matter of how you're managing your configs.

View solution in original post

gcusello
Esteemed Legend

Hi @cfloquet,

only for wineventlog and few other kinds of log, you can easily filter them in inputs.conf blacklisting some EventCodes that you don't consider in your analysises.

For more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Inputsconf#Event_Log_filtering

in few words, in your situation: if you want to filter events with Eventcode =5145 "A network share object was checked to see whether client can be granted desired access", you could use something like this

[WinEventLog://Security]
disabled = 0
start_from = newest
evt_dc_name =
evt_dns_name =
evt_resolve_ad_ds =
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist = EventCode\=5145

 

Ciao.

Giuseppe

cfloquet
Path Finder

Hi @gcusello, thanks for your quick and thorough response. 

To give some context, I am actually already using the blacklist utility within the UF to filter out unwanted codes. What I am looking to do is trim out redundant text at the end of important event codes and messages, so essentially decrease the volume and overhead of these messages without blacklisting them entirely. 

I have done this successfully in a lab environment by placing the following props.conf file within /etc/system/local on the SH/indexer, but haven't tested it when it's on a HF.

[(?::){0}WinEventLog...]
SEDCMD-clean_fluff_from_winsec_events_this_event = s/This event is generated[\S\ s\r\n]+$//g
SEDCMD-clean_fluff_from_winsec_events_token_elevation = s/Token Elevation Type i ndicates[\S\s\r\n]+$//g
SEDCMD-clean_fluff_from_winsec_events_certificate_info = s/Certificate informati on is only provided if[\S\s\r\n]+$//g
SEDCMD-clean_fluff_from_winsec_events_the_subject_fields = s/The subject fields indicate the account[\S\s\r\n]+$//g

 Please advise on whether configs made with props.conf can actually be placed on the HF or if I need to contact Splunk Cloud in order to put it on the cloud-based indexers. 

Thanks

0 Karma

gcusello
Esteemed Legend

Hi @cfloquet,

I prefer to identify non useful EventCodes and completely filter those events, than truncate a part of events but also you approach could be correct.

Anyway, filtering must be performed only on HFs, with the only exception of wineventlogs on inputs.conf and Indexers but only when there isn't any HFs, but, if I correctly understood, you're using Splunk Cloud and all your logs pass through HFs.

Ciao.

Giuseppe

cfloquet
Path Finder

@gcusello thanks again for your quick & thorough response, these will always be rewarded with karma points. 

Unfortunately I'm not sure if we're on the same page, I am currently using blacklists on the UF to filter out noisy or redundant events. However my understanding is that Splunk UF is not able to filter out a section of a log, and instead doesn't send that entire event log if something matches within it to what is on the blacklist. Correct me if I'm wrong. 

What I'm trying to accomplish is essentially when a user logs in (4624), filter out all of the flavor text at the bottom, which is shown below in an example:

Network Information:
	Workstation Name:	EXWKS
	Source Network Address:	123.45.67.89
	Source Port:		0

Detailed Authentication Information:
	Logon Process:		User32 
	Authentication Package:	Negotiate
	Transited Services:	-
	Package Name (NTLM only):	-
	Key Length:		0

This event is generated when a logon session is created. It is generated on the computer that was accessed.

The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).

The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.

The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The impersonation level field indicates the extent to which a process in the logon session can impersonate.

The authentication information fields provide detailed information about this specific logon request.
	- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.
	- Transited services indicate which intermediate services have participated in this logon request.
	- Package name indicates which sub-protocol was used among the NTLM protocols.
	- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

 In other words, how do I trim out all of the text after the "detailed login information" section? I know that this is possible using self-hosted or on-prem indexers and search heads, since I've done it. If someone knows how to accomplish this in Splunk cloud please let me know

0 Karma

PickleRick
Ultra Champion

The UF Windows logs filtering is a feature of one particular input. It doesn't even work with windows event gathered using another method. So UF uses an input which reads from event log and can whitelist/blacklist some events as a whole.

The UF has no event-modifying features.

Props and transforms are applied on the first component in event's path that is based on full install of splunk. If your path is UF->HF->indexers, your index time props and transforms are applied on HF since it's the first "heavy" component which sees the event.

If you have your settings on HF and they don't work, there's something wrong with the configuration. Did you verify it with btool? Did you restart the service on HF?

cfloquet
Path Finder

Hi @PickleRick, thanks for your response it's always appreciated!

To confirm, yes the path of forwarding we are currently using is UF->HF->Indexer (Splunk Cloud)

To give some more details, I am currently using the Splunk TA Add On for Windows, with a modified inputs.conf that I stored within the local directory of the app itself. With all the other options disabled, here is the one enabled input for that app:

[WinEventLog://Security]
disabled = 0
start_from = oldest
index=wineventlog
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
renderXml=false
blacklist = 4689,5158
blacklist1 = EventCode="4688" Message="(?:New Process Name:).+(?:\\Google\\Chrome\\Application\\chrome.exe)|.+(?:\\AppData\\Local\\Microsoft\\Teams\\current\\Teams.exe)|.+(?:C:\\Program Files\\SplunkUniversalForwarder\\bin\\)|.+(?:C:\\Program Files\\SplunkUniversalForwarder\\bin\\splunk.exe)"

 So I have defined no sourcetype, but I have defined an index. This is exactly the same configuration as in my lab environment. On the HF side of things I have also installed the Splunk Add on for Windows, but am storing the props.conf file within /etc/system/local. Does this need to be stored within the Windows Add on app on the heavy forwarder? Here is the exact syntax for props.conf that I am using:

[(?::){0}WinEventLog...]
SEDCMD-clean_fluff_from_winsec_events_this_event = s/This event is generated[\S\s\r\n]+$//g
SEDCMD-clean_fluff_from_winsec_events_token_elevation = s/Token Elevation Type indicates[\S\s\r\n]+$//g
SEDCMD-clean_fluff_from_winsec_events_certificate_info = s/Certificate information is only provided if[\S\s\r\n]+$//g
SEDCMD-clean_fluff_from_winsec_events_the_subject_fields = s/The subject fields indicate the account[\S\s\r\n]+$//g


Many thanks

0 Karma

PickleRick
Ultra Champion

Ok, first things first - you didn't touch the sourcetype for your eventlog input so you'll have the default one. If I remember correctly it might get overwritten depending on whether you're ingesting events as text or xml. To be honest I have no clue where you got this stanza definition from. For starters just try simple [WinEventLog]

And you theoretically you can put the props "anywhere" on your HF as long as the settings aren't overwritten by something else according to the config file precedence. https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Wheretofindtheconfigurationfiles

Personally, I'd put the sedcmds either into local subdir of windows ta or create a separate app for this. It's,just a matter of how you're managing your configs.

cfloquet
Path Finder

@PickleRick as per usual, your suggestion led me directly to the correct answer!

I did not have to change or apply any specific sourcetype, the only change that I made was moving props.conf from /etc/system/local on the HF to the Splunk_TA_Windows/local directory on the HF and restarted Splunk and it worked like a charm. 

Thanks very much for all your help, I will accept your response as the solution! 

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...