topic Re: Windows Event filtering/truncation at IDX and HF in Getting Data In

Windows Event filtering/truncation at IDX and HF

adalbor — Wed, 26 Jun 2019 14:23:21 GMT

Hey All,

I currently have the below props and transforms on my HF's and my IDX's

This works on dropping that extra text in the Windows events on most events but not all.

Support is telling me that putting this on the HF's will not work and that I should try to point all my UF's straight to my IDX's to achieve this. Due to networking limitations we are using HF's as we have differing network segments.

Has anyone else run into this issue or can speak to what support is telling me?

Props.conf

[WinEventLog:Security]
TRANSFORMS-shorten = shorten4624,shorten4634,shorten4648,shorten4769,shorten4771,shorten4688,shorten4625

Transforms.conf

[shorten4624]
REGEX = (?ms)(.EventCode=4624.)This event is generated when a logon session
DEST_KEY = _raw
FORMAT = $1

[shorten4634]
REGEX = (?ms)(.EventCode=4634.)This event is generated when a logon session is destroyed
DEST_KEY = _raw
FORMAT = $1

[shorten4648]
REGEX = (?ms)(.EventCode=4648.)This event is generated when a process attempts
DEST_KEY = _raw
FORMAT = $1

[shorten4769]
REGEX = (?ms)(.EventCode=4769.)This event is generated every time access is requested to a resource
DEST_KEY = _raw
FORMAT = $1

[shorten4771]
REGEX = (?ms)(.EventCode=4771.)Certificate information is only provided
DEST_KEY = _raw
FORMAT = $1

[shorten4688]
REGEX = (?ms)(.EventCode=4688.)Token Elevation Type indicates the type of token that was assigned to the new process
DEST_KEY = _raw
FORMAT = $1

[shorten4625]
REGEX = (?ms)(.EventCode=4625.)This event is generated when a logon request fails
DEST_KEY = _raw
FORMAT = $1

Re: Windows Event filtering/truncation at IDX and HF

jnudell_2 — Wed, 26 Jun 2019 16:05:22 GMT

Hi @adalbor ,

As an alternative, you can install UFs instead of HFs on your intermediate forwarder layer, and put this configuration on your indexers.

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Wed, 26 Jun 2019 17:18:51 GMT

So we have dedicated servers for our HF's. Are you saying have another box with a UF installed just forwarding these events?

Re: Windows Event filtering/truncation at IDX and HF

amitm05 — Wed, 26 Jun 2019 19:20:07 GMT

Hi @adalbor
Can you mention how is the log collection currently done ?
Is it UF -> HF -> Idx
OR HF -> Idx

If its the first case, you can just remove the HF layer for these windows logs and forward your logs directly UF -> Idx.
If its the 2nd case, I think you need to define these rules only on Indexer and not on your HF.

Let me know, if there is a gap in understanding yet.
Thanks

Re: Windows Event filtering/truncation at IDX and HF

jnudell_2 — Wed, 26 Jun 2019 19:28:25 GMT

I'm saying to install the UF on those dedicated servers instead of the full Splunk instance, and they act as your gateway to the Splunk indexers. This is in consideration of an egress point from different networks to Splunk. The only reason you would need an HF is if you have modular inputs (HEC, DB Connect, or anything that requires the python stack).

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Wed, 26 Jun 2019 19:29:57 GMT

@amitm05
Our internal UF's are going to HF's then to IDX's
Our external UF's (DMZ) are going straight to the IDX's

I have the props and transforms on both the HF's and IDX's (in an attempt to get it work) and for some reason it works on about 80% of events but NOT all events.

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Wed, 26 Jun 2019 19:30:55 GMT

Understood. The HF's are running DB Connect, SNMP, O365 inputs, etc so there is def a need for them.

Re: Windows Event filtering/truncation at IDX and HF

jnudell_2 — Wed, 26 Jun 2019 19:33:33 GMT

It sounds like you would need to do some troubleshooting to investigate where the non-transformed events are coming from. Are they coming from a particular subnet, the DMZ only, only events that bypass the HF layer, etc? Once you can figure out where they come from you can start to look at what's different for the path and configuration for that data.

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Wed, 26 Jun 2019 19:38:57 GMT

Thats the problem I presented to support. Asked them if it was possible to have a search that would return events that weren't being truncated properly but they had no clue.

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Wed, 26 Jun 2019 19:47:17 GMT

Just to add, did a quick search. Looked for EventCode=4624
One system with UF installed locally sending to HF then IDX, truncated events properly.
One system sending events via WEF to a WEC with UF installed locally then sending to HF, didnt truncate events properly

Re: Windows Event filtering/truncation at IDX and HF

amitm05 — Thu, 27 Jun 2019 18:49:37 GMT

I'd take that the logs that are getting through UF to Idx directly are fine.
The problem is only with the ones that are though an HF. Have we figured this out ?

I'd discourage to put the transformations on HF if we do not intend to use them there but only at Idx. Basically what we want to do is to have HF only recieve and forward (without any processing) which should be equivalent to what a UF in place would do.

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Thu, 27 Jun 2019 19:13:59 GMT

Some systems sending through HF are truncating with no issues while some aren't.
As far as I can tell the UF > IDX ones are truncating with no issues but its also difficult to get a correct count of hosts working/not working

Re: Windows Event filtering/truncation at IDX and HF

amitm05 — Fri, 28 Jun 2019 08:32:58 GMT

We do not want the count but to understand that if the error lies in how the data is being forwarded and intermediate parsing. Or is it at the source ?

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Fri, 28 Jun 2019 12:35:12 GMT

I dont know where the true issue lies but if I can get a list of hosts that are not having their events truncated then I can see where the majority of them are sending their events through to help trace the issue

Re: Windows Event filtering/truncation at IDX and HF

FrankVl — Fri, 28 Jun 2019 13:08:07 GMT

In theory it should work as long as you ensure this config is present on each full splunk enterprise instance that first touches this data. So with having it on those intermediate HFs and on your Indexers, it should indeed work. I assume you have made sure the config is really there on all relevant HFs and the failed items are not due to regex mismatch?

PS: I've seen very big performance hits from doing exactly this windows event cleaning using TRANSFORMS. Personally I prefer implementing this using SEDCMD in props.conf, like this:

SEDCMD-strip_msg_txt = s/(?ms)[\r\n]+This event is generated.*//

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Fri, 28 Jun 2019 14:40:23 GMT

Yes I have made sure it was applied on all of our HF's and IDX's with an identical config.
I was seeing exact same event codes with some being truncated and some without.

You applied this at your IDX layer in /etc/system/local? I am guessing this has the same effect to ensure that data is not indexed?

Re: Windows Event filtering/truncation at IDX and HF

FrankVl — Fri, 28 Jun 2019 14:55:26 GMT

Usually not in system local, usually through a small TA, at indexers or HF intermediates, seen a few different environments over the past years, but indeed, this also applies before indexing to reduce license usage. In one of those environments (6.5.3 I believe) someone else had done it with transforms like you had and that brought the HFs to their knees when volume increased. Switching to SED solved that.

Doesn't explain / solve your issue, but still thought it might be useful to know as an alternative 🙂

Could be interesting to see if you can trace the issue back to a certain indexer (see splunk_server field) or HF (this will be a bit more tricky, but perhaps you can see from the host value what HF it should pass through)?

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Fri, 28 Jun 2019 17:11:11 GMT

I tried at one time pulling the SED stuff from the Splunk TA for windows that supposedly did the same thing but didn't have any luck with it.

I am working on trying to narrow down at what layer the issues are occurring. I am definitely not opposed to using SED instead, just wanted to make sure what I hit more than one event. Wanted to make sure the top windows events in our environment had that extra text removed as it would make the most license savings.

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Fri, 28 Jun 2019 18:22:06 GMT

I went ahead and created an app with that setting in a props and pushed to all of my IDX's.
Will report back after I have had time for it to bake and validate.

Re: Windows Event filtering/truncation at IDX and HF

adalbor — Wed, 30 Sep 2020 01:16:45 GMT

Well after days of banging my head up against a wall I finally discovered the issue.
Because we collects logs from Forwarded Events log locations and Security log locations we have two main windows sourcetypes.
Our event truncation props.conf only covered WinEventLog:Security.
I had no idea we had another sourcetype as the Splunk_TA_Windows did a rename to normalize all of the sourcetypes to the same at search time. So when I would run a search and see some events truncated and some not I had no clue.
Once I added the Forwarded Events sourcetype to my props (deployed via a custom app) everything worked as it should!

Below is what I used as reference:

[WinEventLog:Security]
SEDCMD-shorten1=s/(?s)This event is generated .+$//
SEDCMD-shorten2=s/(?s)Certificate information is only provided .+$//
SEDCMD-shorten3=s/(?s)Token Elevation Type indicates the type of token .+$//

[WinEventLog]
SEDCMD-shorten4=s/(?s)This event is generated .+$//
SEDCMD-shorten5=s/(?s)Certificate information is only provided .+$//
SEDCMD-shorten6=s/(?s)Token Elevation Type indicates the type of token .+$//

[WinEventLog:ForwardedEvents]
SEDCMD-shorten7=s/(?s)This event is generated .+$//
SEDCMD-shorten8=s/(?s)Certificate information is only provided .+$//
SEDCMD-shorten9=s/(?s)Token Elevation Type indicates the type of token .+$//

[source::WinEventLog:ForwardedEvents]
SEDCMD-shorten10=s/(?s)This event is generated .+$//
SEDCMD-shorten11=s/(?s)Certificate information is only provided .+$//
SEDCMD-shorten12=s/(?s)Token Elevation Type indicates the type of token .+$//