topic Re: Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing? in Getting Data In

Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

Magnus_001 — Tue, 21 Apr 2015 15:55:14 GMT

I am a using a Universal Forwarder on my domain controller to forward security events to a Splunk indexer and would like to filter out the static text "This event is generated..." in the security events to reduce the data consumption by the indexer. I know this can be done with a Heavy Forwarder at the source, but can I do this at the Indexer with the following entries in the props.conf and transforms.conf? Will it reduce my data consumption/license or is it too late by the time it reaches my indexer? Thanks!

Source: Windows Domain Controller (Universal Forwarder)

Splunk Indexer v6.1.x:

Props.conf

[WinEventLog:Security]
TRANSFORMS_ShortenMsg=ShortenSecMsg

Transforms.conf

[ShortenSecMsg]
REGEX=(?msi)(.*)This event is generated
DEST_KEY=_raw
FORMAT=$1

Re: Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

fdi01 — Tue, 21 Apr 2015 16:10:53 GMT

use a heavy forwarder to do it .
because Universal Forwarder can not parse data .
http://docs.splunk.com/Splexicon:Heavyforwarder

Re: Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

martin_mueller — Tue, 21 Apr 2015 22:35:41 GMT

Keep using a Universal Forwarder and check out the settings for WinEventLog type inputs at http://docs.splunk.com/Documentation/Splunk/6.2.2/admin/inputsconf - I'm no Windows expert, but you might just be looking for suppress_text = 1.

Re: Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

schose — Wed, 22 Apr 2015 07:39:10 GMT

Hi,

You can do it exactly as you mentioned. The props.conf and transforms.conf will be used on the indexer when "cooking" the data. So it's not to late to delete some data there before indexing. License meter is used after cooking the data. In this scenario a UF on the DC will be enough.

you will just need to define your the same sourcetype in inputs.conf on the UF as as you do in props.conf on the indexer. make sure all your props.conf are the same on all indexers.

Regards,

Andreas

Re: Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

aweitzman — Wed, 22 Apr 2015 12:13:12 GMT

To add to @schose's answer, your idea as posted is basically correct. However, since you are replacing _raw, you need to have a regex capture group for the content after the string you want to filter out, and include that in your format string, something like:

 [ShortenSecMsg]
 REGEX=^(.*)This event is generated(.*)$
 DEST_KEY=_raw
 FORMAT=$1$2

Re: Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

martin_mueller — Wed, 22 Apr 2015 12:35:10 GMT

Replacing _raw isn't necessary, you can tell the UF to not even include that data.

Re: Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

Magnus_001 — Fri, 24 Apr 2015 18:56:55 GMT

Thank you all. I was able to filter out the unwanted static text with those changes to the Props.conf and Transforms.conf on the Indexer since I am using a UF on my DCs.

Re: Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

martin_mueller — Fri, 24 Apr 2015 23:02:58 GMT

Did you try modifying inputs.conf on the UF to not include the text in the first place?

Re: Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

Magnus_001 — Mon, 28 Sep 2020 19:40:56 GMT

Hi, the suppress_text = 1 works but it also removes many interesting fields we need (Account_Nmae, Account_domain, Logon_type, etc.).

Re: Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

martin_mueller — Sat, 25 Apr 2015 23:30:35 GMT

I see. Based on the docs I'd have expected something different, will ask for docs clarification 🙂