topic Re: Trying to filter ESXi before being indexed in Getting Data In

Trying to filter ESXi before being indexed

snix — Mon, 15 Jan 2018 17:14:41 GMT

Not sure if this is possible on a single server instance of a Splunk setup but I have all my ESXi logs forwarding to my Splunk server over TCP:1514. I did some digging and found references to the props.conf file and adding a regex filter there. So I did some digging and found multiple copies of this config file but I think (and tell me if I am wrong here) that I need to modify the copy found under:
\etc\system\local

If that is the case I just need some guidance no how to filter out everything but logs that contain the string "dfwpktlogs"

I am trying to filter out the rest of the logs as ESXi is very chatty and it eats into the license and I have to set the index it feeds into to only keep logs for a month because it just fills up so fast.

Re: Trying to filter ESXi before being indexed

mayurr98 — Mon, 15 Jan 2018 18:09:56 GMT

hey

So you want to keep specific event and discard the rest.
follow this steps to do that : same is written in the doc as well
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

step 1: Edit props.conf and add the following:you will do this on /local/props.conf of the same path i.e. /opt/splunk/etc/app/<appname>/local

[<specify_sourcetype_name>]
TRANSFORMS-set= setnull,setparsing

step 2: Edit transforms.conf and add the following:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = dfwpktlogs
DEST_KEY = queue
FORMAT = indexQueue

step 3: Restart Splunk Enterprise.

Also, now you want to set the retention period of 1 month i.e. 30 days

so find that index in mostly in /opt/splunk/etc/<appname>/default/indexes.conf and copy the stanza in local/indexes.conf

and add this attribute to that stanza

frozenTimePeriodInSecs = 2592000

Find more information in indexes.conf and props.conf

Let me know if it helps !

Re: Trying to filter ESXi before being indexed

snix — Mon, 15 Jan 2018 23:33:57 GMT

@mayurr98, thank you for the detailed post. I am still a Splunk novice so I just need to clarify a couple things before I make the change in production.

The proper path for the props.conf file if I am not using a specific app for our ESXi logs will be
/opt/splunk/etc/app/search/local?

Also according to the link, you provided it mentions putting the setting you provided at the top of the prop.config file. Just wanted to verify the location where I put the settings in the file matters.

Re: Trying to filter ESXi before being indexed

snix — Tue, 16 Jan 2018 00:22:44 GMT

Okay just testing the suggested settings and just made sure to put the new settings at the top of the props.config file and did a restart and it worked!

Thank you for your help!

Re: Trying to filter ESXi before being indexed

mayurr98 — Tue, 16 Jan 2018 08:18:59 GMT

yes you are right./opt/splunk/etc/system/local/ defines the global path. you should make changes to /app//local is a best practice.