topic Re: Eliminate some values in specific field in All Apps and Add-ons

How to Eliminate some values in specific field?

jeff1 — Wed, 06 Apr 2022 15:28:39 GMT

I am trying to write an add-on to eliminate some values in a specific field by plugging in a file containing props.conf and transforms.conf into the splunk/etc/apps directory but failed to get any result.

Please give me some advice, my configuration files are as follows:

props.conf:

[source: path to the log file]

TRANSFORMS-elim= elimValue

transforms.conf:

[elimValue]
REGEX=^(type=[A-T]+)
DEST_KEY=queue
FORMAT=nullQueue

Grateful for any help, thanks.

Re: Eliminate some values in specific field

gcusello — Wed, 06 Apr 2022 08:06:02 GMT

Hi @jeff1,

your request isn't so clear for me, are you asking of:

discard some full events before indexing,
delete a part of events

In the first case you have to follow the steps at https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad

In the second case, you have to follow the steps at https://docs.splunk.com/Documentation/Splunk/8.2.5/Data/Anonymizedata

Ciao.

Giuseppe

Re: Eliminate some values in specific field

PickleRick — Wed, 06 Apr 2022 08:11:38 GMT

If you want to anonymize some part of the event, it's an index-time operation. Where did you put this app? And what is your event's path to indexers.

And did you verify the effective config with btool?

Re: Eliminate some values in specific field

jeff1 — Wed, 06 Apr 2022 08:12:55 GMT

hi @gcusello ,

I want to discard events that contain specific values in a field in index-time.

Re: Eliminate some values in specific field

gcusello — Wed, 06 Apr 2022 08:24:42 GMT

Hi @jeff1,

in this case, you have to follow the steps in the first link, if few words, you have to create on your Indexers or (when present) on your Heavy Forwarders the files you shared.

If they don't work, please share a sample of your logs (someone to delete and someone to not delete) to check your regex,

then, where do you located props.conf and transforms.conf?

Ciao.

Giuseppe

Re: Eliminate some values in specific field

jeff1 — Wed, 06 Apr 2022 08:30:22 GMT

hi @PickleRick ,

I put those configuration in a file called local and place it under /opt/splunk/etc/apps directory. For the second question, I am sorry I did not understand it.

Re: Eliminate some values in specific field

jeff1 — Wed, 06 Apr 2022 08:43:53 GMT

hi @gcusello ,

I do have both indexer and heavy forwarder in my testing environment, may I ask if you mean all index-time operation conf. file is better put in Indexer or Heavy Forwarder? As both of my conf. files mentioned above are put under the path of /etc/apps in Search Head.

Here are some of my logs:

type=USER_END msg=audit(xxxxxxxxxxxx) source = /var/log/audit/audit.log <- log to keep

type=CRED_DISP msg=audit(xxxxxxxxxx) source = /var/log/audit/audit.log <-log to eliminate

I am trying to eliminate all logs other than (type=USER_*), where [type] is the interesting field.

Sorry for the messy elaboration.

Re: Eliminate some values in specific field

PickleRick — Wed, 06 Apr 2022 08:48:50 GMT

The typical splunk environment consists of indexer(s) and search head(s) (or a single all-in-one installation in case of small setups), optional heavy forwardes and universal forwarders.

I suppose you have a simple setup consisting of an all-in-one server either reading files directly from disk or using universal forwarders to ingest data. In this case your settings should be put on the all-in-one server itself. (if you had a heavy forwarder, they'd need to be put there).

And with the "where" in terms of directory - you skipped one component of the path.

If you're creating an app called ABC, you put it into /opt/splunk/etc/apps/ABC and there you create directories like /opt/splunk/etc/apps/ABC/default, /opt/splunk/etc/apps/ABC/local, /opt/splunk/etc/apps/ABC/lookups and so on.

So if you put your conf file(s) in /opt/splunk/etc/apps/local, splunk will ignore them.

Oh, and remember about the config file precedence https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Wheretofindtheconfigurationfiles

Re: Eliminate some values in specific field

gcusello — Wed, 06 Apr 2022 08:55:34 GMT

Hi @jeff1,

about the location of the conf files, if you're sure that all your logs to filter pass through the HFs, you have to put them on HFs, otherwise you have to put them both on IDXs and HFs.

if you want to take only the events where type=USER_END you have to use the following props.conf and transforms.conf:

props.conf:

[source: path to the log file] TRANSFORMS-set = setnull,setparsing

transforms.conf:

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = ^type\=USER_END DEST_KEY = queue FORMAT = indexQueue

If instead you want to discard only type=CRED_DISP you have to use the following props.conf and transforms.conf:

props.conf:

[source: path to the log file] TRANSFORMS-null= setnull

transforms.conf:

[setnull] REGEX = ^type\=CRED_DISP DEST_KEY = queue FORMAT = nullQueue

Ciao.

Giuseppe

Re: Eliminate some values in specific field

PickleRick — Wed, 06 Apr 2022 09:08:02 GMT

The index-time operations are done on the first "heavy" (based on a full splunk enterprise installaton, not on universal forwarder package) component as the event traverses your network to the indexer.

So if you have simple UF->indexer setup, you need to put your index-time operations configuration on the indexer. If you have more complicated setup, like - for example - UF -> UF -> HF -> indexer, you need to put it on HF.