How to Eliminate some values in specific field?

[jeff1]

I am trying to write an add-on to eliminate some values in a specific field by plugging in a file containing props.conf and transforms.conf into the splunk/etc/apps directory but failed to get any result. 

Please give me some advice, my configuration files are as follows:

props.conf: 

[source: path to the log file]

TRANSFORMS-elim= elimValue

transforms.conf:

[elimValue]
REGEX=^(type=[A-T]+)
DEST_KEY=queue
FORMAT=nullQueue

Grateful for any help, thanks.

[PickleRick]

If you want to anonymize some part of the event, it's an index-time operation. Where did you put this app? And what is your event's path to indexers.

And did you verify the effective config with btool?

[jeff1]

hi @PickleRick ,

I put those configuration in a file called local and place it under /opt/splunk/etc/apps directory.  For the second question, I am sorry I did not understand it.

[PickleRick]

The typical splunk environment consists of indexer(s) and search head(s) (or a single all-in-one installation in case of small setups), optional heavy forwardes and universal forwarders.

I suppose you have a simple setup consisting of an all-in-one server either reading files directly from disk or using universal forwarders to ingest data. In this case your settings should be put on the all-in-one server itself. (if you had a heavy forwarder, they'd need to be put there).

And with the "where" in terms of directory - you skipped one component of the path.

If you're creating an app called ABC, you put it into /opt/splunk/etc/apps/ABC and there you create directories like /opt/splunk/etc/apps/ABC/default, /opt/splunk/etc/apps/ABC/local, /opt/splunk/etc/apps/ABC/lookups and so on.

So if you put your conf file(s) in /opt/splunk/etc/apps/local, splunk will ignore them.

Oh, and remember about the config file precedence https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Wheretofindtheconfigurationfiles

[gcusello]

Hi @jeff1,

your request isn't so clear for me, are you asking of:

• discard some full events before indexing,
• delete a part of events

?

In the first case you have to follow the steps at https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad

In the second case, you have to follow the steps at https://docs.splunk.com/Documentation/Splunk/8.2.5/Data/Anonymizedata

Ciao.

Giuseppe

[jeff1]

hi @gcusello ,

I want to discard events that contain specific values in a field in index-time.

[gcusello]

Hi @jeff1,

in this case, you have to follow the steps in the first link, if few words, you have to create on your Indexers or (when present) on your Heavy Forwarders the files you shared.

If they don't work, please share a sample of your logs (someone to delete and someone to not delete) to check your regex,

then, where do you located props.conf and transforms.conf?

Ciao.

Giuseppe

[jeff1]

hi @gcusello ,

I do have both indexer and heavy forwarder in my testing environment, may I ask if you mean all index-time operation conf. file is better put in Indexer or Heavy Forwarder? As both of my conf. files mentioned above are put under the path of /etc/apps in Search Head.

Here are some of my logs:

type=USER_END msg=audit(xxxxxxxxxxxx) source = /var/log/audit/audit.log    <- log to keep

type=CRED_DISP msg=audit(xxxxxxxxxx) source = /var/log/audit/audit.log  <-log to eliminate

I am trying to eliminate all logs other than (type=USER_*), where [type] is the interesting field.

Sorry for the messy elaboration. 

[PickleRick]

The index-time operations are done on the first "heavy" (based on a full splunk enterprise installaton, not on universal forwarder package) component as the event traverses your network to the indexer.

So if you have simple UF->indexer setup, you need to put your index-time operations configuration on the indexer. If you have more complicated setup, like - for example - UF -> UF -> HF -> indexer, you need to put it on HF.

[gcusello]

Hi @jeff1,

about the location of the conf files, if you're sure that all your logs to filter pass through the HFs, you have to put them on HFs, otherwise you have to put them both on IDXs and HFs.

if you want to take only the events where type=USER_END you have to use the following props.conf and transforms.conf:

props.conf:

[source: path to the log file]
TRANSFORMS-set = setnull,setparsing

transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^type\=USER_END
DEST_KEY = queue
FORMAT = indexQueue

If instead you want to discard only type=CRED_DISP you have to use the following props.conf and transforms.conf:

props.conf:

[source: path to the log file]
TRANSFORMS-null= setnull

transforms.conf:

[setnull]
REGEX = ^type\=CRED_DISP
DEST_KEY = queue
FORMAT = nullQueue

Ciao.

Giuseppe