All Apps and Add-ons

How to Eliminate some values in specific field?

jeff1
Loves-to-Learn

I am trying to write an add-on to eliminate some values in a specific field by plugging in a file containing props.conf and transforms.conf into the splunk/etc/apps directory but failed to get any result. 

Please give me some advice, my configuration files are as follows:

props.conf: 

[source: path to the log file]

TRANSFORMS-elim= elimValue

transforms.conf:

[elimValue]
REGEX=^(type=[A-T]+)
DEST_KEY=queue
FORMAT=nullQueue

Grateful for any help, thanks.

 

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

If you want to anonymize some part of the event, it's an index-time operation. Where did you put this app? And what is your event's path to indexers.

And did you verify the effective config with btool?

0 Karma

jeff1
Loves-to-Learn

hi @PickleRick ,

I put those configuration in a file called local and place it under /opt/splunk/etc/apps directory.  For the second question, I am sorry I did not understand it.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The typical splunk environment consists of indexer(s) and search head(s) (or a single all-in-one installation in case of small setups), optional heavy forwardes and universal forwarders.

I suppose you have a simple setup consisting of an all-in-one server either reading files directly from disk or using universal forwarders to ingest data. In this case your settings should be put on the all-in-one server itself. (if you had a heavy forwarder, they'd need to be put there).

And with the "where" in terms of directory - you skipped one component of the path.

If you're creating an app called ABC, you put it into /opt/splunk/etc/apps/ABC and there you create directories like /opt/splunk/etc/apps/ABC/default, /opt/splunk/etc/apps/ABC/local, /opt/splunk/etc/apps/ABC/lookups and so on.

So if you put your conf file(s) in /opt/splunk/etc/apps/local, splunk will ignore them.

Oh, and remember about the config file precedence https://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Wheretofindtheconfigurationfiles

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jeff1,

your request isn't so clear for me, are you asking of:

  • discard some full events before indexing,
  • delete a part of events

?

In the first case you have to follow the steps at https://docs.splunk.com/Documentation/Splunk/8.2.5/Forwarding/Routeandfilterdatad

In the second case, you have to follow the steps at https://docs.splunk.com/Documentation/Splunk/8.2.5/Data/Anonymizedata

Ciao.

Giuseppe

0 Karma

jeff1
Loves-to-Learn

hi @gcusello ,

I want to discard events that contain specific values in a field in index-time.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jeff1,

in this case, you have to follow the steps in the first link, if few words, you have to create on your Indexers or (when present) on your Heavy Forwarders the files you shared.

If they don't work, please share a sample of your logs (someone to delete and someone to not delete) to check your regex,

then, where do you located props.conf and transforms.conf?

Ciao.

Giuseppe

0 Karma

jeff1
Loves-to-Learn

hi @gcusello ,

I do have both indexer and heavy forwarder in my testing environment, may I ask if you mean all index-time operation conf. file is better put in Indexer or Heavy Forwarder? As both of my conf. files mentioned above are put under the path of /etc/apps in Search Head.

Here are some of my logs:

type=USER_END msg=audit(xxxxxxxxxxxx) source = /var/log/audit/audit.log    <- log to keep

type=CRED_DISP msg=audit(xxxxxxxxxx) source = /var/log/audit/audit.log  <-log to eliminate

I am trying to eliminate all logs other than (type=USER_*), where [type] is the interesting field.

Sorry for the messy elaboration. 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

The index-time operations are done on the first "heavy" (based on a full splunk enterprise installaton, not on universal forwarder package) component as the event traverses your network to the indexer.

So if you have simple UF->indexer setup, you need to put your index-time operations configuration on the indexer. If you have more complicated setup, like - for example - UF -> UF -> HF -> indexer, you need to put it on HF.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jeff1,

about the location of the conf files, if you're sure that all your logs to filter pass through the HFs, you have to put them on HFs, otherwise you have to put them both on IDXs and HFs.

if you want to take only the events where type=USER_END you have to use the following props.conf and transforms.conf:

props.conf:

[source: path to the log file]
TRANSFORMS-set = setnull,setparsing

transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^type\=USER_END
DEST_KEY = queue
FORMAT = indexQueue

If instead you want to discard only type=CRED_DISP you have to use the following props.conf and transforms.conf:

props.conf:

[source: path to the log file]
TRANSFORMS-null= setnull

transforms.conf:

[setnull]
REGEX = ^type\=CRED_DISP
DEST_KEY = queue
FORMAT = nullQueue

Ciao.

Giuseppe 

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Hi Splunky people! We are excited to share the newest updates in Splunk Enterprise 9.3!Admins and Analyst can ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...