Splunk Search

Trimming the amount of data forwarded

tnconners
Explorer

Background: We have an existing indexer, that we have added a lot of data to. We would like to cut down on the amount of logs we index.

Basically, we alert on any logs that come in as HIGH or CRITICAL. I'm fairly certain I can set up the forwarder to only forward logs that have the word HIGH or CRITICAL. My question is how is this accomplished. I've looked through much of the documentation already and haven't found a clear cut answer.

Any help is appreciated!

EDIT: The indexer and Forwarders are both on windows, Splunk version is 5.05. I apologize but based on the nature of the logs I cannot post them. If you could give me a general view of how to do it, I could create the regex myself for it.

0 Karma

lukejadamec
Super Champion

So, basically you need to configure a nullQueue on the system that is cooking the data. Here is an example of dropping unwanted events from a windows security log:

Props.conf

[source::WMI:WinEventLog:Security]

TRANSFORMS-FilterEvent = FilterEvent560

Transforms.conf

[FilterEvent560]

REGEX = (?msi)^EventCode=560

DEST_KEY = queue

FORMAT = nullQueue

You will want to change the source in props.conf to match your source, and probably change the 'FilterEvent560' identifier to something that makes more sense to you, like 'dropLow'. You will also need to create a REGEX that matches what you want to drop.

If you have any questions let us know.

0 Karma

tnconners
Explorer

To further my edit, the logs are actually pretty nice. There is some specific data then |high | or |critical | then more specific data.

0 Karma

lukejadamec
Super Champion

Yes, this can be done, but you need to provide more information.
An example of the logs for both with HIGH and CRITICAL and without, so we can actually give you specific answers would be nice.
What OS, and Splunk version?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...