topic Re: How to exclude JSON entries from the indexing? in Getting Data In

How to exclude JSON entries from the indexing?

craymore — Fri, 02 Mar 2018 17:46:16 GMT

Hello dear splunketeers !

I am seeking some advice.

The splunk architecture I currently manage is fairly simple :

one central splunk server
a few hundreds of universal splunk forwarder agents
each agent is monitoring JSON log files from our applications and sending them to our central splunk server

Everything is working fine, but I am trying to reduce the amount of data being indexed on our central splunk server

e.g :

here's a log sample that is sent by the universal forwarder

{
  "@timestamp": "2018-03-02T17:15:14.756+00:00",
  "@level": "INFO",
  "@message": "foobar"
}
{
  "@timestamp": "2018-03-02T17:15:15.800+00:00",
  "@level": "ERROR",
  "@message": "damnit !"
}
{
  "@timestamp": "2018-03-02T17:15:16.136+00:00",
  "@level": "INFO",
  "@message": "foobar"
}

here's the final log that I would like to see indexed on the central server
```
{
  "@timestamp": "2018-03-02T17:15:15.800+00:00",
  "@level": "ERROR",
  "@message": "damnit !"
}
```
So basically, I would like to get rid of all the JSON entries which level is set to "INFO"

Does anyone how to perform this ( simple ? ) filtering?

Thanks beforehand for any answer

Re: How to exclude JSON entries from the indexing?

somesoni2 — Fri, 02 Mar 2018 19:23:21 GMT

You basically need to setup routing of specific events (events that contain @level as INFO) to nullQueue (drop from being indexed). The reference configuration could be found here: http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

So, in your case (all setting are done in your central Splunk server)

1) In props.conf, set the TRANSFORMS-null attribute:

[YourSourceTypeHere]
TRANSFORMS-null= drop_info_jsons

2) Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[drop_info_jsons]
REGEX = \"\@level\"\:\s*\"INFO\"
DEST_KEY = queue
FORMAT = nullQueue

3) Restart Splunk Enterprise.

Re: How to exclude JSON entries from the indexing?

craymore — Mon, 05 Mar 2018 09:20:03 GMT

I implemented the aformentionned settings in order to a have a redirect to the nullQueue for some entries and indeed it works as intended.

Thank you for advice !