How to exclude JSON entries from the indexing?

[craymore]

Hello dear splunketeers !

I am seeking some advice.

The splunk architecture I currently manage is fairly simple :

• one central splunk server
• a few hundreds of universal splunk forwarder agents
• each agent is monitoring JSON log files from our applications and sending them to our central splunk server

Everything is working fine, but I am trying to reduce the amount of data being indexed on our central splunk server

e.g :

1. here's a log sample that is sent by the universal forwarder

   {
     "@timestamp": "2018-03-02T17:15:14.756+00:00",
     "@level": "INFO",
     "@message": "foobar"
   }
   {
     "@timestamp": "2018-03-02T17:15:15.800+00:00",
     "@level": "ERROR",
     "@message": "damnit !"
   }
   {
     "@timestamp": "2018-03-02T17:15:16.136+00:00",
     "@level": "INFO",
     "@message": "foobar"
   }
   

2. here's the final log that I would like to see indexed on the central server

   {
     "@timestamp": "2018-03-02T17:15:15.800+00:00",
     "@level": "ERROR",
     "@message": "damnit !"
   }
   

   So basically, I would like to get rid of all the JSON entries which level is set to "INFO"

Does anyone how to perform this ( simple ? ) filtering?

Thanks beforehand for any answer

[somesoni2 · Accepted solution]

You basically need to setup routing of specific events (events that contain @level as INFO) to nullQueue (drop from being indexed). The reference configuration could be found here: http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

So, in your case (all setting are done in your central Splunk server)

1) In props.conf, set the TRANSFORMS-null attribute:

[YourSourceTypeHere]
TRANSFORMS-null= drop_info_jsons

2) Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[drop_info_jsons]
REGEX = \"\@level\"\:\s*\"INFO\"
DEST_KEY = queue
FORMAT = nullQueue

3) Restart Splunk Enterprise.

[craymore]

I implemented the aformentionned settings in order to a have a redirect to the nullQueue for some entries and indeed it works as intended.

Thank you for advice !