Getting Data In
Highlighted

How to exclude JSON entries from the indexing?

New Member

Hello dear splunketeers !

I am seeking some advice.

The splunk architecture I currently manage is fairly simple :

  • one central splunk server
  • a few hundreds of universal splunk forwarder agents
  • each agent is monitoring JSON log files from our applications and sending them to our central splunk server

Everything is working fine, but I am trying to reduce the amount of data being indexed on our central splunk server

e.g :

  1. here's a log sample that is sent by the universal forwarder

    {
      "@timestamp": "2018-03-02T17:15:14.756+00:00",
      "@level": "INFO",
      "@message": "foobar"
    }
    {
      "@timestamp": "2018-03-02T17:15:15.800+00:00",
      "@level": "ERROR",
      "@message": "damnit !"
    }
    {
      "@timestamp": "2018-03-02T17:15:16.136+00:00",
      "@level": "INFO",
      "@message": "foobar"
    }
    
  2. here's the final log that I would like to see indexed on the central server

    {
      "@timestamp": "2018-03-02T17:15:15.800+00:00",
      "@level": "ERROR",
      "@message": "damnit !"
    }
    

    So basically, I would like to get rid of all the JSON entries which level is set to "INFO"

Does anyone how to perform this ( simple ? ) filtering?

Thanks beforehand for any answer

0 Karma
Highlighted

Re: How to exclude JSON entries from the indexing?

SplunkTrust
SplunkTrust

You basically need to setup routing of specific events (events that contain @level as INFO) to nullQueue (drop from being indexed). The reference configuration could be found here: http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

So, in your case (all setting are done in your central Splunk server)

1) In props.conf, set the TRANSFORMS-null attribute:

[YourSourceTypeHere]
TRANSFORMS-null= drop_info_jsons

2) Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[drop_info_jsons]
REGEX = \"\@level\"\:\s*\"INFO\"
DEST_KEY = queue
FORMAT = nullQueue

3) Restart Splunk Enterprise.

View solution in original post

0 Karma
Highlighted

Re: How to exclude JSON entries from the indexing?

New Member

I implemented the aformentionned settings in order to a have a redirect to the nullQueue for some entries and indeed it works as intended.

Thank you for advice !

0 Karma