All Apps and Add-ons

Hunk App for MongoDB: How to filter data coming in from MongoDB before getting indexed (virtual index)?

prachisaxena
Explorer

Hi All,

I have a MongoDB and Splunk Enterprise setup with a trial license for Hunk.
Have created a Virtual Index and Provider using Hunk App for MongoDB and it is working fine.
The data in MongoDB is huge and we do not want to get all of that data into Splunk Enterprise because it is getting charged under the Splunk Enterprise License.

Is there a way to filter the data coming in from MongoDB before it is getting indexed through Splunk Indexer.

0 Karma
1 Solution

rdagan_splunk
Splunk Employee
Splunk Employee

You have two main options

1) The first option is to filter based on Time:
If you setup time field in the configurations, you can use the Hunk Time picker to bring less data based on time. For example, Query Database = A, Table = B, For only 3 days worth to data
vix.mongodb.db = hunk
vix.mongodb.collection = test
vix.mongodb.field.time = _id
vix.mongodb.field.time.format = ObjectId

2) The second option is to filter based on Key Value pair:
In this example, only foo=xyz dataset will be brought back from MongoDB
index=mongodb foo=xyz | timechart avg(bar) by baz

View solution in original post

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

You have two main options

1) The first option is to filter based on Time:
If you setup time field in the configurations, you can use the Hunk Time picker to bring less data based on time. For example, Query Database = A, Table = B, For only 3 days worth to data
vix.mongodb.db = hunk
vix.mongodb.collection = test
vix.mongodb.field.time = _id
vix.mongodb.field.time.format = ObjectId

2) The second option is to filter based on Key Value pair:
In this example, only foo=xyz dataset will be brought back from MongoDB
index=mongodb foo=xyz | timechart avg(bar) by baz

0 Karma

kartik13
Communicator

You can black list the data in inputs.conf or even can parse the data using Heavy forwarder or even writing a script for parsing or if you want cheap-fix you can decrease the throughput .It will reduce the speed for ingestion of data in splunk which will be under scheme.

0 Karma

prachisaxena
Explorer

Hi Karthik,

Thanks for helping.. I was wondering if we can use inputs.conf, props.conf and transforms.conf in a similar way as we use on Splunk Enterprise filtering. The requirement is it filter the data on MongoDB app for hunk and currently it is not using any inputs.conf. I am still newbie to splunk to may be i am wrong here

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...