topic Re: How to filter a large amount of index data being generated by the head index server? in Getting Data In

How to filter a large amount of index data being generated by the head index server?

ntripp_element — Tue, 27 Feb 2018 16:34:10 GMT

I've noticed the head index server is generating an absurd amount of index data and I want to filter it out

I have a stanza in props:

[host::<hostname>]
 TRANSFORMS-<hostname> = host_setnull

and

[host_setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Is there something else I'm missing? I'm still seeing the events increment.

Re: How to filter a large amount of index data being generated by the head index server?

somesoni2 — Tue, 27 Feb 2018 16:42:51 GMT

Where did you apply these changes? It should be done on your heavy/intermediate forwarder OR on indexer, whichever comes first in the data flow. A splunk restart is also required after making the change.

Re: How to filter a large amount of index data being generated by the head index server?

ntripp_element — Tue, 27 Feb 2018 16:46:34 GMT

applied on indexer (that's all there is). service was restarted

Re: How to filter a large amount of index data being generated by the head index server?

somesoni2 — Tue, 27 Feb 2018 17:03:40 GMT

Does the <host_name> you put in props.conf matches correctly with host field in the event? Is the head index server a server with forwarder installed on it? What's your environment looks like (topology wise)?

Re: How to filter a large amount of index data being generated by the head index server?

ntripp_element — Tue, 27 Feb 2018 18:17:30 GMT

is just a placeholder for the actual hostname that i put in the conf file. We have 1 splunk instance that we are feeding everything to and DCN node for the vmware stuff. So topology wise couldn't be much simpler.

Re: How to filter a large amount of index data being generated by the head index server?

ntripp_element — Tue, 27 Feb 2018 20:03:08 GMT

I'll take that as this should be working then?

Re: How to filter a large amount of index data being generated by the head index server?

somesoni2 — Tue, 27 Feb 2018 20:50:16 GMT

Looks good to me. So this head index server you're referring to, is it your Splunk indexer OR a server which is feeding data to your indexer?

Re: How to filter a large amount of index data being generated by the head index server?

ntripp_element — Tue, 27 Feb 2018 21:29:54 GMT

It's my Spunk indexer

Re: How to filter a large amount of index data being generated by the head index server?

somesoni2 — Tue, 27 Feb 2018 21:33:07 GMT

So when you say it's generating absurd amount of indexed data, from where that data is coming from? Is being monitored/generated on your indexer server itself? What sourcetype(s) does that unwanted data of your have?

Re: How to filter a large amount of index data being generated by the head index server?

ntripp_element — Tue, 27 Feb 2018 21:37:24 GMT

I thought it must be data from the indexer itself? We're on a trial and hit 100GB today I'm just trying to sort this by the largest volume events that I don't care about and trim this usage into something useful

Re: How to filter a large amount of index data being generated by the head index server?

somesoni2 — Tue, 27 Feb 2018 21:48:49 GMT

I would say you find out which sourcetype or sourcetypes are eating most of your license and then use your nullQueue routing for them. Try running this

index=_internal sourcetype=splunkd component=LicenseUsage type=Usage | stats sum(b) as usage by st | sort 5 -usage | eval usage=round(usage/1024/1024/1024,2)

This will give top 5 sourcetypes based on license usage for selected time range. From this list whatever sourcetypes that you don't want data to be ingested, you can either turn off the monitoring for it (it must be in inputs.conf somewhere) or apply TRANSFORMS for those sourcetypes.