Getting Data In

How to filter a large amount of index data being generated by the head index server?

Explorer

I've noticed the head index server is generating an absurd amount of index data and I want to filter it out

I have a stanza in props:

[host::<hostname>]
 TRANSFORMS-<hostname> = host_setnull

and

[host_setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Is there something else I'm missing? I'm still seeing the events increment.

0 Karma

Explorer

I'll take that as this should be working then?

0 Karma

Revered Legend

Looks good to me. So this head index server you're referring to, is it your Splunk indexer OR a server which is feeding data to your indexer?

0 Karma

Explorer

It's my Spunk indexer

0 Karma

Revered Legend

So when you say it's generating absurd amount of indexed data, from where that data is coming from? Is being monitored/generated on your indexer server itself? What sourcetype(s) does that unwanted data of your have?

0 Karma

Explorer

I thought it must be data from the indexer itself? We're on a trial and hit 100GB today I'm just trying to sort this by the largest volume events that I don't care about and trim this usage into something useful

0 Karma

Revered Legend

I would say you find out which sourcetype or sourcetypes are eating most of your license and then use your nullQueue routing for them. Try running this

index=_internal sourcetype=splunkd component=LicenseUsage type=Usage | stats sum(b) as usage by st | sort 5 -usage | eval usage=round(usage/1024/1024/1024,2)

This will give top 5 sourcetypes based on license usage for selected time range. From this list whatever sourcetypes that you don't want data to be ingested, you can either turn off the monitoring for it (it must be in inputs.conf somewhere) or apply TRANSFORMS for those sourcetypes.

0 Karma

Revered Legend

Where did you apply these changes? It should be done on your heavy/intermediate forwarder OR on indexer, whichever comes first in the data flow. A splunk restart is also required after making the change.

0 Karma

Explorer

applied on indexer (that's all there is). service was restarted

0 Karma

Revered Legend

Does the <host_name> you put in props.conf matches correctly with host field in the event? Is the head index server a server with forwarder installed on it? What's your environment looks like (topology wise)?

0 Karma

Explorer

is just a placeholder for the actual hostname that i put in the conf file. We have 1 splunk instance that we are feeding everything to and DCN node for the vmware stuff. So topology wise couldn't be much simpler.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!