topic Re: FILTERING in Deployment Architecture

FILTERING

dani9 — Fri, 17 Jul 2020 15:21:03 GMT

Hi,

I have 2 sources log so much like 2 gb per day just them.

What could I do to limit it? Is possible to set some filters somewhere (where?)

Either is there another way to permit to send less logs from them?

Re: FILTERING

omerl — Fri, 17 Jul 2020 15:43:49 GMT

What exactly do you want to limit?

If you want to limit the index size you can use `frozenTimePeriodInSecs` to set auto remove by time or `maxTotalDataSizeMB` to restrict index size, using indexes.conf file

Regarding filtering, you can set filtering rules, depending on the input type you are using - see inputs.conf

Re: FILTERING

skoelpin — Fri, 17 Jul 2020 16:01:18 GMT

You can use transforms.conf to preprocess your data before indexing it to cut down on your license usage. This includes excluding certain events, cutting white spaces, unneeded characters etc..

When you scale, you should look into a tool called Cribl which is much easier to use than configuring transforms

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf

Re: FILTERING

gcusello — Fri, 17 Jul 2020 16:02:27 GMT

hi @dani9,

you can filter logs before indexing so they don't consume license but you cannot use those events.

You can find information about how to filter data at:

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues

Ciao.

Giuseppe