topic Re: log file exceeding data limit in splunk in Getting Data In

log file exceeding data limit in splunk

smallwonder — Tue, 03 Dec 2024 14:52:30 GMT

How do I limit the amount of data coming over from

[monitor://path/to/file]

in my splunk forwarder inputs.conf file.

I did see whitelist directory and blacklist directory.

Any other ways to limit the log files from for example WinFIM from exceeding the data.

Re: log file exceeding data limit in splunk

gcusello — Tue, 03 Dec 2024 14:57:49 GMT

Hi @smallwonder ,

when you say limit the amount of data, are you meaning: limiting the files to read or filter events?

if limiting the files to read, you can add whitelist and blacklist options to your inputs.conf.

If instead you want to filter sone data, you have to identify one or more regexes to filter your logs (positive or negative filtering), and then apply the method described at https://docs.splunk.com/Documentation/Splunk/9.3.2/Forwarding/Routeandfilterdatad

Remember that these filters must be applied in the first full Splunk instance they are passing through, in other words on the first Heavy Forwarder present or on Indexers, not on Universal Forwarders.

Ciao.

giuseppe

Re: log file exceeding data limit in splunk

smallwonder — Tue, 03 Dec 2024 15:03:51 GMT

Can I just specify the maximum amount of data I want to send over for that day. If it reaches say 1gb of data per day it will stop forwarding until the next day.

Re: log file exceeding data limit in splunk

SanjayReddy — Tue, 03 Dec 2024 15:11:41 GMT

Hi @smallwonder

In addition to @gcusello said

If you want to reduce the data ingested into Splunk like removing some log events you can also try ingest actions. (similar to null queue)
https://docs.splunk.com/Documentation/Splunk/latest/Data/DataIngest#Filter_with_regular_expression

This can be done on heavy forwarders, it's an UI based and easy to navigate.

Also in case of monitoring new log files you can try to add ignoreolderthan to avoid ingesting older specified time.

Re: log file exceeding data limit in splunk

SanjayReddy — Tue, 03 Dec 2024 16:08:14 GMT

Hi @smallwonder

Currently there is no option to limit data sent to splunk after reaching certian limit. you can try filter the data which i mentioned earlier post.

Re: log file exceeding data limit in splunk

PickleRick — Tue, 03 Dec 2024 18:49:53 GMT

No. There's no such functionality within Splunk itself but you could implement something like this as modular or scripted input but you'd have to write such input yourself.