log file exceeding data limit in splunk

[smallwonder]

How do I limit the amount of data coming over from 

[monitor://path/to/file]

in my splunk forwarder inputs.conf file. 

I did see whitelist directory and blacklist directory.

Any other ways to limit the log files from for example WinFIM from exceeding the data.

[gcusello]

Hi @smallwonder ,

when you say limit the amount of data, are you meaning: limiting the files to read or filter events?

if limiting the files to read, you can add whitelist and blacklist options to your inputs.conf.

If instead you want to filter sone data, you have to identify one or more regexes to filter your logs (positive or negative filtering), and then apply the method described at https://docs.splunk.com/Documentation/Splunk/9.3.2/Forwarding/Routeandfilterdatad

Remember that these filters must be applied in the first full Splunk instance they are passing through, in other words on the first Heavy Forwarder present or on Indexers, not on Universal Forwarders.

Ciao.

giuseppe

[SanjayReddy]

Hi @smallwonder 

In addition to @gcusello  said

If you want to reduce the data ingested into Splunk like removing some log events you can also try ingest actions. (similar to null queue)
 https://docs.splunk.com/Documentation/Splunk/latest/Data/DataIngest#Filter_with_regular_expression

This can be done on heavy forwarders, it's an UI based and easy to navigate.

Also in case of  monitoring new log files you can try to add ignoreolderthan to avoid ingesting older  specified time.

[smallwonder]

Can I just specify the maximum amount of data I want to send over for that day. If it reaches say 1gb of data per day it will stop forwarding until the next day.

[PickleRick]

No. There's no such functionality within Splunk itself but you could implement something like this as modular or scripted input but you'd have to write such input yourself.

[SanjayReddy]

Hi @smallwonder 

Currently there is no option to limit data sent to splunk after reaching certian limit. you can try filter the data which i mentioned earlier post.