Getting Data In

log file exceeding data limit in splunk

smallwonder
Loves-to-Learn

How do I limit the amount of data coming over from 

[monitor://path/to/file]

in my splunk forwarder inputs.conf file. 

I did see whitelist directory and blacklist directory.

Any other ways to limit the log files from for example WinFIM from exceeding the data.

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @smallwonder ,

when you say limit the amount of data, are you meaning: limiting the files to read or filter events?

if limiting the files to read, you can add whitelist and blacklist options to your inputs.conf.

If instead you want to filter sone data, you have to identify one or more regexes to filter your logs (positive or negative filtering), and then apply the method described at https://docs.splunk.com/Documentation/Splunk/9.3.2/Forwarding/Routeandfilterdatad

Remember that these filters must be applied in the first full Splunk instance they are passing through, in other words on the first Heavy Forwarder present or on Indexers, not on Universal Forwarders.

Ciao.

giuseppe

0 Karma

SanjayReddy
SplunkTrust
SplunkTrust

Hi @smallwonder 

In addition to @gcusello  said

If you want to reduce the data ingested into Splunk like removing some log events you can also try ingest actions. (similar to null queue)
 https://docs.splunk.com/Documentation/Splunk/latest/Data/DataIngest#Filter_with_regular_expression

This can be done on heavy forwarders, it's an UI based and easy to navigate.

Also in case of  monitoring new log files you can try to add ignoreolderthan to avoid ingesting older  specified time.
0 Karma

smallwonder
Loves-to-Learn

Can I just specify the maximum amount of data I want to send over for that day. If it reaches say 1gb of data per day it will stop forwarding until the next day.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

No. There's no such functionality within Splunk itself but you could implement something like this as modular or scripted input but you'd have to write such input yourself.

0 Karma

SanjayReddy
SplunkTrust
SplunkTrust

Hi @smallwonder 

Currently there is no option to limit data sent to splunk after reaching certian limit. you can try filter the data which i mentioned earlier post.



0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...