How to start indexing log files based on date

[mriley_cpmi]

I have a new installation of Splunk Enterprise and we're about ready to start indexing our log files from our various applications.

Currently, if we point our various Splunk Forwarders to our log directories, with file name filters, the volume of data imported would exceed our quota within a few hours.

How do we configure the import so we only import log files created on or after a specified date and ignore everything else?

[mdsnmss · Accepted solution]

Have you looked at ignoreOlderThan in inputs.conf? Under your monitor you can tell it to look back and ingest based on time but can't specify exact dates. More info on this setting can be found here: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf#MONITOR:.

[ddrillic]

Migration of Splunk and the quotas don't go hand by hand and as @mdsnmss said ignoreOlderThan is a key feature here. Unfortunately, it's a one time deal per forwarder. You can potentially do it for a set of forwarders per day and then set another group of forwarders with the ignoreOlderThan in the next day - not a simple task.

[mriley_cpmi]

@mdsnmss Is it possible to modify this setting from the central Splunk server and push the change out? We have limited, intermittent access to the servers that the agents are installed on.

[mdsnmss]

Are you managing your forwarders via deployment server? If so, you can push the change out via inputs.conf for each input from that.

[mriley_cpmi]

@mdsnmss I have access to the web administration interface but limited access to the server itself. Do I need to modify the inputs.conf on the deployment server in order for this to work or can I use the web interface?

After the change, I understand that the forwarders will poll the deployment server then apply any changes. Is that true?

[mriley_cpmi]

@mdsnmss I ran a test and what I need to do is apply ignoreOlderThan on or before first run. As I'm adding new data from a new forwarder, I don't have the option to apply ignoreOlderThan before the forwarder begins to index all logs matching the filename pattern.

Do I need to manually config a [monitor] block in $SPLUNK_HOME/etc/system/local/inputs.conf, on each forwarder, apply ignoreOlderThan and restart the forwarder in order to prevent old logs from being imported?