Getting Data In

How to start indexing log files based on date

mriley_cpmi
Explorer

I have a new installation of Splunk Enterprise and we're about ready to start indexing our log files from our various applications.

Currently, if we point our various Splunk Forwarders to our log directories, with file name filters, the volume of data imported would exceed our quota within a few hours.

How do we configure the import so we only import log files created on or after a specified date and ignore everything else?

0 Karma
1 Solution

mdsnmss
SplunkTrust
SplunkTrust

Have you looked at ignoreOlderThan in inputs.conf? Under your monitor you can tell it to look back and ingest based on time but can't specify exact dates. More info on this setting can be found here: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf#MONITOR:.

View solution in original post

0 Karma

mdsnmss
SplunkTrust
SplunkTrust

Have you looked at ignoreOlderThan in inputs.conf? Under your monitor you can tell it to look back and ingest based on time but can't specify exact dates. More info on this setting can be found here: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf#MONITOR:.

View solution in original post

0 Karma

ddrillic
Ultra Champion

Migration of Splunk and the quotas don't go hand by hand and as @mdsnmss said ignoreOlderThan is a key feature here. Unfortunately, it's a one time deal per forwarder. You can potentially do it for a set of forwarders per day and then set another group of forwarders with the ignoreOlderThan in the next day - not a simple task.

0 Karma

mriley_cpmi
Explorer

@mdsnmss Is it possible to modify this setting from the central Splunk server and push the change out? We have limited, intermittent access to the servers that the agents are installed on.

0 Karma

mdsnmss
SplunkTrust
SplunkTrust

Are you managing your forwarders via deployment server? If so, you can push the change out via inputs.conf for each input from that.

0 Karma

mriley_cpmi
Explorer

@mdsnmss I have access to the web administration interface but limited access to the server itself. Do I need to modify the inputs.conf on the deployment server in order for this to work or can I use the web interface?

After the change, I understand that the forwarders will poll the deployment server then apply any changes. Is that true?

0 Karma

mriley_cpmi
Explorer

@mdsnmss I ran a test and what I need to do is apply ignoreOlderThan on or before first run. As I'm adding new data from a new forwarder, I don't have the option to apply ignoreOlderThan before the forwarder begins to index all logs matching the filename pattern.

Do I need to manually config a [monitor] block in $SPLUNK_HOME/etc/system/local/inputs.conf, on each forwarder, apply ignoreOlderThan and restart the forwarder in order to prevent old logs from being imported?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!