topic Re: exclude winevent older than 7 days from ingest in Getting Data In

exclude winevent older than 7 days from ingest

Andre_ — Mon, 16 Jun 2025 04:59:03 GMT

Hello,

I am about to onboard 1000+ Windows UF. Those have windows event logs going back many years. Is there a way to exclude any windows eventlog older than 7 days from being ingested during the initial onboarding?

For log files there's an option for inputs.conf on the UF, but nothing similar for eventlog?

Kind Regards
Andre

Re: exclude winevent older than 7 days from ingest

PrewinThomas — Mon, 16 Jun 2025 05:30:03 GMT

@Andre_

You are correct. Unlike file-based inputs, Windows Event Log inputs in Splunk Universal Forwarder (UF) do not provide a built-in option in inputs.conf to exclude events based on their age at collection time. This means you cannot natively configure the UF to only ingest Windows events newer than 7 days during onboarding.

But, If you want to ingest only new Windows Event Log events (and skip all historical data), set current_only = 1 in your inputs.conf.

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!

Re: exclude winevent older than 7 days from ingest

Andre_ — Mon, 16 Jun 2025 05:45:55 GMT

I've seen the "current_only" option but discarded that as it will not ingest any historical data.

If I set "current_only=1" during initial deployment it will not ingest old data - so far so good.
If the UF goes down for a period of time, after a restart it will not process the events that occurred whilst the UF was down - bad

What happens if I deploy the UF with "current_only =1" and after a week I remove the setting? will it start ingesting all historical? Or could I use that as a temporary setting during the onboarding phase and remove for production phase?

Kind Regards
Andre

Re: exclude winevent older than 7 days from ingest

PrewinThomas — Mon, 16 Jun 2025 05:51:06 GMT

With current_only = 1
On first start, the UF reads only new events that arrive after the input is enabled.It skips all historical events present in the log at the time the input is first started.

If the UF is stopped and restarted, it will pick up where it left off (using checkpoints), so normally it will ingest events that occurred while it was down.

#https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/9.4/get-windows-data/monitor-windows-event-log-data-with-splunk-enterprise

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

Re: exclude winevent older than 7 days from ingest

gcusello — Mon, 16 Jun 2025 06:43:33 GMT

Hi @Andre_ ,

as you can read at https://docs.splunk.com/Documentation/Splunk/9.4.2/Admin/Inputsconf , to read only the events newer than 7 days, you have to use, in you inputs.conf the option ignoreOlderThan:

ignoreOlderThan = <non-negative integer>[s|m|h|d] * The monitor input compares the modification time on files it encounters with the current time. If the time elapsed since the modification time is greater than the value in this setting, Splunk software puts the file on the ignore list. * Files on the ignore list are not checked again until the Splunk platform restarts, or the file monitoring subsystem is reconfigured. This is true even if the file becomes newer again at a later time. * Reconfigurations occur when changes are made to monitor or batch inputs through Splunk Web or the command line. * Use 'ignoreOlderThan' to increase file monitoring performance when monitoring a directory hierarchy that contains many older, unchanging files, and when removing or adding a file to the deny list from the monitoring location is not a reasonable option. * Do NOT select a time that files you want to read could reach in age, even temporarily. Take potential downtime into consideration! * Suggested value: 14d, which means 2 weeks * For example, a time window in significant numbers of days or small numbers of weeks are probably reasonable choices. * If you need a time window in small numbers of days or hours, there are other approaches to consider for performant monitoring beyond the scope of this setting. * NOTE: Most modern Windows file access APIs do not update file modification time while the file is open and being actively written to. Windows delays updating modification time until the file is closed. Therefore you might have to choose a larger time window on Windows hosts where files may be open for long time periods. * Value must be: <number><unit>. For example, "7d" indicates one week. * Valid units are "d" (days), "h" (hours), "m" (minutes), and "s" (seconds). * No default, meaning there is no threshold and no files are ignored for modification time reasons

Ciao.

Giuseppe

Re: exclude winevent older than 7 days from ingest

Andre_ — Mon, 16 Jun 2025 07:09:31 GMT

Hi Giuseppe,

"ignoreOlderThan" only applies to log files, not windows event logs (like security events, application events, etc)

Kind Regards
Andre

Re: exclude winevent older than 7 days from ingest

Andre_ — Mon, 16 Jun 2025 07:13:25 GMT

I think the opposite is the case:

current_only = <boolean>
* Whether or not to acquire only events that arrive while the instance is 
  running.
* A value of "true" means the input only acquires events that arrive
  while the instance runs and the input is on. The input does not read
  data which was stored in the Windows Event Log while the instance was not
  running. This means that there will be gaps in the data if you restart the
  instance or experiences downtime.

Re: exclude winevent older than 7 days from ingest

gcusello — Mon, 16 Jun 2025 08:22:39 GMT

Hi @Andre_ ,

Iì'm not sure about this: I used it on wineventlogs.

Ciao.

Giuseppe

Re: exclude winevent older than 7 days from ingest

Andre_ — Mon, 16 Jun 2025 08:27:48 GMT

It's not in the spec file, I tried and it does not work.

Re: exclude winevent older than 7 days from ingest

PrewinThomas — Mon, 16 Jun 2025 08:59:04 GMT

@Andre_

Another option you can consider is to change the destination path for the Windows Event Logs in Event Viewer and configure Splunk to monitor this new location. This approach allows you to start collecting only new events, effectively avoiding the indexing of historical data. Additionally, by using the standard Splunk input settings (without current_only = 1), you ensure that no events are missed during restarts, as Splunk will continue to track and ingest all new events from the updated log file.

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

Re: exclude winevent older than 7 days from ingest

inventsekar — Mon, 16 Jun 2025 09:15:08 GMT

Hi @Andre_

Pls check the MAX_DAYS_AGO option on the props.conf

https://docs.splunk.com/Documentation/Splunk/9.4.2/Admin/Propsconf

MAX_DAYS_AGO = <integer>
* The maximum number of days in the past, from the current date as
  provided by the input layer (For example forwarder current time, or modtime
  for files), that an extracted date can be valid.
* Splunk software still indexes events with dates older than 'MAX_DAYS_AGO'
  with the timestamp of the last acceptable event.
* If no such acceptable event exists, new events with timestamps older
  than 'MAX_DAYS_AGO' uses the current timestamp.
* For example, if MAX_DAYS_AGO = 10, Splunk software applies the timestamp
  of the last acceptable event to events with extracted timestamps older
  than 10 days in the past. If no acceptable event exists, Splunk software
  applies the current timestamp.
* If your data is older than 2000 days, increase this setting.
* Highest legal value: 10951 (30 years).
* Default: 2000 (5.48 years).

Re: exclude winevent older than 7 days from ingest

Andre_ — Mon, 16 Jun 2025 21:30:35 GMT

MAX_DAYS_AGO - I would set this on the indexer? (our setup is UF -> Indexer)
Will that be a global setting for all incoming data?

Kind Regards
Andre

Re: exclude winevent older than 7 days from ingest

livehybrid — Mon, 16 Jun 2025 23:19:04 GMT

Hi @Andre_

As @inventsekar mentioned, you could use MAX_DAYS_AGO as follows:

== props.conf == # If within 3 days old. [WinEventLog] MAX_DAYS_AGO = 3 [XmlWinEventLog] MAX_DAYS_AGO = 3

This will then only apply to XmlWinEventLog/WinEventLog

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: exclude winevent older than 7 days from ingest

Andre_ — Tue, 17 Jun 2025 05:56:24 GMT

trying that now, does it require a restart?

Re: exclude winevent older than 7 days from ingest

Andre_ — Tue, 17 Jun 2025 21:28:34 GMT

Hello,

so I've created a props.conf on the indexer under the Windows_TA local folder and put his in:

[WinEventLog]
MAX_DAYS_AGO = 7

[XmlWinEventLog]
MAX_DAYS_AGO = 7

onboarded another Windows Server - still ingested windows event logs going back a few years.

Any ideas why that's not working?

Kind Regards
Andre

Re: exclude winevent older than 7 days from ingest

inventsekar — Wed, 18 Jun 2025 00:27:28 GMT

Hi @Andre_

1) after props.conf 's update/creation, did you restart the splunkd on the indexer?

2) if yes for above, then pls use the btool command to check if the props.conf got applied or not(you can search for splunk btool options here in communities).

if any reply helps you in any way, a karma point / upvote would be helpful for the author, thanks.

Re: exclude winevent older than 7 days from ingest

Andre_ — Wed, 18 Jun 2025 01:09:05 GMT

I've done a rolling restart of the cluster and checked. Looks like it "should" work but doesn't.

Since then, I tried this approach:

put a "blacklist_all_WinEvent" app on the UF during initial start. Just an inputs.conf that has "blacklist1 = ." for all winevent sources.

let the UF do it's initial thing and an hour later I remove that app from the UF and restart the UF

whilst not optimal, that would do the trick for onboarding existing servers and automating that is easy enough.

Kind Regards
Andre

Re: exclude winevent older than 7 days from ingest

Andre_ — Wed, 18 Jun 2025 02:08:31 GMT

that does not work, once you remove the blacklist, it ingests the old events.....

Re: exclude winevent older than 7 days from ingest

Andre_ — Wed, 18 Jun 2025 02:49:37 GMT

/opt/splunk/bin/splunk btool props list XmlWinEventLog:Security --debug | grep MAX_DAYS_AGO
/opt/splunk/etc/system/local/props.conf MAX_DAYS_AGO = 7

that should work, right? Present on all indexers. All indexers restarted. (Splunk Enterprise 9.4.2)

TIme to log a support call?

Re: exclude winevent older than 7 days from ingest

PickleRick — Wed, 18 Jun 2025 09:12:45 GMT

MAX_DAYS_AGO doesn't cut it. It will make Splunk still index the events but it will assume that the timestamp parsed from the event was wrong so it would just assume another timestamp (whatever that would effectively be).

@Andre_ What you could to to prevent some data from being indexed could be to add whitelists/blacklists in inputs matching certain timestamp values. That's a relatively ugly solution and is not something to be kept forever but for a start - might be the way to go. Just be careful because you're doing it differently when you're windows logs as "classic" and differently while it's XML.