exclude winevent older than 7 days from ingest

[Andre_]

Hello,

I am about to onboard 1000+ Windows UF. Those have windows event logs going back many years. Is there a way to exclude any windows eventlog older than 7 days from being ingested during the initial onboarding?

For log files there's an option for inputs.conf on the UF, but nothing similar for eventlog?

Kind Regards
Andre

[gcusello]

Hi @Andre_ ,

Iì'm not sure about this: I used it on wineventlogs.

Ciao.

Giuseppe

[PickleRick]

Won't work. As you can see in the spec - it doesn't work on the event level but on the file's mtime. It is a setting for this particular input type and doesn't make sense in another context.

BTW, if you had files created sufficiently long time ago but containing events with present timestamps, it still wouldn't ingest those files.

[Andre_]

It's not in the spec file, I tried and it does not work.

[gcusello]

Hi @Andre_ ,

as you can read at https://docs.splunk.com/Documentation/Splunk/9.4.2/Admin/Inputsconf , to read only the events newer than 7 days, you have to use, in you inputs.conf the option ignoreOlderThan:

ignoreOlderThan = <non-negative integer>[s|m|h|d]
* The monitor input compares the modification time on files it encounters
  with the current time. If the time elapsed since the modification time
  is greater than the value in this setting, Splunk software puts the file
  on the ignore list.
* Files on the ignore list are not checked again until the Splunk
  platform restarts, or the file monitoring subsystem is reconfigured. This
  is true even if the file becomes newer again at a later time.
  * Reconfigurations occur when changes are made to monitor or batch
    inputs through Splunk Web or the command line.
* Use 'ignoreOlderThan' to increase file monitoring performance when
  monitoring a directory hierarchy that contains many older, unchanging
  files, and when removing or adding a file to the deny list from the
  monitoring location is not a reasonable option.
* Do NOT select a time that files you want to read could reach in
  age, even temporarily. Take potential downtime into consideration!
  * Suggested value: 14d, which means 2 weeks
  * For example, a time window in significant numbers of days or small
    numbers of weeks are probably reasonable choices.
  * If you need a time window in small numbers of days or hours,
    there are other approaches to consider for performant monitoring
    beyond the scope of this setting.
* NOTE: Most modern Windows file access APIs do not update file
  modification time while the file is open and being actively written to.
  Windows delays updating modification time until the file is closed.
  Therefore you might have to choose a larger time window on Windows
  hosts where files may be open for long time periods.
* Value must be: <number><unit>. For example, "7d" indicates one week.
* Valid units are "d" (days), "h" (hours), "m" (minutes), and "s"
  (seconds).
* No default, meaning there is no threshold and no files are
  ignored for modification time reasons

Ciao.

Giuseppe

[Andre_]

I've seen the "current_only" option but discarded that as it will not ingest any historical data. 

If I set "current_only=1" during initial deployment it will not ingest old data - so far so good.
If the UF goes down for a period of time, after a restart it will not process the events that occurred whilst the UF was down - bad

What happens if I deploy the UF with "current_only =1" and after a week I remove the setting? will it start ingesting all historical? Or could I use that as a temporary setting during the onboarding phase and remove for production phase?

Kind Regards
Andre

[PrewinThomas]

With current_only = 1
On first start, the UF reads only new events that arrive after the input is enabled.It skips all historical events present in the log at the time the input is first started.

If the UF is stopped and restarted, it will pick up where it left off (using checkpoints), so normally it will ingest events that occurred while it was down.

#https://help.splunk.com/en/splunk-enterprise/get-started/get-data-in/9.4/get-windows-data/monitor-wi...

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[Andre_]

I think the opposite is the case:

current_only = <boolean>
* Whether or not to acquire only events that arrive while the instance is 
  running.
* A value of "true" means the input only acquires events that arrive
  while the instance runs and the input is on. The input does not read
  data which was stored in the Windows Event Log while the instance was not
  running. This means that there will be gaps in the data if you restart the
  instance or experiences downtime.

[PrewinThomas]

@Andre_ 

You are correct. Unlike file-based inputs, Windows Event Log inputs in Splunk Universal Forwarder (UF) do not provide a built-in option in inputs.conf to exclude events based on their age at collection time. This means you cannot natively configure the UF to only ingest Windows events newer than 7 days during onboarding.

But, If you want to ingest only new Windows Event Log events (and skip all historical data), set current_only = 1 in your inputs.conf.

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos/Karma. Thanks!