All Apps and Add-ons

Depth limit

templier
Communicator

Hello all!
Install and configure Splunk Add-on for Microsoft Office 365 and i want ask a question - it's possible install a depth of log? Now our server downloading from cloud more that 7 gb. But we need only new data.

0 Karma

templier
Communicator

For all - ignoreOlderThan not working!

0 Karma

dkeck
Influencer

Please post your inputs and where you set it.

How did you test its not working?

0 Karma

templier
Communicator

Example inputs.conf:

[splunk_ta_o365_management_activity://index_Azure_AD_Audit]
content_type = Audit.AzureActiveDirectory
index = office365
interval = 900
tenant_name = my365
ignoreOlderThan = 1d

In livetime mode i see event from 25-28 jan.

0 Karma

dkeck
Influencer

as I said, it will not take effect on events you have allready indexed. I assmue you allready did, since you were talking about the 7 gb you want to reduce. So this will take effect after 1d.

0 Karma

templier
Communicator

Maybe i not very correctly says, but after i add this block in inputs.conf and restart my HF. I saw how my indexer indexing a new event with timestamp 25-28 day.

0 Karma

mayurr98
Super Champion

give it time, there may be an older event in the queue it wont reflect suddenly. also have you specified this in every monitor stanza ? there may be a monitor stanza with same index name with ignoreolderthan is not specified so check it carefully. after that restart a HF.
The data which is already indexed will not be reflected. only newly indexed data will be reflected. so check it after few hours

0 Karma

templier
Communicator

Use splunk btool check --debug:
Checking: /opt/splunk/etc/apps/splunk_ta_o365/local/inputs.conf
Invalid key in stanza [splunk_ta_o365_management_activity://Ьн365_Azure_AD_Audit] in /opt/splunk/etc/apps/splunk_ta_o365/local/inputs.conf, line 6: ignoreOlderThan (value: 1h).
Did you mean 'index'?

0 Karma

mayurr98
Super Champion

inside the add-on using CLI in inputs.conf you can add

ignoreOlderThan = <non-negative integer>[s|m|h|d]
* The monitor input will compare the modification time on files it encounters
  with the current time. If the time elapsed since the modification time
  is greater than the value in this setting, Splunk software puts the file
  on the ignore list.
* Files on the ignore list will not be checked again until Splunk 
  software restarts, or the file monitoring subsystem is reconfigured.  This
  is true even if the file becomes newer again at a later time.
  * Reconfigurations occur when changes are made to monitor or batch
    inputs through Splunk Web or the command line.
* Use 'ignoreOlderThan' to increase file monitoring performance when
  monitoring a directory hierarchy that contains many older, unchanging
  files, and when removing or blacklisting those files from the monitoring
  location is not a reasonable option.
* Do NOT select a time that files you want to read could reach in
  age, even temporarily. Take potential downtime into consideration!
  * Suggested value: 14d, which means 2 weeks
  * For example, a time window in significant numbers of days or small
    numbers of weeks are probably reasonable choices.
  * If you need a time window in small numbers of days or hours,
    there are other approaches to consider for performant monitoring
    beyond the scope of this setting.
* NOTE: Most modern Windows file access APIs do not update file
  modification time while the file is open and being actively written to.
  Windows delays updating modification time until the file is closed.
  Therefore you might have to choose a larger time window on Windows
  hosts where files may be open for long time periods.
* Value must be: <number><unit>. For example, "7d" indicates one week.
* Valid units are "d" (days), "h" (hours), "m" (minutes), and "s"
  (seconds).
* Default: unset, meaning there is no threshold and no files are
  ignored for modification time reasons.

have a look at this doc :
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

let me know if this helps!

0 Karma

dkeck
Influencer

I see, you can add

[default]
ignoreOlderThan = X

X can be 7d for example

to your inputs.conf for Ofice 360

This will only apply to new logs.

0 Karma

dkeck
Influencer

HI,

you can set in indexes.conf maxTotalDataSizeMB, this will limit the size of your index. If you know have much data per day you get, you can set this to a limit that fits your need for "only new data"

Dont forget to restart when you set this.

If you already want to limit this before license is consumed have a look at the inputs of your addon and the data that is comming in from Office 360 and decide if you really need all of it. You could than disable inputs or discard data.

0 Karma

templier
Communicator

No, i want install a point in log, for example: do not indexing data older than one week.

0 Karma

templier
Communicator

So, now i go test ignoreOlderThan in inputs.conf

0 Karma

dkeck
Influencer
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!