Logging thousend files via Splunk Forwarder causes high CPU load

[tfechner]

Hi there,

we have a oracle logging directory with thousend .aud files for logging to Splunk.
Each day over 700 new files will be created.
We experience a heavy workload on the system caused by the splunkd process.

We think splunkd monitores ALL files and after some weeks a hugh bunch of filemonitoring threads are occuping the CPU.

How can we tell splunk not to monitor already indexed files and only have a look on new created. The closed file will never be changed anymore.

Our inputs.conf:

[monitor:///oracle/Q*/trace/audit/*.aud]
sourcetype=oracle:audit:text
whitelist = \w.+.aud
ignoreOlderThan=7d
index=oracle_sap
disabled = false

[ddrillic]

What's the forwarder version? - Universal Forwarder Using High CPU?

[tfechner]

we use the newst one 7.1.2.X

[amiftah]

I think you have to create your own script to delete/move/rename the indexed files.

[sudosplunk]

Hi,

How are your new files named? Any thing to differentiate new and old.

[tfechner]

fielname_structure:
AppID_OracleID_timestamp.aud
with:
appid= P56
OracleID: 53457673
time: 2018073134756825434785

[sudosplunk]

The naming doesn't seem to be helpful. Since new files are created every day, decrease ignoreOlderThan to 2 or 3 days. This can reduce load.

[tfechner]

fielname_structure:
AppID_OracleID_timestamp.aud
with:
appid= P56
OracleID: 53457673
time: 2018073134756825434785