Intermittent log data ingestion- Why aren't logs ingested regularly?

[drikusc]

I have an issue where the logs aren't ingested regularly.

The log file updates every 5 minutes with the same line entries, and will roll over to a new file end of day.

-rw-r--r--+ 1 novlua novlua 160416 Sep 18 23:55 iga_check_2022-09-18.log
-rw-r--r--+ 1 novlua novlua 197664 Sep 19 23:55 iga_check_2022-09-19.log
-rw-r--r--+ 1 novlua novlua 241056 Sep 20 23:55 iga_check_2022-09-20.log
-rw-r--r--+ 1 novlua novlua 241056 Sep 21 23:55 iga_check_2022-09-21.log
-rw-r--r--+ 1 novlua novlua 241056 Sep 22 23:55 iga_check_2022-09-22.log
-rw-r--r--+ 1 novlua novlua 271783 Sep 23 23:55 iga_check_2022-09-23.log
-rw-r--r--+ 1 novlua novlua 326880 Sep 24 23:55 iga_check_2022-09-24.log
-rw-r--r--+ 1 novlua novlua 326880 Sep 25 23:55 iga_check_2022-09-25.log
-rw-r--r--+ 1 novlua novlua 124783 Sep 26 09:06 iga_check_2022-09-26a.log
-rw-r--r--+ 1 novlua novlua 271376 Sep 26 23:55 iga_check_2022-09-26.log
-rw-r--r--+ 1 novlua novlua 248613 Sep 27 23:55 iga_check_2022-09-27.log
-rw-r--r--+ 1 novlua novlua 97092 Sep 28 09:35 iga_check_2022-09-28.log

Log file entries example:

09:35:02 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/mudad/changeset_*.*
09:35:02 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/mudad/queue/*.csv
09:35:02 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/mudad/work/*.csv
09:35:02 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/mudad/completed/*.csv
09:35:02 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/isimprod/changeset_*.*
09:35:02 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/isimprod/queue/*.csv
09:35:02 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/isimprod/completed/*.csv
09:35:02 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/sanbussroles/changeset_*.*
09:35:02 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/sanbussroles/queue/*.csv
09:35:02 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/sanbussroles/completed/*.csv
09:40:01 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/mudad/changeset_*.*
09:40:01 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/mudad/queue/*.csv
09:40:01 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/mudad/work/*.csv
09:40:01 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/mudad/completed/*.csv
09:40:01 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/isimprod/changeset_*.*
09:40:01 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/isimprod/queue/*.csv
09:40:01 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/isimprod/completed/*.csv
09:40:01 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/sanbussroles/changeset_*.*
09:40:01 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/sanbussroles/queue/*.csv
09:40:01 Processing: /opt/netiq/idm/apps/tomcat/fulfillment/sanbussroles/completed/*.csv

I noted when requesting a forced entry, it gets picked up.

inputs.conf

[monitor:///opt/netiq/idm/apps/tomcat/fulfillment/logs/*.log]
# blacklist = (\.gz)
whitelist = \.log$|\.txt$
# crcSalt = <SOURCE>
# disabled = false
index = IG_RequestLog
sourcetype = IG:RequestLogCheck
time_before_close = 10

[inventsekar]

Hi... Some more details pls...

1. is it a clustered or a single server environment? indxer cluster and/or SHC ?!?!

2. did you upgrade recently? 

3. i see the logs are locally available on the indexer, right..  

4. if 3 is yes, then, how the logs are sent to the indexer, pls suggest.

thanks..

[drikusc]

Hi, 

Thank you for the response.

This is within a Distributed deployment.

Deployment Server, Indexer, Heavy Forwarder all running on it's own server, with a Splunk Forwarder installed on the above mentioned server where the logs reside.

We upgraded to v8.1.6 around 5 months ago.

This is for a new request of logs to be ingested into an existing index where the logs are coming in as normal.