how can I troubleshoot logs not getting forwarded to indexers

[maxruas]

Hi,

I have splunkforwarder installed to monitor garbage collection for glassfish servers A and B side.
I have a dashboard on splunk for that and some of it stopped working "Real time GC pauses by type" as logs stopped coming in for this.
every thing looks correctly configured and looks same as the B side that works.

Inputs.conf:

[monitor:///zones/*/root/opt/SUNWappserver/glassfish/domains/scdomain/logs/gc.log*]
sourcetype=glassfish:gc
blacklist=(.*\.gz$|.*\.xml$|.*\.1\d*$|.*\.2\d*$|.*\.3\d*$|.*\.4\d*$|.*\.5\d*$|.*\.6\d*$|.*\.7\d*$|.*\.8\d*$|.*\.9\d*$)
whitelist = (.*\.log$|.*\.*log.0)
ignoreOlderThan = 30m
index=nld_cdxx_app
host_segment = 2

[monitor:///zones/*/root/opt/SUNWappserver/glassfish/domains/scdomain/logs/access/*.txt]
sourcetype=glassfish:access
blacklist=(.*\.gz$|.*\.xml$|.*\.1\d*$|.*\.2\d*$|.*\.3\d*$|.*\.4\d*$|.*\.5\d*$|.*\.6\d*$|.*\.7\d*$|.*\.8\d*$|.*\.9\d*$)
whitelist = (.*\.txt$)
index=nld_cdxx_app
host_segment = 2
ignoreOlderThan = 60m

I can see stuff with sourcetype=glassfish:access for the Last 15 minutes but nothing since 03.12.2016 for glassfish:gc

I have checked that the path is correct by checking the global zone(solaris) that file exists:

/root/opt/SUNWappserver/glassfish/domains/scdomain/logs.

I have also done telnet tests for all the indexers on port 9997 and works and the deployment server on 8089 works also.

[woodcock]

You can use ingnoreOlderThan but if you do, beware that it does not work the way most people think that it does: once Splunk ignores any file for this reason at any time, it is put into a permanent blacklist and it will never be examined again, even if new data goes into it! This is likely your problem so just remove ingnoreOlderThan. If the files still don't go, then you may need to change their names to get past the previous effects of the blacklisting.

[maxruas]

Hi
thank you for you kind response, Is there a way that I can check what is been permanently black listed.
If so can I remove entries from that list?

thanks

[woodcock]

I have no idea.