Hi, I set up a forwarder, the receiver, the index on the receiving side, and configured the inputs.conf on the forwarder as:
[monitor:///data00/skaushik/cov-platform/config/system.properties]
sourcetype = config_file
[monitor:///data00/skaushik/cov-platform/config/cim.properties]
sourcetype = config_file
[monitor:///data00/skaushik/cov-platform/config/web.properties]
sourcetype = config_file
[monitor:///data00/skaushik/cov-platform/config/pgpass]
sourcetype = config_file
[monitor:///data00/skaushik/cov-platform/database/postgresql.conf]
sourcetype = config_file
[monitor:///data00/skaushik/cov-platform/logs/catalina.out]
[monitor:///data00/skaushik/cov-platform/logs/gc.log*]
sourcetype = gcg1.log
[monitor:///data00/skaushik/cov-platform/logs/cim.log*]
sourcetype = cimlog4j
ignoreOlderThan=1d
[monitor:///data00/skaushik/cov-platform/logs/catalina.log*]
ignoreOlderThan=1d
[monitor:///data00/skaushik/cov-platform/logs/performanceLog.log*]
ignoreOlderThan=1d
[monitor:///data00/skaushik/cov-platform/logs/usageLog.log*]
ignoreOlderThan=1d
[monitor:///data00/skaushik/cov-platform/database/pg_log/postgresql*]
ignoreOlderThan=1d
So the forwarder seems to work- to some extent...
---the sourcetypes are picked up by the receiver, and parsed according to the props.conf definitions - check
---I expected the small config files (*.properties, pgpass) appear at once- none of them do.
---I expected catalina.out and gc.log would appear at once (from the beginning of the file) -they have only limited number of events indexed.
---I expected the monitored files with ignoreOlderThan=1d appear in full at once -they don't seem to.
If the file is younger than a day, it should appear full-it doesn't
the gc.log events started to appear after a day, and even then is about 3 event less than 10%
8000/en-US/manager/search/licenseusage shows minimal ~0 usage.
The files are below 1M
How can I monitor what is actually being detected and sent?
Your problem is probably that you are misunderstanding how ignoreOlderThan
works. Once a file is determined to be older than
, it gets put to a perminent blacklist and even if it gets updated and is no longer older than
your setting, it won't matter; it is blacklisted and none of the data will ever come in, period. The nice thing is if you change the ignoreOlderThan
setting and then restart splunk, it should reconsider the files.
The totally missing files don't have ignoreOlderThan
settings, so I expected them to be forwarded to the indexer.
Is ignoreOlderThan
a global settings instead of per file/folder?
It doesn't seem to handle the rolling
of the log files consistently.
cim.log is renamed daily as cim.log.yyyy-mm-dd, and a new cim.log is opened.
performanceLog.log, usageLog.log is the same.
so cim.log and performanceLog.log always have the source= cim.log and performanceLog.log respectively,
however usageLog.log events always have usageLog.log.2018-05-26 as source (?)
I also have problem that the gc.log.0.current has only 2 events forwarded, while the file is obviously has more content.
And forwarder said it DID read the .properties files
update: editing (adding comment first line) cim.properties, web.properties, system.properties, postgresql.conf made them sail over to the target.
the partial send of gc.log and the usageLog.log mystery remains
Does your inputs.conf have windows formatted line endings because of a copy and paste?
Check index=_internal log_level=error OR log_level=warn
for things like "permission" or "skaushik"
Try adding spaces between your stanza names if you dont already have them.
No, and the https://:8089/services/admin/inputstatus/TailingProcessor:FileStatus
says it finished reading all the relevant ones.
It looks like it is picking up the new (rolled logs) (second day of monitoring), but the initial config files (it saig it finished reading has no trace on the indexer
If you have many thousands of files (even ifyou are not monitoring them) at that same directory level or deeper, Spunk will have a problem keeping track of files either by running out of time/CPU or by running out of file descriptors (inodes). Is this your situation?
No, it is 145 files altogether, and most of them is not monitored due to the ignoreOlderThan=1d settings.
the https://:8089/services/admin/inputstatus/TailingProcessor:FileStatus
says it finished reading all the relevant ones.
Over a day from reinstall and start again, and it is only 4 source there, and the short config files totally missing