topic Re: forwarder only partially forwards data in Getting Data In

forwarder only partially forwards data

ferenc0521 — Thu, 24 May 2018 19:45:05 GMT

Hi, I set up a forwarder, the receiver, the index on the receiving side, and configured the inputs.conf on the forwarder as:

[monitor:///data00/skaushik/cov-platform/config/system.properties]
sourcetype = config_file
[monitor:///data00/skaushik/cov-platform/config/cim.properties]
sourcetype = config_file
[monitor:///data00/skaushik/cov-platform/config/web.properties]
sourcetype = config_file
[monitor:///data00/skaushik/cov-platform/config/pgpass]
sourcetype = config_file
[monitor:///data00/skaushik/cov-platform/database/postgresql.conf]
sourcetype = config_file
[monitor:///data00/skaushik/cov-platform/logs/catalina.out]
[monitor:///data00/skaushik/cov-platform/logs/gc.log*]
sourcetype = gcg1.log
[monitor:///data00/skaushik/cov-platform/logs/cim.log*]
sourcetype = cimlog4j
ignoreOlderThan=1d
[monitor:///data00/skaushik/cov-platform/logs/catalina.log*]
ignoreOlderThan=1d
[monitor:///data00/skaushik/cov-platform/logs/performanceLog.log*]
ignoreOlderThan=1d
[monitor:///data00/skaushik/cov-platform/logs/usageLog.log*]
ignoreOlderThan=1d
[monitor:///data00/skaushik/cov-platform/database/pg_log/postgresql*]
ignoreOlderThan=1d

So the forwarder seems to work- to some extent...
---the sourcetypes are picked up by the receiver, and parsed according to the props.conf definitions - check
---I expected the small config files (*.properties, pgpass) appear at once- none of them do.
---I expected catalina.out and gc.log would appear at once (from the beginning of the file) -they have only limited number of events indexed.
---I expected the monitored files with ignoreOlderThan=1d appear in full at once -they don't seem to.
If the file is younger than a day, it should appear full-it doesn't

the gc.log events started to appear after a day, and even then is about 3 event less than 10%
8000/en-US/manager/search/licenseusage shows minimal ~0 usage.
The files are below 1M

How can I monitor what is actually being detected and sent?

Re: forwarder only partially forwards data

woodcock — Sat, 26 May 2018 14:17:08 GMT

If you have many thousands of files (even ifyou are not monitoring them) at that same directory level or deeper, Spunk will have a problem keeping track of files either by running out of time/CPU or by running out of file descriptors (inodes). Is this your situation?

Re: forwarder only partially forwards data

ferenc0521 — Sat, 26 May 2018 18:03:45 GMT

No, it is 145 files altogether, and most of them is not monitored due to the ignoreOlderThan=1d settings.
the https://:8089/services/admin/inputstatus/TailingProcessor:FileStatus
says it finished reading all the relevant ones.
Over a day from reinstall and start again, and it is only 4 source there, and the short config files totally missing

Re: forwarder only partially forwards data

jkat54 — Sat, 26 May 2018 23:28:32 GMT

Does your inputs.conf have windows formatted line endings because of a copy and paste?

Check index=_internal log_level=error OR log_level=warn for things like "permission" or "skaushik"

Try adding spaces between your stanza names if you dont already have them.

Re: forwarder only partially forwards data

ferenc0521 — Sun, 27 May 2018 17:06:24 GMT

No, and the https://:8089/services/admin/inputstatus/TailingProcessor:FileStatus
says it finished reading all the relevant ones.
It looks like it is picking up the new (rolled logs) (second day of monitoring), but the initial config files (it saig it finished reading has no trace on the indexer

Re: forwarder only partially forwards data

woodcock — Tue, 29 May 2018 00:47:27 GMT

Your problem is probably that you are misunderstanding how ignoreOlderThan works. Once a file is determined to be older than, it gets put to a perminent blacklist and even if it gets updated and is no longer older than your setting, it won't matter; it is blacklisted and none of the data will ever come in, period. The nice thing is if you change the ignoreOlderThan setting and then restart splunk, it should reconsider the files.

Re: forwarder only partially forwards data

ferenc0521 — Tue, 29 May 2018 17:48:50 GMT

The totally missing files don't have ignoreOlderThan settings, so I expected them to be forwarded to the indexer.
Is ignoreOlderThan a global settings instead of per file/folder?
It doesn't seem to handle the rolling of the log files consistently.

cim.log is renamed daily as cim.log.yyyy-mm-dd, and a new cim.log is opened.
performanceLog.log, usageLog.log is the same.

so cim.log and performanceLog.log always have the source= cim.log and performanceLog.log respectively,
however usageLog.log events always have usageLog.log.2018-05-26 as source (?)

I also have problem that the gc.log.0.current has only 2 events forwarded, while the file is obviously has more content.

And forwarder said it DID read the .properties files

Re: forwarder only partially forwards data

ferenc0521 — Tue, 29 May 2018 19:19:50 GMT

update: editing (adding comment first line) cim.properties, web.properties, system.properties, postgresql.conf made them sail over to the target.

the partial send of gc.log and the usageLog.log mystery remains