topic Re: how to index data from a log file which got generated when my forwarder was down? in Getting Data In

how to index data from a log file which got generated when my forwarder was down?

saifuddin9122 — Wed, 28 Jun 2017 13:57:47 GMT

Hello All,

i have a dumb question,

i have few servers which will have heavy traffic and with log files rotating for every few minutes. the problem i had, my forwarder was down for few mins in that down time log files got rotated and new file is generated and my forwarder is reading the current log file but when i search data for the time when my forwarder was down resulting in zero events. does it means i lost that data. if it is so then how should i read that data and for future purpose how should i address this problem.

here are my example log file names
webapp.log (currently written )
webapp.log.1 (rolled)
webapp.log.2 (rolled)
webapp.log.3 (rolled)
webapp.log.4 (rolled)

here are my inputs.conf

[monitor:///var/log/web/app/webapp.log]
index=main
sourcetype=web_application

Thanks for the help.

Re: how to index data from a log file which got generated when my forwarder was down?

jtacy — Wed, 28 Jun 2017 14:35:53 GMT

Consider changing your input to:

[monitor:///var/log/web/app/webapp.log*]
index=main
sourcetype=web_application

If the logs are still in the folder this will capture them, but if the logs aren't on disk anymore I would imagine that the data is lost. The above configuration will help prevent this situation in the future. If you're concerned about duplicate events, consider that Splunk uses the data at the beginning of the file to determine if it's already read it, not the file name (by default).

There should be no negative effects from this change unless the star will match files you don't want to index. If that's the case you might want to review the whitelist and blacklist options on your input to fine tune what you capture:
http://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/Inputsconf#MONITOR: