topic Splunk doesn't index new created logfile. in Getting Data In

Splunk doesn't index new created logfile.

chrisitanmoleck — Fri, 21 Mar 2014 07:24:40 GMT

Hello,

I observe a ftp logfile. The server creates one logfile for one day.
At midnight there will be a new file created. But this is not readable for the forwarder.
If I restart the forwarder, everything is fine, and will be forwarded.

Example:

20th March 11PM: Logfile is forwarding to the indexer
21th March 01AM: No forwarding
21th March 08AM: Forwarder restart
21th March 08AM: Logfile is forwarding to the indexer

The splunkd logfile has three entries:

03-21-2014 00:01:19.664 +0100 WARN FileClassifierManager - The file 'path_to_logfile' is invalid. Reason: binary
03-21-2014 00:01:19.664 +0100 INFO TailingProcessor - Ignoring file 'path_to_logfile' due to: binary
03-21-2014 04:31:09.931 +0100 ERROR TailingProcessor - Ignoring path="path_to_logfile" due to: Bug: tried to check/configure STData processing but have no pending metadata.

inputs.conf

[monitor://path_to_logfile]
disabled = false
sourcetype = FTPLOG
crcSalt = <SOURCE>
Charset = Auto

props.conf

[monitor://path_to_logfile]
NO_BINARY_CHECK = true

Could you help me?
Christian

Re: Splunk doesn't index new created logfile.

MuS — Fri, 21 Mar 2014 08:43:26 GMT

Hi chrisitanmolecki,

Splunk checks the first bits of your file, could there be an invisible control character at the start of your file? That happened to me one time...
Just edited the file with an hex editor and check if there are some strange characters like xA0

hope this helps ...

cheers, MuS

Re: Splunk doesn't index new created logfile.

chrisitanmoleck — Fri, 21 Mar 2014 08:47:25 GMT

The logfile starts with:

#Software: Microsoft Internet Information Services 6.0

in a hex-editor shows like:

2353 6F66 7477....

Re: Splunk doesn't index new created logfile.

MuS — Fri, 21 Mar 2014 08:55:53 GMT

Just saw that your Charset is A in inputs.conf instead of props.conf and B is wrong. It should be charset not Charset. See docs about binary file error http://docs.splunk.com/Documentation/Splunk/6.0.2/Troubleshooting/Binaryfileerror

Re: Splunk doesn't index new created logfile.

kristian_kolb — Fri, 21 Mar 2014 10:10:15 GMT

Also (but maybe it's just a typo) you have a props.conf stanza that says [monitor://path_to_log], when it should say [FTPLOG] (i.e. just the sourcetype). The [monitor]-stanzas are for inputs.conf only.

Re: Splunk doesn't index new created logfile.

chrisitanmoleck — Fri, 21 Mar 2014 10:38:24 GMT

I changed the configs.

First results on monday morning.

Nice Weekend
Christian

Re: Splunk doesn't index new created logfile.

chrisitanmoleck — Mon, 24 Mar 2014 07:09:45 GMT

It works. Thank you MuS and kristian.kolb!!!