Splunk doesn't index new created logfile.

[chrisitanmoleck]

Hello,

I observe a ftp logfile. The server creates one logfile for one day.
At midnight there will be a new file created. But this is not readable for the forwarder.
If I restart the forwarder, everything is fine, and will be forwarded.

Example:

20th March 11PM: Logfile is forwarding to the indexer
21th March 01AM: No forwarding
21th March 08AM: Forwarder restart
21th March 08AM: Logfile is forwarding to the indexer

The splunkd logfile has three entries:

03-21-2014 00:01:19.664 +0100 WARN FileClassifierManager - The file 'path_to_logfile' is invalid. Reason: binary
03-21-2014 00:01:19.664 +0100 INFO TailingProcessor - Ignoring file 'path_to_logfile' due to: binary
03-21-2014 04:31:09.931 +0100 ERROR TailingProcessor - Ignoring path="path_to_logfile" due to: Bug: tried to check/configure STData processing but have no pending metadata.

inputs.conf

[monitor://path_to_logfile]
disabled = false
sourcetype = FTPLOG
crcSalt = <SOURCE>
Charset = Auto

props.conf

[monitor://path_to_logfile]
NO_BINARY_CHECK = true

Could you help me?
Christian

[MuS · Accepted solution]

Hi chrisitanmolecki,

Splunk checks the first bits of your file, could there be an invisible control character at the start of your file? That happened to me one time...
Just edited the file with an hex editor and check if there are some strange characters like xA0

hope this helps ...

cheers, MuS

[chrisitanmoleck]

It works. Thank you MuS and kristian.kolb!!!

[chrisitanmoleck]

I changed the configs.

First results on monday morning.

Nice Weekend
Christian

[kristian_kolb]

Also (but maybe it's just a typo) you have a props.conf stanza that says [monitor://path_to_log], when it should say [FTPLOG] (i.e. just the sourcetype). The [monitor]-stanzas are for inputs.conf only.

[MuS]

Just saw that your Charset is A in inputs.conf instead of props.conf and B is wrong. It should be charset not Charset. See docs about binary file error http://docs.splunk.com/Documentation/Splunk/6.0.2/Troubleshooting/Binaryfileerror

[chrisitanmoleck]

The logfile starts with:

#Software: Microsoft Internet Information Services 6.0

in a hex-editor shows like:

2353 6F66 7477....