topic Re: How can I configure Splunk to properly index my file in production? in Getting Data In

How can I configure Splunk to properly index my file in production?

chrisduimstra — Wed, 14 Sep 2016 21:39:40 GMT

I have a file in production that appears to not be indexed as running a search for index=<name> returns no results. The file has no header and has the following field format.

2016-04-05 02:51:05.4457|Error|Error receiving response: Connection timeout

I have tested this file on my locally installed instance by replacing the first pipe with a space as to isolate the time field as such.

2016-04-05 02:51:05.4457 Error|Error receiving response: Connection timeout

This worked on my local instance. However, I am unable to modify the production file. Is there a way to mimic this change through settings to work on the production file?

EDIT:
I created a new .txt file and copied a couple logs over to the new file. I then added another stanza to monitor that file, and the new file was indexed but not the old. I have tried .txt and .log suffixes. Here is the current inputs.conf

[monitor://C:\Program Files (x86)\Sell\LPClient.txt]
index = LP
sourcetype = LPClient_log
disabled = 0

[monitor://C:\Program Files (x86)\Sell\NewTextDocument.txt]
index = LP
sourcetype = LPClient_log
disabled = 0

Re: How can I configure Splunk to properly index my file in production?

ShaneNewman — Wed, 14 Sep 2016 21:52:05 GMT

Try running index=_internal sourcetype=splunkd host=<hostname> *<filename>* and see what is returned.

You might see something like this: 09-14-2016 21:50:06.008 +0000 INFO TailingProcessor - Ignoring file '/var/log/folder/file.log' due to: binary

Re: How can I configure Splunk to properly index my file in production?

chrisduimstra — Wed, 14 Sep 2016 21:56:37 GMT

That search returned no results.

Re: How can I configure Splunk to properly index my file in production?

twinspop — Tue, 29 Sep 2020 11:01:13 GMT

A few things:

Your timestamp has no timezone. Your log could be interpreted either in the past, or out in the future depending on how the indexer's timezone is set. Or how the TZ is set in props.conf for this source(type). I would expand my search time frame to include at least 24 hours on either side of the expected.
Along those same lines, I can't tell for sure from your timestamp if it's %Y-%m-%d or %Y-%d-%m. Be sure your props.conf explicitly sets your TIME_FORMAT properly for this source(type).
Your timestamp is far in the past. Splunk has a setting to ignore events older than specified time period. I would check the MAX_DAYS_AGO setting on the indexer to be sure it isn't so low as to exclude these events. (Default is 2000, but worth checking anyway.)
EDIT: I don't think the inclusion exclusion of the pipe is making any difference to parsing.

Re: How can I configure Splunk to properly index my file in production?

chrisduimstra — Wed, 14 Sep 2016 23:13:09 GMT

My search is running over all time.
I set TIME_FORMAT = %Y-%m-%d in props.conf for the source
As for the timestamp I listed, that was from the beginning of the log, the latest entry was from yesterday. My search is still returning, no results found.

Re: How can I configure Splunk to properly index my file in production?

MuS — Thu, 15 Sep 2016 01:26:50 GMT

Hi, please look at your original question https://answers.splunk.com/answers/451910/how-to-monitor-a-single-file-to-be-indexed-by-modi.html for additional hints & tips