Archive
Highlighted

Splunk is not indexing my text file

Path Finder

Hi Team,

It looks like splunk is not indexing my log file, If I create a test file then it is indexing it but not my log file.

input.conf

[monitor://D:\logs]
disabled = false
index = prd
sourcetype = sched

Log File sample:

2017-03-24 00:00:00,848 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:00:20,927 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:00:41,161 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:01:01,208 [Thread-2] WARN com.JobInitiator- executed CALL

Please help me on this issue. Thank You!

Tags (2)
0 Karma
Highlighted

Re: Splunk is not indexing my text file

Splunk Employee
Splunk Employee

Hi Abilan1

You can try checking index=_internal tailreader with your log path in the GUI to see perhaps the tail reader is ignoring the file due to CRC checks or permissions otherwise, or check

./splunk list inputstatus to check for the files and the tailprocessor's status.

0 Karma
Highlighted

Re: Splunk is not indexing my text file

Path Finder

Hi ,

Thanks for your reply, I have verified the permission it looks fine. Also If I create any other test file in that folder that is getting indexed properly. so it doesn't seem to be a permission issue.

Checked the tail-processor's status, it says finished reading. No errors related to CRC.

Regards,
Abilan

0 Karma
Highlighted

Re: Splunk is not indexing my text file

Splunk Employee
Splunk Employee

if the files are read, then maybe we are looking in the wrong spot?

does searching index=prd sourcetype=sched source=*\logs* searched over alltime find them?

0 Karma
Highlighted

Re: Splunk is not indexing my text file

Path Finder

Hi,

Hi,

Yes I see that logs are indexing, but If I want to search for current log then I have to search for last 4 hours. I mean indexing time stamp is 4 hour behind.

If I search for last 4 hours in the search, I can see the latest logs there.

We have the same forwarder on linux machines and that is giving proper time stamps. This is happening on windows server's universal forwarder. All our universal forwarder machines are in same time zone but not indexer.

Thanks ,
Abilan

0 Karma
Highlighted

Re: Splunk is not indexing my text file

Splunk Employee
Splunk Employee

Sounds like you need to configure your sourcetype with timestamp settings. There are a few best practices when it comes to defining a sourcetype...but for now lets focus on the props.conf for timestamp recogonition.

What timezone should these logs be in?

can you share the props.conf you are using for sourcetype sched ?

./splunk btool props list sched --debug

Try adding TZ= <logTimezone> to the props to help Splunk determine the correct stamp.

0 Karma
Highlighted

Re: Splunk is not indexing my text file

Splunk Employee
Splunk Employee

Try this and set your timezone according to the machines that are spitting the sched logs to your windows box. Setting SHOULD_LINEMERGE, TIME_FORMAT, TIME_PREFIX, MAX_TIMESTAMP_LOOKAHEAD, TZ, and LINE_BREAKER in props.conf, should become part of all your sourcetyping. Splunk is really good at auto recognizing things..but you can gain indexing performance by not making it work so hard. Try using the add data wizard! it makes creating these configs real easy and lets you validate your choices.

[ sched ]
SHOULD_LINEMERGE=false
TIME_FORMAT=%Y-%m-%d %H:%M:%S,%f
TIME_PREFIX=^
TZ=UTC
MAX_TIMESTAMP_LOOKAHEAD=25
0 Karma
Highlighted

Re: Splunk is not indexing my text file

Path Finder

Hi ,

My Indexer is in GMT. but all my forwarders are in Brazil time zone.

I didn't change anything in props.conf file after installation.

Also please let me know which timezone I need to add and any standard format for the same?

Thanks for your help!

Regards,
Abilan

0 Karma
Highlighted

Re: Splunk is not indexing my text file

Splunk Employee
Splunk Employee

You need to set the timezone to that of the machines spitting the logs. IT best practice puts all machines in UTC, but alas, no one is perfect ;).

If i run date on the machine generating the logs, I want to match that timezone as that is the TZ it will stamp the logs with.

As long as Splunk knows the proper TZ at indextime, the GUI settings will allow the user to set their local timezone and the GUI will adjust for them. But you gotta get the initial TZ correct.

TZ =
* The algorithm for determining the time zone for a particular event is as
follows:
* If the event has a timezone in its raw text (for example, UTC, -08:00),
use that.
* If TZ is set to a valid timezone string, use that.
* If the event was forwarded, and the forwarder-indexer connection is using
the 6.0+ forwarding protocol, use the timezone provided by the forwarder.
* Otherwise, use the timezone of the system that is running splunkd.
* Defaults to empty.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Propsconf

0 Karma
Highlighted

Re: Splunk is not indexing my text file

Path Finder

Hi ,

I have updated the below in my props.conf on forwarder machine.

[ sched ]
SHOULDLINEMERGE=false
TIME
FORMAT=%Y-%m-%d %H:%M:%S,%f
TIMEPREFIX=^
TZ=UTC-3
MAX
TIMESTAMP_LOOKAHEAD=25

And restarted the splunk forwarder service, but still I don't see any change in time from Splunk web.

Regards,
Abilan

0 Karma