It looks like splunk is not indexing my log file, If I create a test file then it is indexing it but not my log file.
disabled = false
index = prd
sourcetype = sched
Log File sample:
2017-03-24 00:00:00,848 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:00:20,927 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:00:41,161 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:01:01,208 [Thread-2] WARN com.JobInitiator- executed CALL
Please help me on this issue. Thank You!
You can try checking
index=_internal tailreader with your log path in the GUI to see perhaps the tail reader is ignoring the file due to CRC checks or permissions otherwise, or check
./splunk list inputstatus to check for the files and the tailprocessor's status.
Thanks for your reply, I have verified the permission it looks fine. Also If I create any other test file in that folder that is getting indexed properly. so it doesn't seem to be a permission issue.
Checked the tail-processor's status, it says finished reading. No errors related to CRC.
if the files are read, then maybe we are looking in the wrong spot?
index=prd sourcetype=sched source=*\logs* searched over alltime find them?
Yes I see that logs are indexing, but If I want to search for current log then I have to search for last 4 hours. I mean indexing time stamp is 4 hour behind.
If I search for last 4 hours in the search, I can see the latest logs there.
We have the same forwarder on linux machines and that is giving proper time stamps. This is happening on windows server's universal forwarder. All our universal forwarder machines are in same time zone but not indexer.
Sounds like you need to configure your sourcetype with timestamp settings. There are a few best practices when it comes to defining a sourcetype...but for now lets focus on the props.conf for timestamp recogonition.
What timezone should these logs be in?
can you share the props.conf you are using for sourcetype
./splunk btool props list sched --debug
TZ= <logTimezone> to the props to help Splunk determine the correct stamp.
Try this and set your timezone according to the machines that are spitting the sched logs to your windows box. Setting
LINE_BREAKER in props.conf, should become part of all your sourcetyping. Splunk is really good at auto recognizing things..but you can gain indexing performance by not making it work so hard. Try using the add data wizard! it makes creating these configs real easy and lets you validate your choices.
[ sched ] SHOULD_LINEMERGE=false TIME_FORMAT=%Y-%m-%d %H:%M:%S,%f TIME_PREFIX=^ TZ=UTC MAX_TIMESTAMP_LOOKAHEAD=25
My Indexer is in GMT. but all my forwarders are in Brazil time zone.
I didn't change anything in props.conf file after installation.
Also please let me know which timezone I need to add and any standard format for the same?
Thanks for your help!
You need to set the timezone to that of the machines spitting the logs. IT best practice puts all machines in UTC, but alas, no one is perfect ;).
If i run
date on the machine generating the logs, I want to match that timezone as that is the TZ it will stamp the logs with.
As long as Splunk knows the proper TZ at indextime, the GUI settings will allow the user to set their local timezone and the GUI will adjust for them. But you gotta get the initial TZ correct.
* The algorithm for determining the time zone for a particular event is as
* If the event has a timezone in its raw text (for example, UTC, -08:00),
* If TZ is set to a valid timezone string, use that.
* If the event was forwarded, and the forwarder-indexer connection is using
the 6.0+ forwarding protocol, use the timezone provided by the forwarder.
* Otherwise, use the timezone of the system that is running splunkd.
* Defaults to empty.
I have updated the below in my props.conf on forwarder machine.
[ sched ]
And restarted the splunk forwarder service, but still I don't see any change in time from Splunk web.