It looks like splunk is not indexing my log file, If I create a test file then it is indexing it but not my log file.
disabled = false
index = prd
sourcetype = sched
Log File sample:
2017-03-24 00:00:00,848 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:00:20,927 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:00:41,161 [Thread-2] WARN com.JobInitiator- executed CALL
2017-03-24 00:01:01,208 [Thread-2] WARN com.JobInitiator- executed CALL
Please help me on this issue. Thank You!
When searching select ALL TIME, this has tripped me up before especially if the test logs are old.
I'm not sure if you want to ingest the whole directory or a file named logs.
Try using creating a folder logs and putting your test file inside and then use the following stanza.
You could also confirm connectivity by using telnet
telnet SPLUNK_IP 9997
Another thing you could do is a one shot command. This will ingest the file regardless of whether the file has been ingested before.
SPLUNK_HOME\bin\splunk add oneshot D:\logs -index pad -sourcetype shed
You can try checking
index=_internal tailreader with your log path in the GUI to see perhaps the tail reader is ignoring the file due to CRC checks or permissions otherwise, or check
./splunk list inputstatus to check for the files and the tailprocessor's status.
your user is set to UTC in the Splunk GUI, correct? click the username dropdown (top right of your screen) > Account Settings > Set timezone
With my user set in UTC, I see these events at 3AM GMT in my Time field....is that what you are trying to achieve? Note the time in the event has not changed, just the Time field now has a local conversion in the gui.
can you please collect the output of
./splunk btool props list sched --debug on your forwarder and your indexer? ( not sure of your setup, I think you said forwarder)
you have the
sched sourcetype updated in both a forwarder and indexer, correct?
`[splunker@n00bserver bin]$ ./splunk btool props list sched --debug
/home/splunker/splunk/etc/apps/search/local/props.conf TIMEFORMAT = %Y-%m-%d %H:%M:%S,%f
/home/splunker/splunk/etc/apps/search/local/props.conf TIMEPREFIX = ^
/home/splunker/splunk/etc/system/default/props.conf TRANSFORMS =
/home/splunker/splunk/etc/system/default/props.conf TRUNCATE = 10000
/home/splunker/splunk/etc/apps/search/local/props.conf TZ = Brazil/East
Brasil event time, UTC Splunk web user, splunker sitting in Canada 😉
Thanks for your reply, I have verified the permission it looks fine. Also If I create any other test file in that folder that is getting indexed properly. so it doesn't seem to be a permission issue.
Checked the tail-processor's status, it says finished reading. No errors related to CRC.
Yes I see that logs are indexing, but If I want to search for current log then I have to search for last 4 hours. I mean indexing time stamp is 4 hour behind.
If I search for last 4 hours in the search, I can see the latest logs there.
We have the same forwarder on linux machines and that is giving proper time stamps. This is happening on windows server's universal forwarder. All our universal forwarder machines are in same time zone but not indexer.
Sounds like you need to configure your sourcetype with timestamp settings. There are a few best practices when it comes to defining a sourcetype...but for now lets focus on the props.conf for timestamp recogonition.
What timezone should these logs be in?
can you share the props.conf you are using for sourcetype
./splunk btool props list sched --debug
TZ= <logTimezone> to the props to help Splunk determine the correct stamp.
My Indexer is in GMT. but all my forwarders are in Brazil time zone.
I didn't change anything in props.conf file after installation.
Also please let me know which timezone I need to add and any standard format for the same?
Thanks for your help!
You need to set the timezone to that of the machines spitting the logs. IT best practice puts all machines in UTC, but alas, no one is perfect ;).
If i run
date on the machine generating the logs, I want to match that timezone as that is the TZ it will stamp the logs with.
As long as Splunk knows the proper TZ at indextime, the GUI settings will allow the user to set their local timezone and the GUI will adjust for them. But you gotta get the initial TZ correct.
* The algorithm for determining the time zone for a particular event is as
* If the event has a timezone in its raw text (for example, UTC, -08:00),
* If TZ is set to a valid timezone string, use that.
* If the event was forwarded, and the forwarder-indexer connection is using
the 6.0+ forwarding protocol, use the timezone provided by the forwarder.
* Otherwise, use the timezone of the system that is running splunkd.
* Defaults to empty.
I have updated the below in my props.conf on forwarder machine.
[ sched ]
And restarted the splunk forwarder service, but still I don't see any change in time from Splunk web.
I have updated this for UTC in my props.conf on forwarder machine.
Still it is not working as expected. Logs are still coming in Brazil Time Zone.
Try this and set your timezone according to the machines that are spitting the sched logs to your windows box. Setting
LINE_BREAKER in props.conf, should become part of all your sourcetyping. Splunk is really good at auto recognizing things..but you can gain indexing performance by not making it work so hard. Try using the add data wizard! it makes creating these configs real easy and lets you validate your choices.
[ sched ] SHOULD_LINEMERGE=false TIME_FORMAT=%Y-%m-%d %H:%M:%S,%f TIME_PREFIX=^ TZ=UTC MAX_TIMESTAMP_LOOKAHEAD=25