Archive

Event time confusion

Builder

I'm having a hard time setting up forwarding and event times. Here's my situation. I have an application that creates a csv file in Denver and places it on a collector server in California. I have Splunkforwarder running on the server in California and pushes the csv file to a Splunk server, (Indexer), back in Denver along with multiple other log files. The problem is that when I look at the events in Splunk, the event time is an hour off for the csv files but are fine for the log files.

csv events
Time             Event
11/5/13          2013-11-05 03:07:26, name.name2@place, 10.xx.xx.xx, login, bob
4:07:26.000 AM  

My question is, how do I get the event time to be exactly like the data in the event itself just for the csv files.

On the forwarder:

inputs.conf
[monitor:///directoryName/csvFiles]
sourcetype = csv


outputs.conf
#   Version 6.0
[tcpout]
defaultGroup = splunk

[tcpout:splunk]
server = DenverServerName:9997
useACK = false

On the Splunk main server

inputs.conf
[default]
host = DenverServerName

[splunktcp://9997]
disabled = 0

I've tried to add a props.conf file containing,

[source::/directoryName/csvFiles]
TZ = US/Mountain

but that did not work.

Any help will be greatly appreciated.

Tags (2)
0 Karma
1 Solution

Ultra Champion

It looks to me as if the TZ setting is working as intended - i.e. Splunk is adapting to the fact that there is a timezone difference between California and Colorado. (You are aware that the timestamp inside the event itself will NOT be changed, right?)

Put it like this; if, for example, you are investigating a malware outbreak, and you have logs coming in from London, Moscow, Manila and New Dehli to your Denver-based indexer. You want to know if there were infections in your other sites at the same time, you can search for say 10 - 11 AM. Instead of getting the logs from 10 to 11 for each site in their local time zone (which is not what you want), your search will return the logs for the site-local time period corresponding to 10-11 Denver time.

This is A Good Thing.

Otherwise you would have to craft clumsy searches and keeping track of the timezone differences yourself.

Hope this explains the feature a bit,

K

Btw, you might want to check up on the logs that have the 'correct' time.

EDIT: To elaborate further, it may be interesting to mention that Splunk internally converts timestamps to epoch timestamps (which is the number of seconds passed since midnight Jan 1 1970 in the UTC timezone. epoch as such thus has no timezones (or it can be considered to be a single, worldwide timezone). What is being shown in Splunk, next to your event, is the local time for that event's epoch. If you change the timezone setting for your user account in Splunk, you can see that this changes. Change to PST and the CSV events will look ok, but the others will be 'wrong'.

View solution in original post

0 Karma

Builder

OK, I'm done with this. I don't know what happened, but things started to work. I went back in to my test data and took out the "MST" entry,

Created On, UserName, SourceIP, Event, SNC_Instance
2013-10-13 05:36:44, name1, 45.45.45.45, login, SomeServer
2013-10-13 07:48:48, name2, 56.56.56.56, login, SomeServer
2013-11-07 10:30:00, bob, 99.99.99.99, login, SomeServer

ran it through the Data Preview, and the damn thing started to work correctly;

Timestamp           Event
1   11/7/13 2:14:33.000 PM    Created On, UserName, SourceIP, Event, SNC_Instance
2   10/13/13 5:36:44.000 AM   2013-10-13 05:36:44, name1, 45.45.45.45, login, SomeServer
3   10/13/13 7:48:48.000 AM   2013-10-13 07:48:48, name2, 56.56.56.56, login, SomeServer
4   11/7/13 10:30:45.000 AM   2013-11-07 10:30:45, bob, 99.99.99.99, login, SomeServer

I don't know why or what I did to get it to function correctly, but it's giving me the information I was expecting. Now every time I made a change I would shut down Splunk and run ./splunk clean all, so that was not unique to this last run.

So at this point the Preview Data function is giving me what I want. Now all I have to do is figure out how to get it to work with forwarded data.

0 Karma

Builder

Hmmm.., I seem to still have a little problem. I created some test data and added the Timezone to the timestamp. What I have below is the test data after running through Splunk Data Preview. The indexer/Splunk server is in the MST timezone. I would expect to see the Timestamp and Event times as the same. Any other ideas?

    Timestamp                 Event
2   10/13/13 6:36:44.000 AM   2013-10-13 05:36:44 MST, userName1, 99.99.99.99, login, someserver
3   10/13/13 8:48:48.000 AM   2013-10-13 07:48:48 MST, userName2, 99.99.99.98, login, someserver

==========================================================================================================

OK, now I'm even more confused. I've been working all morning trying to get Splunk to work correctly with no success. My last effort was to create a csv file and add some test data;

Created On, UserName, SourceIP, Event, SNC_Instance
2013-10-13 05:36:44 MST, name1, 45.45.45.45, login, SomeServer
2013-10-13 07:48:48 MST, name2, 56.56.56.56, login, SomeServer
2013-11-07 10:30:00 MST, bob, 99.99.99.99, login, SomeServer

and then run it through the Splunk Data Preview process. Everything was run on the Splunk indexer/server directly and located in the Mountain timezone. The results are inconsistent to say the least;

    Timestamp             Event
1   11/7/13 10:31:05.000 AM   Created On, UserName, SourceIP, Event, SNC_Instance
2   10/13/13 6:36:44.000 AM   2013-10-13 05:36:44 MST, name1, 45.45.45.45, login, SomeServer
3   10/13/13 8:48:48.000 AM   2013-10-13 07:48:48 MST, name2, 56.56.56.56, login, SomeServer
4   11/7/13 10:30:00.000 AM   2013-11-07 10:30:00 MST, bob, 99.99.99.99, login, SomeServer 

The first row is a throwaway. The next two are in error, (an hour off), but the 4th event worked fine with the correct timestamp. Does anyone what's going on here? I can use all the help I can get on this.

0 Karma

Ultra Champion

It looks to me as if the TZ setting is working as intended - i.e. Splunk is adapting to the fact that there is a timezone difference between California and Colorado. (You are aware that the timestamp inside the event itself will NOT be changed, right?)

Put it like this; if, for example, you are investigating a malware outbreak, and you have logs coming in from London, Moscow, Manila and New Dehli to your Denver-based indexer. You want to know if there were infections in your other sites at the same time, you can search for say 10 - 11 AM. Instead of getting the logs from 10 to 11 for each site in their local time zone (which is not what you want), your search will return the logs for the site-local time period corresponding to 10-11 Denver time.

This is A Good Thing.

Otherwise you would have to craft clumsy searches and keeping track of the timezone differences yourself.

Hope this explains the feature a bit,

K

Btw, you might want to check up on the logs that have the 'correct' time.

EDIT: To elaborate further, it may be interesting to mention that Splunk internally converts timestamps to epoch timestamps (which is the number of seconds passed since midnight Jan 1 1970 in the UTC timezone. epoch as such thus has no timezones (or it can be considered to be a single, worldwide timezone). What is being shown in Splunk, next to your event, is the local time for that event's epoch. If you change the timezone setting for your user account in Splunk, you can see that this changes. Change to PST and the CSV events will look ok, but the others will be 'wrong'.

View solution in original post

0 Karma

Builder

Second line:

3 10/13/13 8:48:48.000 AM 2013-10-13 07:48:48 MST, userName2, 99.99.99.98, login, someserver

0 Karma

Builder

Hmmm.., I seem to still have a little problem. I created some test data and added the Timezone to the timestamp. What I have below is the test data after running through Splunk Data Preview. The indexer/Splunk server is in the MST timezone. I would expect to see the Timestamp and Event times as the same. Any other ideas?

    Timestamp                 Event
2   10/13/13 6:36:44.000 AM   2013-10-13 05:36:44 MST, userName1, 99.99.99.99, login, someserver
3
0 Karma

SplunkTrust
SplunkTrust

My splunk (Eastern Time zone) was able to parse as file kept on a machine in Eastern Timezone with MST.

0 Karma

Ultra Champion
0 Karma

Builder

Hmmm, hey somesoni2, would Splunk be able to pick up something like "2013-11-05 03:07:26 MST" instead of the -07:00 value?

0 Karma

Ultra Champion

Ooops. Sorry for lecturing you, and for not reading through the post properly 🙂

Are you sure that the props.conf settings are applied properly for the csv-files. What's the props.conf setting for the other files coming from the same UF?

/k

0 Karma

SplunkTrust
SplunkTrust

Will it be possible for you to make any changes in the application generating the log file?
Best way to resolve this issue will be to have the timezone/offset value logged in the timestamp. For example, instead of logging "2013-11-05 03:07:26" if you can log "2013-11-05 03:07:26 -07:00" in the file, Splunk will automatically have the event time parsed as per timestamp with timezone. [I had similar issue with logs generated in India]

0 Karma

Builder

I am aware of the time delta between California and Colorado, and that's where the problem is. The event has the correct time in the event, 03:07:26. And this is in Mountain time. That is where the event originated.

This is also the time I would like/expect to see in the "Time" column in the search data table. Or do I have this completely wrong?

Is the time in that field normalized to local time or not?

0 Karma

Builder

I'm running Splunk 6.0.

I had the props.conf file in the /opt/splunk/etc/system/local directory on the Splunk server/indexer.

Yes, I downloaded splunkforwarder-6.0-182037.i386.rpm and loaded it.

0 Karma

Legend

What version of Splunk are you running? Where did you put the props.conf file?

Is the Splunk forwarder a Universal Forwarder?

0 Karma