Reporting

Splunk Logs Reporting GMT Time Only

kmsnyde
Explorer

I have an indexer running on Windows 2008 R2, receiving logs from linux and windows machines. All computers are set to EST. However, when I conduct a search of my logs Splunk reports the time as GMT. In my search strings, I have to add the following to the get the correct time of the event: ""eval date_hour=date_hour-5". I shouldn't have to do this since, as I stated, all computers are set to EST and all are actually in the same time zone. I have also ensured that under the Splunk administrator tab, Time zone is set to (GMT-05:00) Eastern Time (US & Canada).

What could cause this issue? Appreciate anyones assistance. Thanks in advance.

Tags (1)
0 Karma
1 Solution

lukejadamec
Super Champion

Here is how you can work with the _time field:

sourcetype=Linux-Audit.Log NOT type=CONFIG_CHANGE | eval HourMinute=strftime(_time, "%H:%M") |eval tHour=strftime(_time, "%H") | eval tMinute=strftime(_time, "%M") |table _time HourMinute tHour tMinute

View solution in original post

lukejadamec
Super Champion

By the way, the underscore in front of _time means that it is hidden. That is why you don't see it listed. However, it is not really hidden. The time displayed to the left of the event is the _time field.

0 Karma

kmsnyde
Explorer

lukejadamec,

Using "table _time with the example your provided answered the mail and I have obviously learned something new about using _time. Everyone's assistance was greatly appreciated. Many Thanks.

0 Karma

lukejadamec
Super Champion

Good news. I remember when I found _time...
Feel free to accept my answer:)

0 Karma

lukejadamec
Super Champion

Here is how you can work with the _time field:

sourcetype=Linux-Audit.Log NOT type=CONFIG_CHANGE | eval HourMinute=strftime(_time, "%H:%M") |eval tHour=strftime(_time, "%H") | eval tMinute=strftime(_time, "%M") |table _time HourMinute tHour tMinute

kmsnyde
Explorer

Thanks for your time lukejadamec. I will give that a try when I can implement it and give feedback on it.

0 Karma

lukejadamec
Super Champion

It is a default field created by Splunk. You just call it in your search or output. Try this:
sourcetype=Linux-Audit.Log NOT type=CONFIG_CHANGE |table _time _raw

kmsnyde
Explorer

Thansk lukejadamec. I don't have the _time field for my events to select from in the left panel. How can I do that.

0 Karma

lukejadamec
Super Champion

The date_hour field is not adjusted for timezone. It is the date_hour of the actual raw event timestamp. If you want work with the timezone corrected time, then you should extract your time values from the _time field, which is corrected for your timezone.

kmsnyde
Explorer

lukejadamec - using the shortend search you suggested, splunk itself shows the correct time of the event (i.e., to the left of the linux event is in EST.) However, if I go to "View all XX fields" in the left pane of splunk, and select "date_hour", it comes up to the bottom right of my linux event next to host, sourcetype, source, "date_hour" and has time in GMT. For example, Splunk time of event (left of the log entry) after I search is 9:00:01.837 AM; however, the "date_hour" under the event is "14" (GMT time). This is my problem area. Thanks for your assistance.

0 Karma

lukejadamec
Super Champion

What does Splunk show for event time when you run this search?
sourcetype=Linux-Audit.Log NOT type=CONFIG_CHANGE | Transaction by msg
Compare Splunk's event time to the converted epoch time.

0 Karma

lukejadamec
Super Champion

Use an epoch timestamp converter for 1331897657.359, not sure what the :8435 represents. The unix timestamp is GMT.
http://www.epochconverter.com/

kmsnyde
Explorer

Chris,

Thanks for your continued feedback. My logs are in the format your represented. I'm not sure how (or where i.e., linux or splunk) you are asking to check the time zone. I don't know how to read the data/time stamp after msg=audit(....). Appreciate your help. Thanks.

0 Karma

chris
Motivator

Did you restart splunk after you made the changes to props.conf? Does your linux audit log contain time stamps?
If the log contains a unix epoch timestamp splunk might recognize it an maybe the system is writing in GMT ...
If your logs have the following format check out the timezone of the unix time stamp in the log:
type=USER_AUTH msg=audit(1331897657.359:8435): user pid=15610 uid=0 auid=4294967295 subj=system_u:system_r:unconfined_t:s0-s0:c0.c1023 msg='PAM: authentication acct="?" : exe="/usr/sbin/sshd" (hostname=c165-130.i02-2.onvol.net, addr=213.165.165.130, terminal=ssh res=failed)'

0 Karma

kmsnyde
Explorer

RESULT: I get all data in a table and the time is (H)11 (M) 20 (Actual time in EST is 0620). All computers set to EST.

VARIATION: In my search above after "eval a2=mvindex(a2, -2)" if I insert | eval data_hour=date_hour-5 | my time in my search resutl is (H)6 (M)20. (original time from result minus 5 hours (GMT - 5hrs) This is the correct time of the event in EST.

I don't understand why I have to insert that variation piece to get the right time to show when all computers are set to EST in the first place.

0 Karma

kmsnyde
Explorer

The search for my ISSO looks like this (names changed to save the innocent) without an eval for date_hour:

sourcetype=Linux-Audit.Log NOT type=CONFIG_CHANGE | Transaction by msg | search (a0=rm .sh or .py) | eval a0=mvindex(a0, -1) | eval a2=mvindex(a2, -1) | stats list(date_hour) as Hour, list(date_minute) as Minute, list(date_month) as Month, list(date_year) as Year by host, auid, a0, a2 | rename host as Host | rename a0 as CMD | rename a2 as File

See next comment for continuation please.

0 Karma

lukejadamec
Super Champion

I'm not sure were on the same page.
eval date_hour=date_hour-5 will not change the timestamp in the event, nor will it change the event time displayed in the search results. It will change the date_hour field, which is extracted from the raw event data.
For a problem event, can you post the following info:
The event data copied from the search window.
The event time to the left of the event data copied from the search window.
The date_hour value for the event without the eval statement.
And, if you can run w32tm /tz from a cmd.exe window that would be good - verify there 300 minute offset.

0 Karma

kmsnyde
Explorer

No timezones are different across my network. All events occur only in EST. I am running Splunk 5.0.1. I have reviewed the timezone information pertaining to adding 2 lines in props.conf, however, after adding a Host line and TZ line for US/Eastern, there was no change. When I do a search for use of the "rm" command for instance, I have the results come back with hour minute and day. The problem is that hour comes back with GMT and I still have to add a line in my search for "eval date_hour=date_hour-5".

Appreciate your continuing help.

0 Karma

lukejadamec
Super Champion

Do the events contain timezones?
What version of Splunk are you running?
Have you reviewed the 'how Splunk assigns timezones info?
http://docs.splunk.com/Documentation/Splunk/6.0.1/data/Applytimezoneoffsetstotimestamps

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...