Props.conf timezone settings for Eastern? And do I need to reboot any peers?

[hrithiktej]

In our Slave-Apps directory on the 2 peers/indexers we have a custom app created by the prev admin which has setting for TZ to UTC for network devices that are on UTC. Now i am adding new data source (i.e. AD security logs) using UFs on DCs and our DCs are all in EST TZ and hence i would need to list EST TZ in the props.conf.

My Questions are

1) Is this the right stanza for EST time entry
[WinEventLog://Security]
TZ = US/Eastern

I understand i will have to do this on master-apps folder on cluster master and then apply config bundle

2) Will this require a reboot of any peers ?

[sbbadri]

@hrithiktej

props.conf seems correct.

[host::yourhostdefinition]
TZ = US/Eastern

or

[source::yousourcedefinition]
TZ = US/Eastern

or
[yoursourcetypedefinition]
TZ = US/Eastern

Can you please check path of the props.conf and check it has enough permission for splunk to read.

CM - $SPLUNK_HOME$/etc/master-apps/_cluster/local/props.conf --- use this location if you don't have separate app.
CM - $SPLUNK_HOME$/etc/master-apps/your app/local/props.conf -- use this location if you have separate app for it.

[hrithiktej]

@sbbadri Thank you fr your reply

mwdbhyat
was very helpful and kind enough to help me so much. This is resolved by changing these two values in my inputs.conf

start_from = newest
current_only = 1

[mwdbhyat · Accepted solution]

Hi there,

1 - Yes thats correct

2 - Yes, the cluster master will initiate a restart of its cluster members once you apply the new cluster bundle. Please see here for what requires a restart and what doesnt..

http://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Updatepeerconfigurations#Restart_or_reload...

[hrithiktej]

This is my props.conf on indexers

[cisco:asa]
TZ = UTC

[cisco:ise:syslog]
TZ = UTC

[cisco:acs]
TZ = UTC

[cisco:ios]
TZ = UTC

[cisco:sourcefire]
TZ = UTC

[f5:bigip:syslog]
TZ = UTC

[pan:log]
TZ = UTC

[WinEventLog://Security]
TZ = US/Eastern

[catchall:catchall]
TZ = UTC

it is not working for events that are coming for [WinEventLog://Security] because if i search for last 15mins or 60 mins i dont get results ONLY when i select last 4hours i can see results. I also tried switching my user time zone from UTC to EST through settings>users>my user timezone as EST and log out/login but still the same issue.

And I have installed Splunk_TA_windows on my UF that sits on DC

Any help will be appreciated

[mwdbhyat]

What happens if you dont apply a TZ ? What time are you getting for your sourcetype then?

[hrithiktej]

@mwdbhyat

I changed these two values in inputs.conf

start_from = newest
current_only = 1

and it resolved my issue THANK YOUUUUU SO MUCH MAN! for some reason i dont see your comment here can you please paste it again I want to mark that as accepted answer.

Thanks once again

[hrithiktej]

Hi, Guys, I have run into issues with UF again this time it has stopped working altogether I have started a new question for this. if you have some time please help

https://answers.splunk.com/answers/578313/splunk-winevtlog-wineventlogchanneldeletecheckpoin.html

thanks in advance

[sbbadri]

@hrithiktej please vote or accept the answer if mwdbhyat solved your issue.

[mwdbhyat]

Haha no worries man !.. Dunno why my comments are disappearing!

[hrithiktej]

Same. I did not apply a TZ before only when this issue started i realised i should enter TZ in props.conf and entering did not make a difference. I also tried entering TZ by creating a props.conf in UFs local but no joy

[mwdbhyat]

Can you send me an example of the search you are running and a snip of the results?

[hrithiktej]

I am simply typing sourcetype="WinEventLog:Security" in search and i do not find anything when i do last15mins or 60 mins i can only see for last 4hrs and event time is real time like if you convert from UTC to IST (which is the TZ I live in).

Also it does not allow me here to paste an image only an url

[mwdbhyat]

Check if there is some kind of indexing lag in your environment -

source=mysource | eval delay_sec=_indextime-_time | timechart min(delay_sec) avg(delay_sec) max(delay_sec) by host

Alternatively - has that DC host been set a timezone in another app? Can you run a btool to check that?

[mwdbhyat]

Use this search to verify the source type, the time stamp detected (_time), the time of the user on the search head (now), and the time zone applied (date_zone)

source=mysource host=myhost | eval delay_sec=_indextime-_time | convert ctime(_indextime) AS indextime | eval now=now() | table _time indextime now date_zone source sourcetype host

[hrithiktej]

Thanks, I ran the queries and below are the results, there seems to be a 3-4hrs delay and time diff I do not know why, if I run these queries for other sources it does not show any delay or time difference.

source=mysource | eval delay_sec=_indextime-_time | timechart min(delay_sec) avg(delay_sec) max(delay_sec) by host

_time

2017-09-25 06:45:00

avg(delay_sec)

8618

max(delay_sec)
8656

min(delay_sec)
8593

source=mysource host=myhost | eval delay_sec=_indextime-_time | convert ctime(_indextime) AS indextime | eval now=now() | table _time indextime now date_zone source sourcetype host

_time

2017-09-25 06:59:30

indextime

09/25/2017 09:16:39

source

WinEventLog:Security

Also please note to troubleshoot this now I have changed the timezone for all my Splunk servers to match with my Domain controller so now both indexer and source have same TZ = EST but still I am not able to search logs in last 60mins or 15mins.

[hrithiktej]

@mwdbhyat

I see your comment in email notification but not here.

And this is my inputs I am using the default inputs.conf from the Splunk_TA_Windows app.

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)"
index = wineventlog
renderXml=false

[mwdbhyat]

You should make changes in the local folder, just in case someone comes along and creates a stanza for the same sourcetype and overwrites your settings in default(it wont fix your problem but is a best practice).

Regarding you inputs it looks fine - but there is clearly a lag in indexing.. Can your environment handle the amount of data that is flowing into your indexers from wineventlog?

This article has a few tricks you can try:

https://docs.splunk.com/Documentation/Splunk/6.6.3/Troubleshooting/Troubleshootingeventsindexingdela...

[hrithiktej]

Yeah thx man I have made it in local only and i do not see accept option for your comment How do i accept your answer?

[mwdbhyat]

You could accept the main initial answer - would sitll guide people here.. there should be an "accept as answer" option

[mwdbhyat]

Can you send a snip of your input stanza for the security logs in wineventlog ?