topic Re: I'm trying to determine if by date_time stamps if we are getting the logs we should be getting in Splunk Search

I'm trying to determine if by date_time stamps if we are getting the logs we should be getting

clunde — Fri, 28 Aug 2020 15:33:10 GMT

Hello,

I'm trying to determine if we are getting all the TrendMicro logs by comparing what's in Splunk and what's in Trend. There are 2 date/time stamps in the Splunk logs which I assume 1 is the actual event date/time and the other is the Splunk index date/time.

I've ran the following 2 searches which return the same date_time stamps but I would expect to be different since the 2 date/times are different.

Times:

Aug 28 11:18:43 x.x.x.x Aug 28 15:12:19

index=trendmicro | eval mytime=strftime(_time,"%Y-%m-%dT%H:%M:%S.%Q %Z")

2020-08-28T11:18:43.000 EDT

index=trendmicro | eval mytime=strftime(_indextime,"%Y-%m-%dT%H:%M:%S.%Q %Z")

2020-08-28T11:18:43.000 EDT

How can I pull/report on both of these fields with both of the date_time stamps so we can determine we are getting all logs as well as if the indexer(s) are under resourced?

Re: I'm trying to determine if by date_time stamps if we are getting the logs we should be getting

isoutamo — Fri, 28 Aug 2020 15:41:56 GMT

_time is (usually) when event is originally created on source system.

_indextime is when event is ingested to splunk and written to splunk index.

Usually those two should be quite similar (difference some seconds), but if there are some issues to collect and deliver events to splunk there could be long difference between those.

Another reason for that is wrongly configured TZ (time zone) information and/or your equipment don't use same time source to sync their time (ntp is suitable for that).

You could put those to report just like you already show in your guestion. Just use different names for those.

Best and easiest way to solve why there are difference between those is take MC (monitoring console) into use and look e.g. Settings -> MC -> Indexing -> performance is there any bottle necks sawn.

There are lot of other questions about this issue and also excellent .conf presentations how to solve this which you could found by using google.

r. Ismo

Re: I'm trying to determine if by date_time stamps if we are getting the logs we should be getting

clunde — Fri, 28 Aug 2020 16:29:24 GMT

Thank you for the information!

Do you know if the first time stamp is the _time or if it's the _indextime?

Thank you!

Re: I'm trying to determine if by date_time stamps if we are getting the logs we should be getting

isoutamo — Fri, 28 Aug 2020 16:35:29 GMT

Usually the earlier timestamp is the event's creation time and the second one indexing time. Of course if your event ( source system) has wrong timezone then it could be otherwise.

Have you raw event and props.conf (/transforms.conf) where we could try to figure it out?

r. Ismo