topic Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS) in All Apps and Add-ons

Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

493669 — Fri, 31 Aug 2018 13:01:27 GMT

Hi,
1. Timestamp Issue-
I have installed OMS add-on with an interval of 60 seconds and haven't set any timeformat in props.conf and set default lag of 15 min.
Now I am seeing lag of around 2hours. for ex. in my event:

TimeGenerated: 2018-08-31T12:16:30Z and _time:8/31/18 2:32:01.000 PM

2.Checkpoint issue

I have created an input, and due to our internal problem ports, we got disconnected(for 5hours). And then, once ports got connected, I checked to see if data was still coming but data had stopped indexing. i checked enabling/disabling input, but it had not started indexing.

Then I created a new input with a new index and deleted previous input and indexes and given fetch date as old date.
But in new index timestamp was from date previous input was disabled. It has not been indexing old data. is there any checkpoint value getting stored?

3.logs parsing issue
I haven't faced this yet but as per @thambisetty, Single event is broken into multiple events
Modified Line number 91
from

value = str(data["tables"][0]["rows"][i][n]).replace('"',"'").replace("\\", "\\\\").replace("None", "")

value = str(data["tables"][0]["rows"][i][n]).replace('"',"'").replace("\\", "\\\\").replace("None", "").replace("\r\n","")

This will remove newlines and carriage returns if the field value is dictionary. Due to field values have dictionary and it contains new lines I could see line breaking. This change will avoid line breaking .

@jkat54, @dpanych , @thambisetty Any insight?

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Fri, 31 Aug 2018 13:58:18 GMT

Perfect write up! Now I’ll work on my answers 😉

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

dpanych — Fri, 31 Aug 2018 16:04:47 GMT

Just curious, for issue #1, what timezone do you live in? Also make sure you have your timezone set in your Splunk User Profile settings. I'm not seeing issues on my end with TimeGenerated + _time. For example, when TimeGenerated is 2018-08-31T15:38:07.77Z, _time is 8/31/2018 8:38:07.770 AM. For me, because I'm in Pacific time, TimeGenerated will be 7 hours ahead because it is in UTC.

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

493669 — Fri, 31 Aug 2018 17:51:58 GMT

CEST(Central European Summer Time) is my timezone which is 2 hours ahead of UTC.
so it seems I need to set my Splunk User Profile settings to CEST zone.

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Fri, 31 Aug 2018 18:54:09 GMT

Nice work. I’ll wrap these other bug fixes up along with a solution for the checkpoints in a new version and publish soon.

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Sun, 02 Sep 2018 15:08:36 GMT

Hello,

Please test the latest version, v1.0.3. Found here: https://splunkbase.splunk.com/app/4127

Version 1.0.3:
- Checkpoint bug-fix
- Added TIME_PREFIX and TIME_FORMAT to props
- Fixed line break issue

Thanks to those who reported problems and those who helped solve them as well!

https://answers.splunk.com/answers/684463/issue-in-microsoft-log-analytics-add-onformerly-kn.html
https://answers.splunk.com/answers/680955/why-am-i-getting-a-winsock-error-10053-while-using.html

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

493669 — Mon, 03 Sep 2018 03:54:11 GMT

Thank you @jkat54
Can you please explain about checkpoint bug-fix..

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Tue, 29 Sep 2020 21:07:41 GMT

Before the checkpoints used were named “start_date” and consisted of the start_date supplied to the input OR the last time the input ran.

Now it writes the same date/time values but into checkpoints named after the input_name instead.

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

493669 — Tue, 29 Sep 2020 21:06:31 GMT

thanks .
1.My current issue is my _time and TimeGenerated has 2 hours difference....and I told that my oms data where it is generating and my splunk is in same timezone. If it is so then do I need to set any timezone or default will be fine? also I tried updated props with TIME_PREFIX and TIME_FORMAT..but still I see exact 2 hours lag.

2 is there any file where checkpoint value is stored..
and I am currently using search * query does this means every time it will run it will pick up old data as well ..or it will take only recent data.

3.My query interval is 60 seconds, does it will impact performance?(i am using 1 HF, 1 indexer and 1 SH) and if I set interval as 15 min and my lag time is also 15 min. then will I get data with lag of almost 30 min?

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Mon, 03 Sep 2018 13:18:52 GMT

Did you download and install the latest version of the app (1.0.3)?

I’ve configured the props for you in the latest version.

I thought we concluded your time stamps are fine. They are in UTC as per the log analytics spec.

_time is determined by offsetting the event timestamp with your users time zone setting.

Using the default should be more than fine, it should be perfect.

As for performance with 60s intervals. Seems fine in testing... this should give you about a 16 minute total possible lag with the other default settings.

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

493669 — Mon, 03 Sep 2018 13:30:07 GMT

I did installed latest app..
but still I see 2 hours difference with _time and TimeGenerated.(I have use default timezone)
also I saw TimeGenerated fields value is some time have milliseconds value as well
for ex.TimeGenerated=2018-09-03T12:02:23.897Z
and in props time format is upto seconds only and why I see Z in last instead of any timezone..is it an issue?

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Mon, 03 Sep 2018 13:46:31 GMT

If you set your user time zone to UTC, does _time match the event time?

If so, there’s absolutely no problem with timestamp extractions.

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Mon, 03 Sep 2018 13:59:32 GMT

See this answer for converting to your Timezone:

https://answers.splunk.com/answers/455210/how-to-edit-my-command-to-convert-utc-time-to-est.html

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

493669 — Mon, 03 Sep 2018 14:08:42 GMT

if I set timezone to UTC then _time match with TimeGenerated time i.e. eventtime.
but when I set default time it has difference...can you please explain how it works..
also one more thing before I updated app I was able to see data in last 15 min. but after I updated app I am unable to see data in last 15 min and I do see data in last 20 min...

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Mon, 03 Sep 2018 14:29:19 GMT

Ok, so if an event happens at 4am UTC and at your time it’s 6am CET... _time be 6am if you have your Timezone set to CET. If you have your Timezone set to UTC then it would show up at 4am.

_time is a function of the timestamp detected at index time plus the users timezone modifier.

This way you can search splunk in your timezone and know the event happened at x time local to you...

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Mon, 03 Sep 2018 14:32:25 GMT

You can also modify your loganalytics query to convert the timezone prior to splunk.

But the app specifies UTC at index time so if you change the time returned by the query, you’d have to update props appropriately.

https://docs.microsoft.com/en-us/azure/log-analytics/query-language/datetime-operations#time-zones

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Mon, 03 Sep 2018 14:45:56 GMT

With 1 minute interval and 15 minute lag, data will come with up to 16 minutes or more of lag. So it makes sense that data would not show up if searching for last 15 minutes.

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

493669 — Mon, 03 Sep 2018 17:56:18 GMT

So it seems Splunk considers TimeGenerated time in UTC format. and as my splunk timezone is set to deault i.e. in CET. So _time will be in CET .
But my Timegenerated field i.e. raw data is also in CET timezone.
So in current situation am I getting 2hours(+16 min) old data?
like as you said if event happens at 4 am UTC and my timezone is CET then _time will be 6am but my event also happens at CET timezone only.
if my understanding is correct then I will need to change timezone settings in props.conf?

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Mon, 03 Sep 2018 18:15:05 GMT

Correct,

Just add

TZ = CET

To the props.

Re: Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

jkat54 — Mon, 03 Sep 2018 18:15:55 GMT

Or use the loganalytics query language to convert it UTC before indexing.