Issue in Microsoft Log Analytics Add-on(Formerly Know as OMS)

[493669]

Hi,
1. Timestamp Issue-
I have installed OMS add-on with an interval of 60 seconds and haven't set any timeformat in props.conf and set default lag of 15 min.
Now I am seeing lag of around 2hours. for ex. in my event:

TimeGenerated: 2018-08-31T12:16:30Z and _time:8/31/18 2:32:01.000 PM

2.Checkpoint issue

I have created an input, and due to our internal problem ports, we got disconnected(for 5hours). And then, once ports got connected, I checked to see if data was still coming but data had stopped indexing. i checked enabling/disabling input, but it had not started indexing.

Then I created a new input with a new index and deleted previous input and indexes and given fetch date as old date.
But in new index timestamp was from date previous input was disabled. It has not been indexing old data. is there any checkpoint value getting stored?

3.logs parsing issue
I haven't faced this yet but as per @thambisetty, Single event is broken into multiple events
Modified Line number 91
from

value = str(data["tables"][0]["rows"][i][n]).replace('"',"'").replace("\\", "\\\\").replace("None", "")

to

value = str(data["tables"][0]["rows"][i][n]).replace('"',"'").replace("\\", "\\\\").replace("None", "").replace("\r\n","")

This will remove newlines and carriage returns if the field value is dictionary. Due to field values have dictionary and it contains new lines I could see line breaking. This change will avoid line breaking .

@jkat54, @dpanych , @thambisetty Any insight?

[jkat54 · Accepted solution]

Hello,

Please test the latest version, v1.0.3. Found here: https://splunkbase.splunk.com/app/4127

Version 1.0.3:
- Checkpoint bug-fix
- Added TIME_PREFIX and TIME_FORMAT to props
- Fixed line break issue

Thanks to those who reported problems and those who helped solve them as well!

https://answers.splunk.com/answers/684463/issue-in-microsoft-log-analytics-add-onformerly-kn.html
https://answers.splunk.com/answers/680955/why-am-i-getting-a-winsock-error-10053-while-using.html

[anwar114]

Hi, @jkat54

i have version  1.0.3.  There is line break issue happening still. appreciate a start up where to for fixing it. Let me know more details needed. 

Thanks in advance 

[anwar114]

hi, 

I am having single event breaking issue for couple of events.

versions is 1.0.3.

Would you be able to suggest ways to tackle the issue.

Only below errors from logs , but these WARN message doesn't coincide with the event which is broken :

WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 11423 - data_source="log_analytics://xxxxxxxx", data_host="xxxxxxxxx", data_sourcetype="loganalytics"

WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. Defaulting to timestamp of previous event (Thu Jun 18 09:14:20 2020). Context: source=log_analytics://xxxxxxxxx|host=xxxxxxxx|loganalytics|\n 5 similar messages suppressed. First occurred at: Thu Jun 18 09:07:52 2020

Thanks in Advance.

[493669]

Thank you @jkat54
Can you please explain about checkpoint bug-fix..

[jkat54]

Before the checkpoints used were named “start_date” and consisted of the start_date supplied to the input OR the last time the input ran.

Now it writes the same date/time values but into checkpoints named after the input_name instead.

[493669]

thanks .
1.My current issue is my _time and TimeGenerated has 2 hours difference....and I told that my oms data where it is generating and my splunk is in same timezone. If it is so then do I need to set any timezone or default will be fine? also I tried updated props with TIME_PREFIX and TIME_FORMAT..but still I see exact 2 hours lag.

2 is there any file where checkpoint value is stored..
and I am currently using search * query does this means every time it will run it will pick up old data as well ..or it will take only recent data.

3.My query interval is 60 seconds, does it will impact performance?(i am using 1 HF, 1 indexer and 1 SH) and if I set interval as 15 min and my lag time is also 15 min. then will I get data with lag of almost 30 min?

[jkat54]

Did you download and install the latest version of the app (1.0.3)?

I’ve configured the props for you in the latest version.

I thought we concluded your time stamps are fine. They are in UTC as per the log analytics spec.

_time is determined by offsetting the event timestamp with your users time zone setting.

Using the default should be more than fine, it should be perfect.

As for performance with 60s intervals. Seems fine in testing... this should give you about a 16 minute total possible lag with the other default settings.

[493669]

I did installed latest app..
but still I see 2 hours difference with _time and TimeGenerated.(I have use default timezone)
also I saw TimeGenerated fields value is some time have milliseconds value as well
for ex.TimeGenerated=2018-09-03T12:02:23.897Z
and in props time format is upto seconds only and why I see Z in last instead of any timezone..is it an issue?

[jkat54]

If you set your user time zone to UTC, does _time match the event time?

If so, there’s absolutely no problem with timestamp extractions.

[493669]

if I set timezone to UTC then _time match with TimeGenerated time i.e. eventtime.
but when I set default time it has difference...can you please explain how it works..
also one more thing before I updated app I was able to see data in last 15 min. but after I updated app I am unable to see data in last 15 min and I do see data in last 20 min...

[jkat54]

With 1 minute interval and 15 minute lag, data will come with up to 16 minutes or more of lag. So it makes sense that data would not show up if searching for last 15 minutes.

[493669]

So it seems Splunk considers TimeGenerated time in UTC format. and as my splunk timezone is set to deault i.e. in CET. So _time will be in CET .
But my Timegenerated field i.e. raw data is also in CET timezone.
So in current situation am I getting 2hours(+16 min) old data?
like as you said if event happens at 4 am UTC and my timezone is CET then _time will be 6am but my event also happens at CET timezone only.
if my understanding is correct then I will need to change timezone settings in props.conf?

[jkat54]

Correct,

Just add

TZ = CET

To the props.

[jkat54]

Or use the loganalytics query language to convert it UTC before indexing.

[493669]

I have added TZ= CET in props.conf
but I not understanding in my raw data TimeGenerated field still comes in UTC...ex.
I checked one event in azure Log analytics whose timeGenerated field is 6 am but when this event comes in Splunk then in raw event I see TimeGenerated field as 4 am ye my _time is shows correct as 6am....So why in raw data this field is getting converted to UTC although I set timezone.

[jkat54]

https://docs.microsoft.com/en-us/azure/log-analytics/query-language/datetime-operations#time-zones

All time in loganalytics is UTC.

[493669]

thanks a lot....so it seems TZ in props.conf won't affect timezone to change in event from UTC to CET.
So I can add in query to convert UTC to CET as

search *| extend localTimestamp = TimeGenerated + 2h

[jkat54]

Cheers! Thanks for accepting my answer.

[jkat54]

Can you please accept my answer for the current question?

Everything is working correctly.

[jkat54]

Because everything from loganalytics is in UTC.

Did you read the link I provided above? It talks all about time in log analytics.

Again,
If you want log analytics to force a different timezone, change the query...

Please read the link.