Issues getting logs from Microsoft Log Analytics Add-on

[kforr74]

Hoping you can help me @jkat54. We are working with the Azure team on testing the pull of log from the Cloud to one of their standalone Search Heads. They provided the inputs from Azure (Workspace ID, Subscription, Tenant, etc) with the exception of the Log Analytics query. Is that something they are supposed to define or set and provide to us, or something we are just supposed to define for searching? I just put SecurityBaseline in there because it is required, but get nothing. I have tried SecurityEvent as well and a few other examples in the documentation. Also, what would you recommend we set for the Interval and Start Date to test?

[jkat54 · Accepted solution]

Hello,

As per our emails we found that the app is hardcoded for the non-Goverment Azure cloud, and you need the Gov Cloud version of the API instead.

In short you need to use this API:

api.loganalytics.us

and the app is trying to use this API:

api.loganalytics.io

To fix this you'll have to modify code in the app.

Please modify TA-ms-loganalytics\bin\input_module_log_analytics.py as follows:

LINE 45:        resource  = 'https://api.loganalytics.io/'

Becomes (not actually used in the code that i see, but you can still update it)

LINE 45:        resource  = 'https://api.loganalytics.us/'

And

LINE 49:        token_response = context.acquire_token_with_client_credentials('https://api.loganalytics.io/', application_id, application_key)

Becomes

Line 49:        token_response = context.acquire_token_with_client_credentials('https://api.loganalytics.us/', application_id, application_key)

And

LINE 59:        uri_base = 'https://api.loganalytics.io/'

Becomes:

LINE 59:        uri_base = 'https://api.loganalytics.us/'

[kforr74]

I made those changes, and guess what, I am now getting logs! Thanks again!

7/15/19
5:56:55.000 PM

{ [-]
Category: Alert

count_: 506

}
Show as raw text
host = vac20logecs205.va.gov source = log_analytics://splunk_va_azure_cloud sourcetype = loganalytics
7/15/19
5:56:55.000 PM

{ [-]
Category: Policy
count_: 430301
}
Show as raw text
host = vac20logecs205.va.gov source = log_analytics://splunk_va_azure_cloud sourcetype = loganalytics
7/15/19
5:56:55.000 PM

{ [-]
Category: Administrative
count_: 708207
}
Show as raw text
host = vac20logecs205.va.gov source = log_analytics://splunk_va_azure_cloud sourcetype = loganalytics

[jkat54]

Thanks for troubleshooting with me all the way through to the end!

If you appreciate the effort I put in, feel free to let others know by recommending me on linkedin.

https://linkedin.com/in/global-splunk-consultant

Thanks in advance!

[jkat54]

For simplicity, i have updated the code and hosted it on pastebin so you can just copy and replace:

https://pastebin.com/TYsV6iAa

[jkat54]

@jaxjohnny2000 got his working by using the workspace ID, you're already using the ID. Have you tried using the name instead?

[jkat54]

See this:

https://answers.splunk.com/answers/700795/v103-seeing-a-get-error-when-collecting-events.html

tested with several other searches - and all fail when requesting SecurityEvent - eg :

 SecurityEvent
 | top 10 by TimeGenerated

fails - and

 AzureActivity
 | top 10 by TimeGenerated

writes to index

[kforr74]

Thanks @jkat54 The errors have reduces when I use
AzureActivity
top 10 by TimeGenerated
as the query, however, I am still receiving errors and nothing in the index yet:

7/11/19
3:49:21.776 PM

07-11-2019 15:49:21.776 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ERRORlocal variable 'data' referenced before assignment
host = ***** source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
7/11/19
3:49:21.747 PM

07-11-2019 15:49:21.747 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" UnboundLocalError: local variable 'data' referenced before assignment
host = ***** source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
7/11/19
3:49:21.747 PM

07-11-2019 15:49:21.747 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" for i in range(len(data["tables"][0]["rows"])):
host = ***** source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
7/11/19
3:49:21.747 PM

07-11-2019 15:49:21.747 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/splunk/etc/apps/TA-ms-loganalytics/bin/input_module_log_analytics.py", line 86, in collect_events
host = ***** source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
7/11/19
3:49:21.747 PM

07-11-2019 15:49:21.747 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" input_module.collect_events(self, ew)
host = ***** source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
7/11/19
3:49:21.747 PM

07-11-2019 15:49:21.747 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py", line 96, in collect_events
host =***** source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
7/11/19
3:49:21.747 PM

07-11-2019 15:49:21.747 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" self.collect_events(ew)
host = ***** source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd
7/11/19
3:49:21.747 PM

07-11-2019 15:49:21.747 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 127, in stream_events
host = ***** source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd

[jkat54]

I recommend whatever suites you for start date and interval.

Just note that azure storage underneath this is "eventual consistency", and that's why we added the lag to it.

Most users have reported lag of 15 - 30 minutes works well with intervals of 15 or 30 minutes.

[jaxjohnny2000]

Tons of errors for me, with almost every query I put In here

file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py", line 96, in collect_events

file=connectionpool.py:_make_request:400 | https://api.loganalytics.io:443 "POST /v1/workspaces/SPZE2HSTPRD001OMSLogAnalytics/query HTTP/1.1" 404 84

TokenRequest:No user_id passed for cache query

file=connectionpool.py:_make_request:387 | "GET /servicesNS/nobody/TA-ms-loganalytics/storage/collections/data/TA_ms_loganalytics_checkpointer/SoftPro_Log_Analytics_Input HTTP/1.1" 404 140