Log Analytics (OMS) API updated - NEW working code in here.

[dpanych]

I've worked with the app developer, Michael @jkat54, and we have updated the python script, input_module_oms_inputs.py, to use the Microsoft's new Log Analytics/OMS API. We have not updated the GUI so it will still contain some "*" required fields in the Inputs section, which are really not required - just fill out all the fields and the app should work. As an FYI, you may need to play with the query and escape some characters if something isn't working right. Also, the top parameter (Max Count) is no longer required, but if you still want to spit out a max amount of event, update your query to include " | limit # ".

I recommend reinstalling the app from clean and do not use the old /local/*.conf - setup everything from new.
Here's the steps I took to get it working again:
1.Re-downloaded app from Splunkbase

1. Inputs > Create New Inputs
   Name: oms
   Interval: 60
   Index: main
   Resource Group: xx
   Workspace ID: Use Workspace ID, NOT Name - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   Subscription ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   Tenant ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   Application ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   Application Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
   OMS Query: search DeviceName == "ext-device-01"
   Max Count: 1000
   Start Date: 14/08/2018 01:01:01

2. Changed source code

3. Worked! Data starts flowing in

Code with indention issues corrected and hosted on pastebin.com:
https://pastebin.com/D3dPNBXJ

import os
import sys
import time
import datetime
from splunklib.modularinput import *

def validate_input(helper, definition):
    inputs=helper.get_input_stanza()
    for input_name, input_item in inputs.iteritems():
        max_count = str(input_item["max_count"])
        start_date = str(input_item["start_date"])
        if int(max_count) <= 0:
            helper.log_error("Max count must be greater than zero (0); found max_count=" + str(max_count))
        try:
            valid_date = datetime.datetime.strptime(start_date, '%d/%m/%Y %H:%M:%S')
        except ValueError:
            helper.log_error("Start date must be in the format of dd/mm/yyyy hh:mm:ss")
    pass

def collect_events(helper, ew):
    import adal
    import json
    import requests
    # Go through each input for this modular input
    inputs=helper.get_input_stanza()
    for input_name, input_item in inputs.iteritems():
        # Get the values, cast them as floats
        resource_group = str(input_item["resource_group"])
        workspace = str(input_item["workspace_id"])
        query = str(input_item["oms_query"])
        max_count = str(input_item["max_count"])
        subscription_id = str(input_item["subscription_id"])
        tenant_id = str(input_item["tenant_id"])
        application_id = str(input_item["application_id"])
        application_key = str(input_item["application_key"])

        # Date and delta
        if helper.get_check_point('last_date'):
            start_datetime = datetime.datetime.strptime(helper.get_check_point('last_date'),'%d/%m/%Y %H:%M:%S')
        else:
            start_datetime = datetime.datetime.strptime(str(input_item['start_date']),'%d/%m/%Y %H:%M:%S')

        now = datetime.datetime.utcnow() - datetime.timedelta(minutes=15)
        now_dt = now.replace(microsecond=0)

        # URLs for authentication
        authentication_endpoint = 'https://login.microsoftonline.com/'
        resource  = 'https://api.loganalytics.io/'

        # Get access token
        context = adal.AuthenticationContext('https://login.microsoftonline.com/' + tenant_id)
        token_response = context.acquire_token_with_client_credentials('https://api.loganalytics.io/', application_id, application_key)
        access_token = token_response.get('accessToken')

        # Add token to header
        headers = {
            "Authorization": 'Bearer ' + access_token,
            "Content-Type":'application/json'
        }

        # URLs for retrieving data
        uri_base = 'https://api.loganalytics.io/'
        uri_api = 'v1/'
        uri_workspace = 'workspaces/' + workspace + '/'
        uri_area = "query"
        uri = uri_base + uri_api + uri_workspace + uri_area

        # Build search parameters from query details
        search_params = {
            "query": query,
            "timespan": start_datetime.strftime('%Y-%m-%dT%H:%M:%S') + '/' + now_dt.strftime('%Y-%m-%dT%H:%M:%S')
        }

        # Send post request
        response = requests.post(uri,json=search_params,headers=headers)

        # Response of 200 if successful
        if response.status_code == 200:
            # If debug, log event
            helper.log_debug('OMSInputName="' + str(input_name) + '" status="' + str(response.status_code) + '" step="Post Query" search_params="' + str(search_params) + "'")
            # Parse the response to get the ID and status
            data = response.json()
        else:
            # Request failed
            helper.log_error('OMSInputName="' + str(input_name) + '" status="' + str(response.status_code) + '" step="Post Query" response="' + str(response.text) + '"')

        #Building proper json format from original request
        #First loop checks how many events returned is in response
        for i in range(len(data["tables"][0]["rows"])):
            data1 = "{"
            #This nested loop goes through each field, in each event, and concatenates the field name to the field value
            for n in range(len(data["tables"][0]["rows"][i])):
                field = str(data["tables"][0]["columns"][n]["name"])
                value = str(data["tables"][0]["rows"][i][n]).replace('"',"'").replace("\\", "\\\\").replace("None", "")
                if value == "":
                    continue
                else:
                    data1 += '"%s":"%s",' % (field, value)
            data1 += "}"
            data1 = data1.replace(",}", "}")
            event = Event()
            event.stanza = input_name
            event.data = data1
            ew.write_event(event)

        #Delta
        state = now_dt.strftime("%d/%m/%Y %H:%M:%S")
        helper.save_check_point('last_date', state)

[jkat54]

I have released a new app for log analytics, it is currently pending approval:

https://splunkbase.splunk.com/app/4127

Once approved, the link above will work.

Also, here is the code @dpanych shared above but with the indention issues corrected and hosted on pastebin.com:

https://pastebin.com/D3dPNBXJ

[ips_mandar]

@dpanych @jkat54 Thanks
I have updated above code in input_module_oms_inputs.py and now I am getting below errors:-

08-14-2018 06:21:51.679 +0200 ERROR ModularInputs - Unable to initialize modular input "oms_inputs" defined inside the app "TA-OMS_Inputs": Introspecting scheme=oms_inputs: script running failed (exited with code 1).
08-14-2018 06:21:51.679 +0200 ERROR ModularInputs - Introspecting scheme=oms_inputs: script running failed (exited with code 1).
08-14-2018 06:19:52.655 +0200 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n  File "D:\Splunk\Python-2.7\Lib\site-packages\splunk\admin.py", line 130, in init\n    hand.execute(info)\n  File "D:\Splunk\Python-2.7\Lib\site-packages\splunk\admin.py", line 594, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "D:\Splunk\etc\apps\TA-OMS_Inputs\bin\ta_oms_inputs\splunk_aoblib\rest_migration.py", line 38, in handleList\n    AdminExternalHandler.handleList(self, confInfo)\n  File "D:\Splunk\etc\apps\TA-OMS_Inputs\bin\ta_oms_inputs\splunktaucclib\rest_handler\admin_external.py", line 40, in wrapper\n    for entity in result:\n  File "D:\Splunk\etc\apps\TA-OMS_Inputs\bin\ta_oms_inputs\splunktaucclib\rest_handler\handler.py", line 118, in wrapper\n    raise RestError(exc.status, exc.message)\nRestError: REST Error [404]: Not Found -- HTTP 404 Not Found -- {"messages":[{"type":"ERROR","text":"Not Found"}]}\n

Currently I am using below inputs.conf -

[oms_inputs://test_oms]
application_id = xxxx-xxxx-xxxxx
application_key = ********
index = main
interval = 60
max_count = 1000
oms_query = search *
resource_group = xxxx
start_date = 10/08/2018 03:43:00
subscription_id = xxxx-xxxx-xxxxxx
tenant_id = xxxxx-xxxx-xxxx
workspace_id = xxxxx-xxxx-xxxx

After updating code I am unable to view inputs from Web UI . It shows message as Unable to initialize modular input "oms_inputs" defined inside the app "TA-OMS_Inputs": Introspecting scheme=oms_inputs: script running failed (exited with code 1).
@dpanych not sure what I am missing..

[ips_mandar]

also in Inputs.conf - In workspace_id whether we need to provide workspace Name or ID.
If i remove all inputs still error message shows Unable to initialize modular input "oms_inputs" defined inside the app "TA-OMS_Inputs": Introspecting scheme=oms_inputs: script running failed (exited with code 1)

[jkat54]

Sorry it is supposed to be the workspace name. Not the ID.

[phularah]

I used @jkat54 's new app. (https://splunkbase.splunk.com/app/4127). I used workspace ID instead of workspace name and it is now working. If You use workspace name, it gives winsock error 10053.

Once again Thanks @jkat54 and @dpanych.

[dpanych]

I got mine working with Workspace ID.

[ips_mandar]

thank you.. but before adding input I am getting error message Unable to initialize modular input "oms_inputs" defined inside the app "TA-OMS_Inputs": Introspecting scheme=oms_inputs: script running failed (exited with code 1)
I have only modified input_module_oms_inputs.py ..Is there anything I need to modify?

[phularah]

I am also getting same message. I am also unable to access UI of app where we need to provide inputs, but we can work around that by using conf files.

[jkat54]

The exit code you’re getting typically means there is a syntax error which can happen very easily with copy pasta python.

I’ll repackage the app ASAP so you can just download and install.

[payal4296]

@jkat54 @dpanych

Hi, I have checked each line and could not find any syntax errors or problem with indents. I modified the script, but am still getting same exit code.

Also, I am using query as search * and I am putting workspace name instead of workspaceID in inputs.conf.

Could you please help on this. Any definitive date you are going to repackage app.

---

import os
import sys
import time
import datetime

def validate_input(helper, definition):
    inputs=helper.get_input_stanza()
    for input_name, input_item in inputs.iteritems():
        max_count = str(input_item["max_count"])
        start_date = str(input_item["start_date"])
        if int(max_count) <= 0:
            helper.log_error("Max count must be greater than zero (0); found max_count=" + str(max_count))
        try:
            valid_date = datetime.datetime.strptime(start_date, '%d/%m/%Y %H:%M:%S')
        except ValueError:
            helper.log_error("Start date must be in the format of dd/mm/yyyy hh:mm:ss")
    pass

def collect_events(helper, ew):
    import adal
    import json
    import requests
    # Go through each input for this modular input
    inputs=helper.get_input_stanza()
    for input_name, input_item in inputs.iteritems():
        # Get the values, cast them as floats
        resource_group = str(input_item["resource_group"])
        workspace = str(input_item["workspace_id"])
        query = str(input_item["oms_query"])
        max_count = str(input_item["max_count"])
        subscription_id = str(input_item["subscription_id"])
        tenant_id = str(input_item["tenant_id"])
        application_id = str(input_item["application_id"])
        application_key = str(input_item["application_key"])

        # Date and delta
        if helper.get_check_point('last_date'):
            start_datetime = datetime.datetime.strptime(helper.get_check_point('last_date'),'%d/%m/%Y %H:%M:%S')
        else:
            start_datetime = datetime.datetime.strptime(str(input_item['start_date']),'%d/%m/%Y %H:%M:%S')
        now = datetime.datetime.now()
        now_dt = now.replace(microsecond=0)

        # URLs for authentication
        authentication_endpoint = 'https://login.microsoftonline.com/'
        resource  = 'https://management.azure.com/'

        # Get access token
        context = adal.AuthenticationContext('https://login.microsoftonline.com/' + tenant_id)
        token_response = context.acquire_token_with_client_credentials('https://management.azure.com/', application_id, application_key)
        access_token = token_response.get('accessToken')

        # Add token to header
        headers = {
            "Authorization": 'Bearer ' + access_token,
            "Content-Type":'application/json'
        }

        # URLs for retrieving data
        uri_base = 'https://management.azure.com'
        uri_api = 'api-version=2017-10-01-preview'
        uri_subscription = 'https://management.azure.com/subscriptions/' + subscription_id
        uri_resourcegroup = uri_subscription + '/resourceGroups/'+ resource_group
        uri_workspace = uri_resourcegroup + '/providers/Microsoft.OperationalInsights/workspaces/' + workspace
        #uri_search = uri_workspace + '/search'

        # Build search parameters from query details
        search_params = {
            "query": query,
            "top": max_count,
            "start": start_datetime.strftime('%Y-%m-%dT%H:%M:%S'),
            "end": now_dt.strftime('%Y-%m-%dT%H:%M:%S')
            }

        # Build URL and send post request
        #uri = uri_search + '?' + uri_api
        uri = uri_workspace + '?' + uri_api
        response = requests.post(uri,json=search_params,headers=headers)

        # Response of 200 if successful
        if response.status_code == 200:
            # If debug, log event
            helper.log_debug('OMSInputName="' + str(input_name) + '" status="' + str(response.status_code) + '" step="Post Query" search_params="' + str(search_params) + "'")
            # Parse the response to get the ID and status
            data = response.json()

            '''
            search_id = data["id"].split("/")
            id = search_id[len(search_id)-1]
            status = data["__metadata"]["Status"]

            # If status is pending, then keep checking until complete
            while status == "Pending":

                # Build URL to get search from ID and send request
                uri_search = uri_search + '/' + id
                uri = uri_search + '?' + uri_api
                response = requests.get(uri,headers=headers)

                # Parse the response to get the status
                data = response.json()
                status = data["__metadata"]["Status"]
                '''

        else:
            # Request failed
            helper.log_error('OMSInputName="' + str(input_name) + '" status="' + str(response.status_code) + '" step="Post Query" response="' + str(response.text) + '"')

        # Print the results of the search to std_out
        for i in range(len(data["tables"][0]["rows"])):
             data1 = "{"
             #This nested loop goes through each field, in each event, and concatenates the field name to the field value
             for n in range(len(data["tables"][0]["rows"][i])):
                 field = str(data["tables"][0]["columns"][n]["name"])
                 value = str(data["tables"][0]["rows"][i][n]).replace('"',"'").replace("\\", "\\\\").replace("None", "")
                 if value == "":
                     continue
                 else:
                     data1 += '"%s":"%s",' % (field, value)
             data1 += "}"
             data1 = data1.replace(",}", "}")
             event = Event()
             event.stanza = input_name
             event.data = data1
             ew.write_event(event)

         #Delta
         state = now_dt.strftime("%d/%m/%Y %H:%M:%S")
         helper.save_check_point('last_date', state)




******

[dpanych]

I would recommend reinstalling the app from clean and do not use the old /local/*.conf - setup everything from new.

Re-downloaded app from Splunkbase

Inputs > Create New Inputs
Name: oms
Interval: 60
Index: main
Resource Group: xx
Workspace ID: Use Workspace ID, NOT Name - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Subscription ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Tenant ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Application ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Application Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
OMS Query: search DeviceName == "ext-device-01"
Max Count: 1000
Start Date: 14/08/2018 01:01:01

Changed source code

Worked! Data starts flowing in

[dpanych · Accepted solution]

See original post.

Michael @jkat54, again, thank you for creating this app!

[jkat54]

I’ll wrap this up into a new release and be sure to give credit to you for the update.

Many thanks again David!