Python script to be integrated at MS Azure in Splunk

[magic333]

Hello dear community,
could you please clarify the following:

We would like to deploy Splunk in Azure and we receive messages with telemetry in Azure IotHUB. Using a python code we prepare and analize this data, as a result we need to integrate this code in to Splunk cloud. Then we would like to visualize data in the Splunk resource. What are the steps to integrate this script in to Splunk in Azure.

https://docs.splunk.com/Documentation/Splunk/6.5.2/SearchReference/script - there is a link of course, but this is kind of a pre-check procedure and further installation. What are the installation steps? (a manual perhaps)

Can we make any changes in the code or this requires some re-deployment ? For us, it is important to be able to make changes in the python code time to time in Azure. (similar to Azure server less functions) What could be an easy solution for Splunk?

Thank you in advance
Best regards
Igor

[cduffey_splunk]

Does it have to be python script? There are ways to send the data from Azure IoT to Azure Event Hub, then Splunk can collect the data from event hub. There are existing Splunk IoT customers who do this currently.

[magic333]

We are using Pandas lib. and our data analysis is based on this code, thus Python script. We don't need to send it by this script from IoT to Splunk, we need to detect some issues in data (anomaly) The question is what is the best way to integrate this script with the Splunk App. It can be like a running backend in Splunk and triggered an every minute, for instance.

[cduffey_splunk]

If you python script needs to retrieve data from Splunk-
You might consider leveraging the Python SDK (https://dev.splunk.com/enterprise/docs/python/sdk-python/examplespython/) to retrieve the data periodically from Splunk (say every minute as you stated) perform the analysis.

If you just want Splunk Apps to periodically run the script or pick up the results of the script-
You could use the Splunk App to to schedule the script to run every minute, 5 minutes, etc. To make sure the app has proper permissions I would run the script using the Splunk python libraries to make sure if doesn't have any errors (SPLUNK_HOME/bin/splunk cmd python ). The other consideration doing it this way is python 3 is only supported on Splunk Enterprise 8.x. If the current library uses python 3, you might need to consider running it outside of Splunk (such as a cron job or windows task scheduler depending on your OS).

You can then pump the data results back into Splunk (if needed) in various ways (e.g. writes the output to a data file and Splunk monitors the data file and imports the data).

Hopefully this helps. Search for "splunk app run python script" should also yield examples and how to do this.

[magic333]

Thank you for the answer. I would like you to clarify your suggestion.
Are you talking about Splunk in Azure since it is delivered just as Linux VM. Is everything you told apllicable to MS Azure and its Splunk resource ?

Best regards
Igor

[cduffey_splunk]

Splunk in Azure would work but it would be managed by your own organization (Splunk Cloud is only currently available on AWS) as a Linux VM. Splunk can also be run in a hybrid model where you have on-premise instance of Splunk that queries Azure or another cloud hosted instance of Splunk. There really are multiple options; however, going with a cloud provider might be better if you are doing a lot of machine learning and AI, just so you have the necessary compute resources. I think there may also be special considerations for them to allow you run the python script from the app. Would probably be best to consult with your sales team to clarify everything that is needed for the approach you want. They might be able to give you more advice as well on the exact setup.