All Apps and Add-ons

Splunk Machine Learning Toolkit: Can I apply a machine learning model to external data via REST?

walterd
Engager

Hi all,

I work in a Security Operations Center for an organization, which is targeted by lots of Spam and Phishing mails.

Our users have the opportunity to report those Phishing mails to us, where we continue analysing them.
The issue is, that we get plenty of Spam mails reported too.

Now we would like to filter out only the Phishing ones.
Therefore, I have obtained a dataset and trained a model in Splunk.

Regarding the worklow, I am not sure on how to integrate Splunk in the chain.
In our current situation, the raw metadata of the extracted features cannot be streamed in realtime to Splunk but has to be uploaded manually one a week.

I am looking for a solution something like this:
1. Extract the features from newly reported mails
2. Call a ML algorithm with the previously obtained features as paramaters in Splunk via REST
3. Get the result of the classification in an instant

Can this be done within Splunk without having the data to be first forwarded and indexed?

Cheers,
Dan

1 Solution

astein_splunk
Splunk Employee
Splunk Employee

Hi

"In our current situation, the raw metadata of the extracted features cannot be streamed in realtime to Splunk but has to be uploaded manually one a week."

Why is this? Most of the customers I work with can extract their features from the streaming data in Splunk, which are used in the training of the model (|fit) . Can you elaborate?

If you go through the SDK/API documentation for Splunk, you can submit a search to splunk and get the results back from an external system. An example search can be making an empty result set (makeresults) , filling the correct fields with features (|eval) and then applying the ML model to get the result back (|apply).

View solution in original post

astein_splunk
Splunk Employee
Splunk Employee

Hi

"In our current situation, the raw metadata of the extracted features cannot be streamed in realtime to Splunk but has to be uploaded manually one a week."

Why is this? Most of the customers I work with can extract their features from the streaming data in Splunk, which are used in the training of the model (|fit) . Can you elaborate?

If you go through the SDK/API documentation for Splunk, you can submit a search to splunk and get the results back from an external system. An example search can be making an empty result set (makeresults) , filling the correct fields with features (|eval) and then applying the ML model to get the result back (|apply).

View solution in original post

walterd
Engager

I only saw now your reply.

Thank you very much! This was exactly the solution I was looking for.

Regarding your question, we have some sort of a process bottleneck in place, which will need time to get removed.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!