All Apps and Add-ons
Highlighted

Add-on or KV Stores approach

Explorer

Hi,

So I'm trying to build a dashboard that will provide statistics (& visualizations) based on 2 types of data. Here is some background information:
1. Data that is fed into Splunk (say, firewall logs)
2. Data fetched a third party API (note: data is fetched on a daily-basis and only the latest data pull should be considered for any analysis)

A restriction is I don't wish to run a script outside of the Splunk environment (no external script fetching data from API).

I have a few questions:

  1. Is there a way from within the add-on builder to request data from the third-party API and feed it to a KV store?
    I've taken a look at the Add-on builder documentation; modular input using REST API and the Python helper functions, but those force the fetched data to be written to an index using writeEvent method. Any other approach?

  2. I've chosen the index-based approach. I feed the data into the index. In order to correlate data, I built a data model, but is there a way to perform lookup operation between 2 indexes ? (So I have IP in the firewall log and ip being fetched from API as well — I want to perform A field based lookup, if that makes it more clear)

  3. Another big question: so firewall logs have {src_ip, dest_ip} and from the third party API, I get {ip, score}. Is there a way to perform mapping of IP to srcip and destip at the same and join the score automatically for both in one query?
    Currently I'm using 2 different queries (one for src and one for dest)

    | from datamodel:"Network.All_Traffic" | join type=left src_ip [ search source=third_party_api_data | rename ipAddress AS src_ip]
    | from datamodel:"Network.All_Traffic" | join type=left dest_ip [ search source=third_party_api_data | rename ipAddress AS dest_ip]
    then i am forced to perform an outer join combining the above two datasets then performing a DEDUP on the ip and score to get A UNIQUE SCORE for every ip. = Which is CRASHING my splunk instance? - it that because I'm having 2 streaming datasets (being updated every second for new logs) > perform mapping based on src and dest then join those and then perform dedup?

Do advise 🙂

0 Karma
Highlighted

Re: Add-on or KV Stores approach

Communicator

I am a little confused about #1. It's actually easier to create a lookup to store the fetched data, if you do not want to index it as you hint at.

#2. This can also be solved with a lookup. Just populate the lookup with the outputlookup command, referencing the first index. Then run the query on the second index, referencing your lookup with inputlookup command.

I leave #3 for someone with a little more experience, thus my reason for submitting this as a comment and not an Answer.

0 Karma
Highlighted

Re: Add-on or KV Stores approach

Explorer

1 So yeah I did create a python script fetch the data and store in KV . But using the add-on builder, it not possible or at least based on my reading of the documentation. Here link to the Addon Builder's Python Functions. They do have checkpoints but that just stores 1 key and 1 value; so thats pretty much useless.

Can you share some documentation / snippet that allow feeding data into KV from add-ons ?

2 Yeah if lookups are used, then I've tried that. But using the index based approach is what I'm doing as I cannot achieve (1) using lookups.

0 Karma
Highlighted

Re: Add-on or KV Stores approach

Splunk Employee
Splunk Employee

| from datamodel:"Network.AllTraffic" | join type=left srcip [ search source=thirdpartyapidata | rename ipAddress AS srcip | stats latest(score) AS src_score BY ipAddress ]
something like that should limit the amount of data returned by your subsearch (for the crashing bit, just guessing at the cause though).

If you make this thirdpartyapi_data a lookup, then you can actually just do automatic lookups in your datamodel, much easier, and no need for two searches

There is already apps on splunkbase that can pull in data from REST endpoint, just do a scheduled search with a | outputlookup at the end to put the output in a lookup:
https://splunkbase.splunk.com/app/4253/#/details
https://splunkbase.splunk.com/app/4172/#/details

Highlighted

Re: Add-on or KV Stores approach

Explorer

I'll definitely take a look at those add-ons, thanks for the share.

The reason I've not invested in third-party add-ons as this project uses confidential data so clubbing together third-party add-ons along with it is going to be a big NO from management. But I should be able to understand how they do the request using a Splunk search query and use it to my advantage..

If not, I'll try out out the query you shared at the start of the message and see if that helps with the crashing.

0 Karma
Highlighted

Re: Add-on or KV Stores approach

Explorer

New Error "Error in 'SearchProcessor': Found circular dependency when expanding from.NetworkTraffic.AllTraffic"

0 Karma