Hi,
So I'm trying to build a dashboard that will provide statistics (& visualizations) based on 2 types of data. Here is some background information:
1. Data that is fed into Splunk (say, firewall logs)
2. Data fetched a third party API (note: data is fetched on a daily-basis and only the latest data pull should be considered for any analysis)
A restriction is I don't wish to run a script outside of the Splunk environment (no external script fetching data from API).
I have a few questions:
Is there a way from within the add-on builder to request data from the third-party API and feed it to a KV store?
I've taken a look at the Add-on builder documentation; modular input using REST API and the Python helper functions, but those force the fetched data to be written to an index using writeEvent method. Any other approach?
I've chosen the index-based approach. I feed the data into the index. In order to correlate data, I built a data model, but is there a way to perform lookup operation between 2 indexes ? (So I have IP in the firewall log and ip being fetched from API as well — I want to perform A field based lookup, if that makes it more clear)
Another big question: so firewall logs have {src_ip, dest_ip} and from the third party API, I get {ip, score} . Is there a way to perform mapping of IP to src_ip and dest_ip at the same and join the score automatically for both in one query?
Currently I'm using 2 different queries (one for src and one for dest)
| from datamodel:"Network.All_Traffic" | join type=left src_ip [ search source=third_party_api_data | rename ipAddress AS src_ip]
| from datamodel:"Network.All_Traffic" | join type=left dest_ip [ search source=third_party_api_data | rename ipAddress AS dest_ip]
then i am forced to perform an outer join combining the above two datasets then performing a DEDUP on the ip and score to get A UNIQUE SCORE for every ip. = Which is CRASHING my splunk instance? - it that because I'm having 2 streaming datasets (being updated every second for new logs) > perform mapping based on src and dest then join those and then perform dedup?
Do advise 🙂
... View more