Getting Data In

Should I index data from my external REST API in Splunk before making dashboards out of it?

umairahmad3985
Path Finder

Hi Awesome People,

We are making a Splunk App for one of our products and the goal is to display the stats collected from that product's usage to the customer using that in the form of pretty dashboards. We have exposed all of those stats as REST APIs which can be used from anywhere with an API key authentication. So far so good.

Now here's a decision I cannot make and need your help in deciding. Which is the preferred method of achieving the above?

1- Use a modular input to poll our APIs and index the results in Splunk and then simply make use of Splunk's query language to get the stats from the indexed data.

2- Create custom search commands that communicate to our REST APIs and then use these custom commands in dashboards to render the data.

I don't have much experience with using Splunk so I don't know which one of the above options is less complex in terms of time, memory, storage. So, please guide me on which method should I better use?

Thanking you all for reading my query and helping me out in any way.

Regards,
Umair

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

I recommend indexing the data. Once the data is indexed it can be used for dashboards, but it can also be used for other purposes. It also gives you a historical record of the data.
Custom commands put additional load on your API servers each time the dashboard is opened or refreshed. By indexing the data you avoid this extra server load.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I recommend indexing the data. Once the data is indexed it can be used for dashboards, but it can also be used for other purposes. It also gives you a historical record of the data.
Custom commands put additional load on your API servers each time the dashboard is opened or refreshed. By indexing the data you avoid this extra server load.

---
If this reply helps you, Karma would be appreciated.
0 Karma

anthonymelita
Contributor

Another way to avoid ad-hoc load would be to run the custom search commands as a scheduled report and access the report on the dashboard instead of the search itself.
That works if all you care about is the latest result. If you want historical data, or would have a need to track the API reliability, then ingesting is the way to go.

0 Karma

umairahmad3985
Path Finder

Hi @richgalloway and @anthonymelita,

Thanks for your responses. I do see the value in indexing data as well as the scheduled reports method but here is my concern: The stats from our APIs are given based on a few parameters provided by the user (e.g. timerange, usertype etc). Now, since we don't have any knowledge of what the user might input, we cannot make the REST API call without knowing his/her input first. Wouldn't the whole idea of indexing or scheduled reports, fail here? Let me know your thoughts on this.

Thanks again!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Create a modular input that indexes API data continuously for all user types. Then any user query can be satisfied from the index.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...