Architecting for Splunk Total Beginner

[MatthewMcD]

I am completely new to Splunk. I understand the basics but am lost on where to start with the designing for and supporting the following scenario for Splunk (or any SEIM). I didn't see a Community Location for this type of question so feel free to direct me to the "Total Nube" section.  

We run a multi-tenant cloud application and our customers who use Splunk want us to "Log to Splunk". Looking through the "Getting Data In" sections it is unclear to me how we would support Splunk. In our software we allow our tenant admins to preform configurations themselves. So my basic question is:

As the developer of a cloud based app, how do we provide support for Splunk? 

1. Do we "push" event info to a Splunk server that we store the endpoint information for each tenant separately?
2. Do we create a REST endpoint that Splunk can pole on a specific frequency from each Splunk instance? 

Bear in mind that we will have tens of customers configuring their tenants to work with their own servers. All the info I have found is geared toward configuring Splunk for my use for my team and not this multi-tenant scenario.

Thanks in advance.

[gcusello]

Hi @MatthewMcD ,

Splunk is really different than other platforms, so I suggest to follow the basic training (Fundamentals I, Splunk Architecture Free course and Search Tutorial).

Then find a Certification Path and follow training!

In the meantime, if you have time and access to the partner Portal, there are many free courses to follow.

Ciao.

Giuseppe

[esix_splunk · Accepted solution]

Regarding serverless "push" to Splunk, as opposed to Splunk polling. This is the future of real time analytics, and polling will eventually die. 

With that, Splunk has support for HTTP Event Collection (HEC), which is a standard logging library in Java and most major languages now. This allows logging to send, via a key, to a remote https location its logs in a structured format. 

There's quite a bit of information on this here : https://dev.splunk.com/enterprise/docs/dataapps/httpeventcollector/.

Splunk Architecture courses will touch on this a bit, HEC is typically enabled on the indexer(s), and in larger environments a load balancer is put in front of this for clients to send to and distribute events evenly. See more here : https://docs.splunk.com/Documentation/Splunk/8.0.4/Data/UsetheHTTPEventCollector

As for adding REST endpoints, this is always a good idea to enable other apps access to your systems, as RESTful connectivity is pretty much standard for webapps now. And with this, as long as you have a REST logging endpoint, Splunk can poll this. This circles back around to having a requirement for a Agent that does polling, and the lack of data during the polling intervals...

[MatthewMcD]

Thanks for this reply. The links helped me get up and running!