Http Event Collector vs Add-on

[noamm91]

Some of my customers are using Splunk as their SIEM solution.

I have a security platform that needs to integrate into their Splunk to send security events (probably syslog) into a certain index (might be an existing or brand new one).

I already made a PoC using HEC and successfully managed to deliver my syslog events into an index in my test Splunk account (using Splunk Cloud Platform).

The setup process that my customers will have to do for the integration using HEC is to create a new data input, create a token, and eventually deliver it to me (alongside their Splunk hostname).

Now I'm wondering if this process can somehow be simplified using an app/add-on.

Not sure exactly what is functionality using an add-on gives and if I can somehow leverage it in order to simplify the integration onboarding process between my security product and my customers.

Is there anything else I should consider? Would love to know, I'm completely new to Splunk.

Also, case it matters, most of my customers, are using Splunk Cloud Platform but in the future there might be customers that will have Splunk Enterprise, case it matters.

Thanks

[tscroggins]

As a Splunk practitioner, I prefer third-party products that allow me to manage the flow of data, typically through a published API. Allowing a third-party to connect to a Splunk Cloud HEC endpoint requires modifying a network ACL, which in turn exposes the Splunk Cloud stack to additional risk. Connections to a self-hosted Splunk Enterprise HEC endpoint require implementing and maintaining edge infrastructure--firewalls, proxies, etc.--which expose not only Splunk Enterprise but typically, an entire enterprise network to additional risk.

The best ISVs write and support Splunk add-ons to integrate with their product's API. Whether that makes sense for you depends entirely on your business. Allowing external integration exposes you to risk, but that may be justified by the reward.

The process you followed to enabled HEC and create a token in your test environment is the same process other Splunk Cloud customers would follow, although some may choose to manage HEC tokens via configuration files uploaded as custom apps. Splunk Enterprise customers would do the latter. Either is a relatively low effort task for experienced Splunk administrators, but I wouldn't depend on every customer having experienced staff.

I do not recommend creating an add-on for your product that simply enables HEC. That's a misuse of the feature, and you'd want your customers to control authentication and authorization through their own HEC token definitions.

[noamm91]

Thank you for replying.

OK, so sounds like asking customers to enable HEC, create a token, and give it to me so that my product can send events to their Splunk isn't a good practice.

Writing an add-on does make sense to me, considering what you wrote, I believe that this is the way to go.

Followup questions:

1. Can you describe the process of creating an add-on and publish to the Splunkbase (?) so that my customers can install(?) it?

2. What's the actual logic that this add-on should have? i.e. how would the add-on installed at the customers' Splunk (either cloud or enterprise deployment) be used to deliver events?

[tscroggins]

You can ask customers to use HEC. Many big players do. I just find it a bit cringe to ask customers to poke holes in their infrastructure. Opinions vary.

[tscroggins]

The first question is answered directly by the developer guide:

https://dev.splunk.com/enterprise/docs/developapps

https://dev.splunk.com/enterprise/docs/releaseapps

The second question depends on your product, but modular (or otherwise custom) inputs typically do one of three things:

1. Write events to stdout, which is automatically indexed by Splunk.
2. Write events to rotating logs files, which are separately indexed by Splunk using monitor stanzas.
3. Write events through a Splunk-provided API. See https://dev.splunk.com/enterprise/docs/devtools.

If you have more specific questions about a particular topic, post a new question, and the community will gladly assist. Welcome to Splunk!

[noamm91]

Thanks.

Regarding the 1st answer - understood.

Regarding the 2nd answer - I'm not sure understand (because I didn't explain well) the setup.

My system is a SaaS application that is running somewhere in a public cloud, it doesn't know Splunk, so I don't see how writing to stdout or rotating log files is relevant here. The link you sent for the developer tools API doesn't seem relevant either. 

The add-on, if I understand correctly, is a piece of code that runs in the customer's Splunk deployment (either enterprise or in their Splunk cloud) and should somehow interact with my SaaS security product in order to get events from it, right? My question is - how does the add-on interact with my SaaS system that isn't where the Splunk /add-on is running? 

I can open an additional question it just seems related to my first question which is about the difference between add-on and HEC and I still haven't figured out how the add-on actually works (getting events from an external system that runs elsewhere).

[tscroggins]

If your application already supports a standard API--TAXII for threat intelligence, for example--there may be an existing app or add-on you can leverage.

[tscroggins · Accepted solution]

It depends on your SaaS application. The add-on is the integration layer or "glue" between your application and Splunk. The add-on is a combination of the code you write, the configuration needed to parse events before they're indexed, and the configuration needed to search events after they're indexed. There's a good overview of the Splunk pipeline at https://docs.splunk.com/Documentation/Splunk/9.1.2/Deploy/Datapipeline; however, you don't necessarily need to know everything about Splunk to write an add-on.

The code you write depends on your application. Do you have an API? If yes, your code would read or receive data from your API, break it into logical events, and write the events to Splunk using Splunk's API. When I write "read or receive," I'm implying that the add-on establishes a connection your application and polls an API for new data, waits for new data through a publisher/subscriber interface, or otherwise gets data from your application. It's up to you to determine how data is retrieved, how checkpoints are tracked, etc.

[noamm91]

Got it. Thanks.

My application has an API. If the add-on can send HTTP requests to my API in order to fetch events and then index them in Splunk then it sounds like exactly what I need. I'll start writing the add-on then. Thank you for being so responsive and informative.