All Apps and Add-ons

What is the best practice for distributing sample data for app?

Graham_Hanningt
Builder

I'm developing a Splunk app that I plan to make available on Splunkbase.

The app contains dashboards that visualize data from various proprietary source types.

In a test or production environment, users will forward data to the app from the system that generates those source types. (More correctly: uses will forward data to a TCP data input defined by the app.)

For demo purposes, for users who want to see examples of the visualizations without forwarding their own data, I want to provide sample data.

I have 50 MB of sample data in a JSON Lines file. This compresses to a 1.5 MB .zip file.

I'm debating how and where to make this sample data available to app users.

I'm anticipating that users will download the .zip file from somewhere, uncompress it, and then upload it using Splunk Web, following some simple instructions (to select the appropriate custom source type for upload) that I'll probably provide in the detailed description of the app in Splunkbase.

I've previously built Docker images for such demo purposes, and that's worked fine: a single docker run command creates a working Splunk installation with the app and sample data. However, this time, I want to make the sample data available separately, outside of a Docker image, to give users the choice of which Splunk installation to use to host the app.

I don't like the idea of bloating the Splunk app, which is currently only a few dozen kilobytes, with a 1.5 MB .zip file that is only useful for demos.

How do other Splunk app developers support this use case? What's the best practice for providing sample data for an app on Splunkbase, without "bloating" the app itself? (Note that my source types are proprietary, non-trivial to programmatically synthesize from scratch, and not generally available from public sources.)

0 Karma
1 Solution

sduchene_splunk
Splunk Employee
Splunk Employee

Hello, usually samples are put in sample folder within the app, and eventgen.conf in local or default.
in your case why not providing the app with eventgen.conf but without the sample, and simply put your sample on public github, then in doc, ask to manually copy the sample in the app' sample folder?

View solution in original post

0 Karma

sduchene_splunk
Splunk Employee
Splunk Employee

Hello, usually samples are put in sample folder within the app, and eventgen.conf in local or default.
in your case why not providing the app with eventgen.conf but without the sample, and simply put your sample on public github, then in doc, ask to manually copy the sample in the app' sample folder?

0 Karma

Graham_Hanningt
Builder

Thanks for your answer, much appreciated.

I've not used an eventgen.conf file before. That requires the Eventgen app to be installed, right? That is, this app:

http://splunk.github.io/eventgen

(also available on Splunkbase), correct?

With apologies—I realize I'm probably missing something—I don't see the benefits of using Eventgen in this context. It seems like an unnecessary prerequisite. I don't need to alter timestamps in the sample data; I don't need to simulate real-time event arrival. I just want to upload the sample data, as is, as quickly as possible.

In my app description in Splunkbase, I'd planned to point users to wherever on the web I'd put the zipped JSON Lines sample data file, and then provide instructions to unzip the file, and then use Splunk Web to upload the data in the file. A few mouse clicks.

What am I missing? How does using Eventgen improve on this?

0 Karma

Graham_Hanningt
Builder

Another reason why, for my use case, I prefer uploading via Splunk Web over the method that you describe: uploading via Splunk Web does not require the user to have direct access to the Splunk app folder.

It's conceivable that the users who want to try out my app with sample data might not be that Splunk-savvy. Installing Splunk, installing the app from Splunkbase via Splunk Web, and then uploading sample data (after downloading the data from somewhere else) via Splunk Web: these are—thanks to you, the Splunk developers—easy GUI procedures.

Furthermore, if the user decides to run Splunk in a Docker container, rather than installing it on their native OS, then that adds another level of complexity to the method that you propose: having to copy the sample file into the Docker container.

I'm sincerely not being deliberately adversarial here: I really do want to understand whether, for my use case, the Eventgen-based method that you suggest is better than uploading via Splunk Web. Currently, however, I see more cons than pros.

0 Karma

sduchene_splunk
Splunk Employee
Splunk Employee

hello, i see 3 ways to achieve it :
a/The first method i suggested indeed needs the eventgen app you mentionned.
if you want your user to use the GUI to upload the sample, i'd simply create a 2nd (standalone) app : that will use eventgen :
mysampleapp-local-eventgen.conf
mysampleapp-samples-mysample.txt
this way your users upload 3 apps : originalapp, eventgen, mysampleapp

b/simply put the file on github, and ask people to use the gui : settings->add data, and select your sourcetype. (this requires a small knowledge on Splunk)

c/without eventgen :
create a simple app containing inputs.conf, your file, indexes.conf, so that it indexes the file when app is installed. the app would contain an indexes.conf to put the file in a new index, and to not mess with their existing indexes. if you do this i recommend you warn the user that this will create a new index, and index the file.:
your app contains : mysimpleapp-local-inputs.conf
[monitor:///$SPLUNK_HOME/etc/apps/mysimpleapp/samples/mysample.txt]
sourcetype=xy
index=toto
disabled=0
(note, if you are willing to do a good job : add two stanza, one for unix, 1 for windows)

for the indexes.conf stanza : simply declare a new index

0 Karma

Graham_Hanningt
Builder

Thanks very much for the follow-up.

I really appreciate your descriptions in options (a) and (c) of "data-only" apps, with and without Eventgen. While option (c) had previously occurred to me, I'd never gotten around to thinking in detail about the specific configuration required (the monitor input).

I'm going with option (b), which is the option I was originally considering. You've helped convince me that this is the best fit for my use case. I'm okay with providing instructions to step through the upload procedure, such as selecting the appropriate custom source type.

Thanks again for your help and advice, much appreciated.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...