Dashboards & Visualizations
Highlighted

Splunk for a boat race?

Explorer

Hello!

I am a bit of a lurker here. I am considering using Splunk as both the frontend and backend for a support crew telemetry website for a yacht race.

The boats transmit frequent but irregular telemetry data.

Given the competitive nature of the endeavour security is a paramount concern 😉

As the support crews are not necessarily IT literate, I propose that the site is basically one big dashboard, with very limited access to the underlying Splunk functionality.

So, here's the question(s):

  1. For security reasons, I propose to allocate a separate index to each boat. Is this the best approach?
  2. For simplicity, let's say there are two types of telemetry data, depending on the boats' equipment. I'm going to develop two dashboards, each displaying optimum information based on the raw data.

Do I need to replicate the dashboard/ app across each of the relevant indices? Or can I use the same app across each of the relevant indices?

  1. If I have to replicate the dashboard across each, is there some easy way to abstract the source data from the dashboard? What I mean, is that if I have index="boat1", index="boat2", etc, do I have to change all my queries, or can I have app1 written to expect a source of "boat_telemetry1 and abstract/ or 'rename' in one place boat1= boat_telemetry1, boat10=boat_telemetry2, etc? Does that make sense?

  2. Given the irregular nature of the comms, is there anything I should bear in mind?

Any advice appreciated.

OB.

PS. I'm considering ammaps. Has anyone used it for non-IP location data?

OB.

Tags (2)
Highlighted

Re: Splunk for a boat race?

Builder

So, here's the question(s):

For security reasons, I propose to allocate a separate index to each boat. Is this the best approach?

Not needed, you can have each boat in the same index, use a sourcetype flag and then limit each users search to their respected boats.

Do I need to replicate the dashboard/ app across each of the relevant indices? Or can I use the same app across each of the relevant indices?

Nope, you can use the same app and once again user their username to drive the saved search for the dashboard. If you store everything in a different index, you can also set a default index for each user, and then their search will be constrained to that index.

If I have to replicate the dashboard across each, is there some easy way to abstract the source data from the dashboard? What I mean, is that if I have index="boat1", index="boat2", etc, do I have to change all my queries, or can I have app1 written to expect a source of "boat_telemetry1 and abstract/ or 'rename' in one place boat1= boat_telemetry1, boat10=boat_telemetry2, etc? Does that make sense?

This is the main reason I feel you should do this all under 1 index. You'll be able to pull a correlation accross all boat times etc easily if they are contained in one index, and then set host="boata" host="boatb" etc. This will allow you to have very fine control over boats who's comms are similar, and those who are very unique. You'll be able to set each sourcetype with the field extraction for the unique data type, and the host tag can filter out down to a unique boat.

Given the irregular nature of the comms, is there anything I should bear in mind?

Since you don't fully know what data is coming it, it's hard to build any extraction type. By setting up a unique file for each boat and then controlling the inputs.conf for each file, you'll be able to account for any unique comms as well as set a global override. Given with what you're trying to do, I'd use the "host" flag for each boat, and then try to standardize the sourcetype based on each major comm style you receive. You'll be able to lock down each user to a specific host under the user permissions and should account for any major concerns you've mentioned. The only other major thing to make sure of, each "event" will need a date/time stamp. After that, splunk can manipulate the data any way you'd like!

Hope I've been of some help!

Highlighted

Re: Splunk for a boat race?

Splunk Employee
Splunk Employee

I think that you're mistaken about multiple indexes. You can search (and correlate) across many indexes if you have access to several indexes using either index=boat* or index=boat1 OR index=boat2 OR ....

Highlighted

Re: Splunk for a boat race?

Builder

Isn't searching across multiple indexes slower? This is for my own information, but I thought once a search crosses over to a new index it has to search the entire specified timespan for each index vs. if all the information was in 1 index, the main search only runs one time, am I mistaken?

0 Karma
Highlighted

Re: Splunk for a boat race?

Path Finder

One critical note about security: be sure to disable typeahead for your users. The suggestions won't take in account the search filters that are in place and depending on the data layout and the EQ (evil quotient) of your users this can cause problems as they can extract info from this source.

Highlighted

Re: Splunk for a boat race?

Splunk Employee
Splunk Employee

This isn't really accurate if you make a separate index per boat. Typeahead only will access indexes that the user has access to. This is one of the key benefits of using indexes to segregate data.

Highlighted

Re: Splunk for a boat race?

Splunk Employee
Splunk Employee

If security is your most important concern, you should ABSOLUTELY use a separate index per boat and have individual users only have access to their boat. The administrative user can have access to all indexes and hence can report across all boats at the same time, for example:

index=boat* | stats ... by index

You do not need a separate dashboard per boat. Each user's dashboard will only reference the data in the index they have access to. It is very, very important here to NOT schedule the search for the dashboard. Shared scheduled searches (for dashboards) will run as the owner, but the results will be available to all users with access. In this case, if the administrative user shares the scheduled search, all users will see data about all boats.

For bursty and sparse data, the biggest challenge is that a simple timechart. If you know when the bursts of data start, you can use a time-based lookup table to assign a run id (and perhaps start time to data in each burst. This can be done after the fact. Then you can make it easier to retrieve and summarize each batch of data.

Highlighted

Re: Splunk for a boat race?

Explorer

Thanks for that Stephen (et al). So let's say I have 10 indices, index="boat1", index="boat2", etc and I have two dashboards, one expecting source type "boattelemetryA", the other "boattelemetryB". How do I link the boat's raw telemetry data to the appropriate dashboard? I hope I am explaining that clearly. I don't expect Splunk to do it automatically, but for each boat, I would hope to edit the minimum number of variables to get the dashboards to work. Otherwise it won't scale, (I hope to have more than 10 boats).

0 Karma
Highlighted

Re: Splunk for a boat race?

Explorer

With regard to 'irregular', I was referring to the timing of received data. I'll always get it in the correct format, but some days I'll get multiple updates, some days none at all. That's what I meant by frequent but irregular. Sorry for any confusion.
On a separate but related subject, what would the search look like to see what teams have logged onto the dashboards, when, etc?
Thanks,
OB.

0 Karma
Highlighted

Re: Splunk for a boat race?

Splunk Employee
Splunk Employee

You can search index=_audit to see which users ran searches. The user activity dashboard in the Search app provides an example of this.

0 Karma