Archive

Can Splunk be adapted for use with human-keyed data rather than machine data?

New Member

For example, I want to key in some random personal observations, to-do lists, saved links, quotes from publications, etc in no particular order but with the inclusion, perhaps, of some topic-identifying tags...then I would like to be able to query and organize this data in various ways.

Tags (1)
0 Karma

Influencer

My short answer is: I would believe that it would be possible to use Splunk in such a manner, but it also feels like not the best solution.

I've seen people use Splunk for indexing IRC chat logs and emails for example. But these are both time series data that doesn't change after creation.

The reason this is important is Data is written to a Splunk index once only. No editing of the stored event is available after index time. if you update something, you'll have to re-index it in its entirety. You can go back and mark indexed events as deleted or the buckets of data eventually expire out (per retention policies, and whatnot). So, whatever scheme you develop for your data you'd likely want to keep this in mind.

Now pulling data into a Splunk search from external sources outside of Splunk indexing is of course also an option that might be helpful to leverage as well. DBConnect for example in addition to letting you pull data into a Splunk index, defines a command that lets you run a query against a relational database at search time to retrieve results. SA-ldapsearch has a similar command for querying LDAP. The Hunk product lets you pull data with a Splunk search from other no-SQL data stores (primarily Hadoop, but APIs to build other connectors).

But also it seems like there may be other products out there that may be better suited to this space. Things like Evernote and Google Drive let you store arbitrary notes and pictures and tags, edit/update them and search across things as well. Out of curiosity then, why try to fit Splunk to such a problem?

New Member

Thx. What you say makes sense.

Why Splunk? Because a like it and thought it had potential to be adapted for wider/alternative usesr (perhaps with the use of add-ons).

0 Karma

Influencer

Makes sense and as I mentioned it's a possibility. But you'd need a scheme to make sure you're not fighting Splunk's indexing or an addon to pull the data from somewhere else. If you're a developing type you can roll your own Splunk app / addon. It might make for an interesting app. If you're ambitious you could build and submit to their recently begun contest

0 Karma

Builder

Can you start by monitoring a Text file and keep writing you notes there?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!