topic Re: Structuring nested data in Getting Data In

Structuring nested data

Tim — Fri, 16 Jul 2010 11:08:11 GMT

Hi all,

I have an application that needs to write some data that may be several levels deep and I'm struggling to understand how I can best represent it for Splunk.

Here's an example of a data set. It consists of:

DeviceID=nnnnn

Application=App1

    Permissions=X,Y,Z

Application=App2

    Permissions=Y,Z

In other words, a device may have multiple applications and each app may have multiple permissions. If I wanted to write a report that list all applications that contain Permission=X, how do you recommend we model the data?

Any guidance you can provide will be greatly appreciated!

Re: Structuring nested data

gkanapathy — Fri, 16 Jul 2010 12:51:11 GMT

Easiest by far is to denormalize each fact (i.e., permission or set of permissions) into a self-contained and fully-qualifed distinct record, e.g. (each line below is a separate event/record):

DeviceID=nnnn,Application=App1,Permission=X
DeviceID=nnnn,Application=App1,Permission=Y
DeviceID=nnnn,Application=App1,Permission=Z
DeviceID=nnnn,Application=App2,Permission=Y
DeviceID=nnnn,Application=App2,Permission=Z

The Splunk query for applications containing permission X would be Permission=X | stats count by Application (which would give you the count for each application as well, which you could discard.)

You could compact it up one level, though for some reporting purposes you may wind up expanding that one level out to the equivalent of the above at search time (in the specific case of this report, though Permission=X | stats count by Application will still work the same):

DeviceID=nnnn,Application=App1,Permission=X,Permission=Y,Permission=Z
DeviceID=nnnn,Application=App2,Permission=Y,Permission=Z

Note, btw, that while the raw forms suggested here look much larger than a tree, the indexed and compressed data set (which is how it would be stored on disk by Splunk) for large sets of data is not larger than if it were represented as a tree or a more normalized form.

Re: Structuring nested data

smolcj — Mon, 25 Feb 2013 05:52:45 GMT

There isn't any concept of nested table in splunk?