Splunk Search

Data Model and Macro Search: Code, Fields and Data Structure Exploration

Motivator

I am looking at how to see the details of the events which drive dashboard panels when the results are brought in through pre-made means and not through traditional searches. For example, we have a dashboard called "State of the Network" which has a panel called "Bytes Transferred and Duration by User - Last Hour" with the following in the Search portion of the panel:

| tstats count(User) from pan_threatDetail groupby User Bytes Duration Threat External_IP Location | eval KB=Bytes/1024 | stats sum(KB) as "Transfer (KB)" sum(Duration) as "Total Duration" by User Threat External_IP Location  | sort -"Total Duration" limit=8

While I am familiar with the sourcetype=panthreat, I am not familiar with panthreatDetail. I also cannot find "User", "Bytes", "Duration, or "Threats" in panthreat sourcetype records. Also, the stats functions make it difficult to see the data being combined to create the stats. Oftentimes, I want to take the sum(Bytes) and sort the individual events by Bytes, or group them by source ip address and then sort by Bytes. Without seeing what is on the left of that bar or behind the panthreatDetail, it is impossible to know the source of the tstats data for sure; for instance, the formula used to calculate "Duration". It seems like searches can be hidden in reports and perhaps even in indexes. Where can searches be hidden and what is the best way to found where they are hidden to see the exact search?

My question is, with Dashboard Panels like these, where the search is not a traditional search, and many of the terms are hidden, what is the best way to understand the search and drill down into various aspects of the search for more details of where the data is coming from?

0 Karma

Path Finder

The tstats command is used by the Pivot functionality of Splunk 6. That being the case, you probably can't find "User", "Bytes", Duration", or "Threats" in your sourcetype because they are created by the Data Model that was used by Pivot during the creation of the dashboard you are trying to "drill" into.

When building a Data Model in Slunk 6, you can create new fields using regular expressions, eval, lookups, etc. Also, during the configuration of the Data Model, you can choose display different field names than the ones in the data (i.e. the username field can be renamed to User).

Regarding your statement about finding the exact search, in this case, the search you posted is the exact search. Since it's using the Pivot and Data Model, you won't see the "behind-the-scenes" regexes, lookups, and/or evals used to create the fields you can't find in the data. Your ability to see the logic behind each field would depend on your user role and the Data Model permissions.

0 Karma

Motivator

Also, when do macros excel over saved searches/reports?

0 Karma

Contributor

you would use a macro at times when you are writing the same bit of SPL code repeatedly in multiple situations. Saved searches are used for populating summery indexes, creating correlated searches in the Enterprise Security App., populating dashboards, setting up alerts, scheduling reports etc. Macros and saved searches are very different as well.

0 Karma

Motivator

Is a data model the same as a macro? I don't see any ticks in the search that I originally posted. How do you know that the search posted in the original post of this thread is a data model? Is there any way to see the search code of the data model? Where would the data model search code be found? Likewise, where would macro search code be found?

0 Karma

Contributor

A data model is definitely not a macro. A macro operates like macros or functions do in other programs. A datamodel is a knowledge object based on a base search that produces a set of search results (such as tag = network tag = communicate) The datamodel provides a framework for working with the dataset that the base search creates. A data model is usually designed to reference an aggregate of similar sourcetypes such as firewall data and assigns the same field extractions etc to all of the contained sourcetypes, no matter what type of device it comes from (cisco, juniper etc.) These datamodels ride on top of various Technology AddOns which format similar event data to be CIM compliant, so that the event data will populate the relevant datamodel. Newer Splunk Apps, Such as Splunk for Enterprise Security depend on datamodels for their operations.

0 Karma

Path Finder

First off, I'll use to indicate that character. If you see something like convertToMB(someField), then someone has set up a search macro that converts the value of someField into megabytes.

I don't believe there is any other use, in Splunk, for the character.

0 Karma

Motivator

I read recently that the left single quote surrounding a term indicates that it is a macro. I would give an example, but this form seems to interpret the left single quote as an italics code of some kind and will not display it except by itself `
If I discover a search in the dashboard with those left single quotes, is there a good chance that they are macros? Or are these left single quotes used for other object types as well?

0 Karma

Path Finder

Go to Pivot>Manage Data Models>name of Data Model ( or Pivot>name of Data Model>Edit Objects) then click the Edit link next to the field you want. The Attributes column will tell you generally how the field was created. Clicking Edit will give you the details of the eval, regex, rename/transform, lookup, etc that is being used to create that field.

I believe your role would have to be explicitly given the ability to edit Data Models. If you don't see the buttons to manage/edit Data Models, you won't be able to see what I'm talking about with contacting an admin.

0 Karma

Motivator

So you are saying that this is a Data Model and that I might be able to see it if I have the right role? If I have the right roles, and I probably do, then where do I go to see the "behind the scenes" of a data model? When I go to Settings-Data models, I do not see any entries resembling "pan_threatDetail"

0 Karma

Motivator

Hidden searches (correct me if I am wrong):
eventtypes (settings menu) hide search terms
searches can be converted to reports, but not back to a search
indexes (settings menu) store search data and cycle old data when the data size limit is reached

I know how to view and manage indexes (settings menu), but I do not know how to view or create the search creating the index, yet.

0 Karma