Archive
Highlighted

Data Pipeline commands

Builder

I am looking for a way to verify what commands are getting executed on my data. This link gives me a general indication of what will get processed, but I would like to know exactly which of my rules get executed where.

http://docs.splunk.com/Documentation/Splunk/4.2.3/Admin/Configurationparametersandthedatapipeline

is there a data trace command or somthing similiar?

Tags (2)
Highlighted

Re: Data Pipeline commands

Ultra Champion

The Splunk on Splunk(SoS) app has some useful views on the metrics of the search commands being executed in your Splunk deployment.

Highlighted

Re: Data Pipeline commands

Splunk Employee
Splunk Employee

Indeed! Check out the "Indexing Performance" view which shows the size of the queues feeding the event-processing pipelines, as well as a breakdown of CPU time consumed per Splunk processor. You can also find a reference diagram for pipelines and processors in the "Learn more" in-view documentation of that view.

Highlighted

Re: Data Pipeline commands

Splunk Employee
Splunk Employee

By the way, the correct terminology is that data moves through data pipelines where it is transformed by processors that perform operations such as defining event boundaries, time stamp extraction or field transformations.

0 Karma
Highlighted

Re: Data Pipeline commands

Builder

I do not see right away what I was asking. I do see some interesting things, so I will spend some time with it after I get mu data burst today.

0 Karma
Highlighted

Re: Data Pipeline commands

Splunk Employee
Splunk Employee

Could you be more explicit about you what you mean by "what commands are getting executed on my data"?

Note that splunkd does not execute commands (in the context of say, a shell command) on your data. Instead, your data is sliced up in chunks that are referred to as "pipeline data", which is then processed pipelines where processors perform specific operations on it such as translating binary to text, cutting the text stream into lines, extracting the time stamp, etc...

0 Karma
Highlighted

Re: Data Pipeline commands

Builder

the specific operations that you refer to, I would like to know what they are for my configuration. The link that you sent me was good as it told me about all the stanzas, but I would like to know which ones are executed.

0 Karma
Highlighted

Re: Data Pipeline commands

Splunk Employee
Splunk Employee

Considering the information just added in your comments, I understand that you want to see which index-time configurations parameters are applied to your data. Your best tool to do this is the btool command scoped to the props.conf file, which contains almost all index- and search-time configuration parameters.

If you're not fond of using the command line, you can use the "Configuration Viewer" and "Configuration Comparator" views of the Splunk on Splunk app to achieve similar results. Just make sure to scope the view to the props.conf file and search for the configuration stanza that affects your data in the search bar. For example, if you want to see all index-time parameters that are applied to the syslog sourcetype, go to the "Configuration Viewer" view, select props.conf as the configuration file to view and search for :

stanza=syslog

Do note that although it is quite easy to search in this way to see how index-time parameters are applied to sourcetypes, it's a little bit more difficult for source- and host-based stanzas in props.conf as those are typically defined with wildcards. That being said, you can always search for :

stanza="source::*" OR stanza="host::*"

...and see if any of the returned stanzas would match your data based on the source and host fields.

View solution in original post