topic Re: Is dynamic query construction possble in Splunk without using any SDKs ? in Splunk Search

Is dynamic query construction possble in Splunk without using any SDKs ?

Arun_N_007 — Fri, 22 May 2015 04:09:37 GMT

I need to modify the query of saved search based on some conditions. Is it possible using only Splunk query language?

Re: Is dynamic query construction possble in Splunk without using any SDKs ?

acharlieh — Fri, 22 May 2015 04:30:52 GMT

Based on what sort of conditions and what sort of modifications do you need to make? Query replacement tokens and subsearches might be helpful here but a complete answer requires more detail as to what you have and are trying to accomplish. The docs on savedsearch gives a hint at using string substitution for replacement tokens

Re: Is dynamic query construction possble in Splunk without using any SDKs ?

Arun_N_007 — Fri, 22 May 2015 04:39:07 GMT

For Example,

I want to have a single saved search query which will get executed for every 1hr and it will be processing the particular batch of records.
And for each batch query logic should be modifed dynamically.

Can we accomplish this using custom command? Can we invoke another query inside custom command without using Splunk Python SDK?

Re: Is dynamic query construction possble in Splunk without using any SDKs ?

woodcock — Sat, 23 May 2015 02:55:05 GMT

I think what you may need is a macro:
http://docs.splunk.com/Documentation/Splunk/latest/admin/macrosconf

Re: Is dynamic query construction possble in Splunk without using any SDKs ?

Arun_N_007 — Sat, 23 May 2015 19:22:39 GMT

Hmm I tried that you cant return a macro name with parameters using return statement.

Re: Is dynamic query construction possble in Splunk without using any SDKs ?

acharlieh — Sat, 23 May 2015 22:10:46 GMT

How do you identify "the particular batch of records" to process? Is it just events from the last hour or is that variable each hour? Are you processing multiple batches each hour? By "batch query logic should be modified dynamically" What determines the dynamics of this query? Something in how the search is launched? Something in the data of the batch? Something else?

Another potential thought is a combination of gentimes and map but without understanding what your goal is and what the data looks like it's still just a guess in the dark.

Re: Is dynamic query construction possble in Splunk without using any SDKs ?

woodcock — Sun, 24 May 2015 01:03:25 GMT

What I meant was that maybe you could abandon your current approach and start over using a macro-based approach.

Re: Is dynamic query construction possble in Splunk without using any SDKs ?

Arun_N_007 — Mon, 25 May 2015 06:05:04 GMT

Hi,

Use case is like,

I need to calculate the average completion time of Jobs.
Period is configurable.
I have list of Jobs in Lookup for which i need to caluclate averages.
In that list i have different type of Jobs for which calculation Logic (Due to time format and all...) differs.
Since Splunk got limit on transformation commands like stats,eventstats(50K) i need to make sure that scaling should be done based on limit.
I should be able to design buckets dynamically using queries. And Average query should run for each buckets without any manual intervention.

Re: Is dynamic query construction possble in Splunk without using any SDKs ?

woodcock — Mon, 25 May 2015 13:41:25 GMT

This is crazy but you could use the ouputlookup command to use the KV store as a registry of sorts like this:

Write your initial lookup :

index=* | eval HardCodedKey= 0 | stats first(HardCodedKey) AS HardCodedKey BY host | dedup HardCodedKey | eval NextSearchString="Your Inital or Default Search Here" | stats count by HardCodedKey, NextSearchString| outputlookup MyLookup

Then your scheduled search could do something like this (and also something like above, to refresh the registry for the next run):

index=* | eval HardCodedKey= 0 | stats first(HardCodedKey) AS HardCodedKey BY host | dedup HardCodedKey | lookup MyLookup HardCodedKey OUTPUT NextSearchString | map search="Some Search Stuff $NextSearchString$"

Re: Is dynamic query construction possble in Splunk without using any SDKs ?

Arun_N_007 — Tue, 26 May 2015 11:36:23 GMT

Thank you woodcock,
For showing unknown approach 🙂

But still map got limit of 10K (due to Sub Search limit) :)..

I know that if am using SDKs I can accomplish inside commands but I don't have SDK.

Re: Is dynamic query construction possble in Splunk without using any SDKs ?

Arun_N_007 — Wed, 27 May 2015 03:39:40 GMT

Hi Woodcock,

Any other way to achieve this use case other than using map..

Any equivalent to map without 10K limit???