topic Re: Splunk - Output of one query as an input to another query in Splunk Search

Splunk - Output of one query as an input to another query

kumarnis45 — Tue, 05 Oct 2021 00:49:26 GMT

Hi,

I have two different queries running on same dashboard but a different panel. Below is the query one which results the "reqid" as the output (ex:123456)

Query 1: sourcetype=test*-cloudwatch-logs file.txt | rex "RequestId: (?<reqid>[\S+]*)\s" | table reqid | dedup reqid

Output return as ex : 123456

Then, i would like feed the output of query1 as input of query2.

Query 2: $query1_output$ | rex "uploaded to: s3://sample-us-east-1-s3/transmit-os/(?<filename>.*)" | table filename

Can someone suggest me is this right way of passing? How can i update the source code (XML) for the changes?

Thanks.

Re: Splunk - Output of one query as an input to another query

richgalloway — Tue, 05 Oct 2021 12:39:23 GMT

One does not "pass" results from one query to another in Splunk dashboards. If you need to use the results of one query in another query then use post processing (https://docs.splunk.com/Documentation/Splunk/8.2.2/Viz/Savedsearches#Post-process_searches_2). Post processing defines a base search and one or more post-processing searches that refine or enhance the results of the base search.

In your example, Query 1 would be the base search and Query 2 the post-processing search. The code would look something like this

<panel> ... <search id="base"> <query>sourcetype=test*-cloudwatch-logs file.txt | rex "RequestId: (?<reqid>[\S+]*)\s" | dedup reqid | table reqid</query> </search> </panel> <panel> ... <search base="base"> <query>| rex "uploaded to: s3://sample-us-east-1-s3/transmit-os/(?<filename>.*)" | table filename</query> </search> </panel>

Re: Splunk - Output of one query as an input to another query

kumarnis45 — Tue, 05 Oct 2021 17:49:23 GMT

@richgalloway ,

The two queries are working individually great. Also I tried as you suggested , but its not returning anything. Even if running two queries together in splunk search not working. Can you please suggest in any another way i can pass the output of first query as an input to the second query?

I appreciate your help.

Thanks.

Re: Splunk - Output of one query as an input to another query

richgalloway — Tue, 05 Oct 2021 18:59:50 GMT

This is the only way, AFAIK, to share results among panels in a dashboard.

Please share your dashboard code (sanitized as necessary). Which panel is not returning results? Have you tried running the queries in a search window?

Re: Splunk - Output of one query as an input to another query

kumarnis45 — Tue, 05 Oct 2021 19:31:53 GMT

@richgalloway ,

I tried as you suggested. Below is the screen shot of running two commands as one in splunk search. It doesn't return anything. (same query runds through dashboard). Its taking the command as whole instaed of running first query and then pass it as an input to second query.

The code looks like this in xml file:

<row> <panel> <table> <search id="base"> <query>sourcetype=sample*-cloudwatch-logs file.txt | rex "RequestId: (?<reqid>[\S+]*)\s" | table reqid | dedup reqid</query> <earliest>-7d@h</earliest> <latest>now</latest> </search> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> <row> <panel> <table> <search base = "base"> <query>| rex "uploaded to: s3:\/\/sample.*?-test-.*?-us-east-1-s3/transmit-os/(?<filename>.*)" | table filename</query> </search> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row>

Please check and let me know what else i can do to make it work. Also, please check the attached pic of splunk running in my UI.

Thanks.

Re: Splunk - Output of one query as an input to another query

ITWhisperer — Tue, 05 Oct 2021 19:42:56 GMT

Try declaring the first search as a base search and base both panel on it, one with the straight results and the other with the additional search commands

Re: Splunk - Output of one query as an input to another query

richgalloway — Tue, 05 Oct 2021 19:44:29 GMT

The query in the screenshot looks good to me, but I don't know your data so I can't tell what may need to be corrected. In particular, does the reqid field contain text that matches what the rex command expects? I'm guessing it does not, which means the two queries cannot be linked. (Recall that the table command passes on only the named fields.)

Perhaps we need to refactor the queries. The base search should include everything up to (but not including) the first table command.

The second search, which will populate the first panel, uses the base search and adds | table reqid.

The third search populates the second panel as it currently does.

Re: Splunk - Output of one query as an input to another query

somesoni2 — Tue, 05 Oct 2021 20:19:02 GMT

Does the first query always returns one row with one field?

Re: Splunk - Output of one query as an input to another query

kumarnis45 — Tue, 05 Oct 2021 21:51:06 GMT

@somesoni2

Yes, thats right. It always returns Only one row with one field. How can i make my queries better to get the result i need?

Thank you so much for the response.

Thanks.

Re: Splunk - Output of one query as an input to another query

kumarnis45 — Wed, 06 Oct 2021 01:30:59 GMT

I tried using subsearch for the same as below. For some reason its not working with sub query.

The overall query for subsearch is,

query 2 is,

query 1 is,

[search sourcetype=sample*-cloudwatch-logs file.txt | rex "RequestId: (?<reqid>[\S+]*)\s" | table reqid | dedup reqid]

Its not returning anything in splunk search.

Re: Splunk - Output of one query as an input to another query

kumarnis45 — Wed, 06 Oct 2021 13:24:53 GMT

can you please share some examples i can try with?

Thanks.

Re: Splunk - Output of one query as an input to another query

richgalloway — Wed, 06 Oct 2021 13:33:23 GMT

Where did the subsearches come from? We never mentioned those in this thread. Subsearches behave differently and take this thread in a new direction.

FTR, there is nothing gained by starting a query with a subsearch (subsearches run first, anyway) nor by making the entire query a subsearch.

Re: Splunk - Output of one query as an input to another query

kumarnis45 — Wed, 06 Oct 2021 13:35:54 GMT

I came across this approach when i was looking for solution. So i just gave a try to check if it works. I am still looking for the solution for this issue. I am really not sure what else i can try

Re: Splunk - Output of one query as an input to another query

somesoni2 — Wed, 06 Oct 2021 13:44:05 GMT

I think this approach might work for you.

Step 1: Run your first search/query1 (which gives a single row/column result) and use approach from below link to capture the result as token. In this example, you'll be displaying the result as table as well as saving it as a token.

https://community.splunk.com/t5/Dashboards-Visualizations/How-do-you-store-search-results-in-a-token-or-variable/m-p/424746

Step 2: Use the token generated in Step 1 in your second search/query2.

Now, you can do a text base search (like google search) in your query2 but it's better to specify the index/sourcetype you want to search against, it'll perform much better.

Re: Splunk - Output of one query as an input to another query

kumarnis45 — Wed, 06 Oct 2021 14:16:35 GMT

Thanks. It worked 👏