Splunk - Output of one query as an input to another query

[kumarnis45]

Hi,

   I have two different queries running on same dashboard but a different panel.  Below is the query one which results the "reqid" as the output (ex:123456)

Query 1: sourcetype=test*-cloudwatch-logs file.txt | rex "RequestId: (?<reqid>[\S+]*)\s" | table reqid | dedup reqid

Output return as ex : 123456

Then, i would like feed the output of query1 as input of query2.

Query 2:  $query1_output$ | rex "uploaded to: s3://sample-us-east-1-s3/transmit-os/(?<filename>.*)" | table filename

  Can someone suggest me is this right way of passing? How can i update the source code (XML) for the changes?

Thanks.

[somesoni2]

Does the first query always returns one row with one field?

[kumarnis45]

@somesoni2 

       Yes, thats right. It always returns Only one row with one field. How can i make my queries better to get the result i need?

  Thank you so much for the response.

Thanks.

[somesoni2 · Accepted solution]

I think this approach might work for you.

Step 1: Run your first search/query1 (which gives a single row/column result) and use approach from below link to capture the result as token. In this example, you'll be displaying the result as table as well as saving it as a token.

https://community.splunk.com/t5/Dashboards-Visualizations/How-do-you-store-search-results-in-a-token...

Step 2: Use the token generated in Step 1 in your second search/query2.

Now, you can do a text base search (like google search) in your query2 but it's better to specify the index/sourcetype you want to search against, it'll perform much better.

[kumarnis45]

Thanks. It worked 👏

[ITWhisperer]

Try declaring the first search as a base search and base both panel on it, one with the straight results and the other with the additional search commands

[kumarnis45]

can you please share some examples i can try with?

Thanks.

[richgalloway]

One does not "pass" results from one query to another in Splunk dashboards.  If you need to use the results of one query in another query then use post processing (https://docs.splunk.com/Documentation/Splunk/8.2.2/Viz/Savedsearches#Post-process_searches_2).  Post processing defines a base search and one or more post-processing searches that refine or enhance the results of the base search.

In your example, Query 1 would be the base search and Query 2 the post-processing search.  The code would look something like this

<panel>
  ...
  <search id="base">
    <query>sourcetype=test*-cloudwatch-logs file.txt | rex "RequestId: (?&lt;reqid&gt;[\S+]*)\s" | dedup reqid | table reqid</query>
  </search>
</panel>
<panel>
  ...
  <search base="base">
    <query>| rex "uploaded to: s3://sample-us-east-1-s3/transmit-os/(?&lt;filename&gt;.*)" | table filename</query>
  </search>
</panel>

[kumarnis45]

@richgalloway ,

   The two queries are working individually great. Also I tried as you suggested , but its not returning anything. Even if running two queries together in splunk search not working. Can you please suggest in any another way i can pass the output of first query as an input to the second query?

I appreciate your help.

Thanks.

[richgalloway]

This is the only way, AFAIK, to share results among panels in a dashboard.

Please share your dashboard code (sanitized as necessary).  Which panel is not returning results?  Have you tried running the queries in a search window?

[kumarnis45]

@richgalloway ,

   I tried as you suggested. Below is the screen shot of running two commands as one in splunk search. It doesn't return anything. (same query runds through dashboard). Its taking the command as whole instaed of running first query and then pass it as an input to second query.

The code looks like this in xml file:

<row>
  <panel>
    <table>
     <search id="base">
       <query>sourcetype=sample*-cloudwatch-logs file.txt | rex "RequestId: (?&lt;reqid&gt;[\S+]*)\s" | table reqid | dedup reqid</query>
     <earliest>-7d@h</earliest>
    <latest>now</latest>
   </search>
   <option name="drilldown">none</option>
   <option name="refresh.display">progressbar</option>
   </table>
  </panel>
</row>
<row>
<panel>
  <table>
   <search base = "base">
    <query>| rex "uploaded to: s3:\/\/sample.*?-test-.*?-us-east-1-s3/transmit-os/(?&lt;filename&gt;.*)" | table filename</query>
   </search>
   <option name="drilldown">none</option>
   <option name="refresh.display">progressbar</option>
  </table>
 </panel>
</row>

Please check and let me know what else i can do to make it work. Also, please check the attached pic of splunk running in my UI.

Thanks.

[richgalloway]

The query in the screenshot looks good to me, but I don't know your data so I can't tell what may need to be corrected.  In particular, does the reqid field contain text that matches what the rex command expects?  I'm guessing it does not, which means the two queries cannot be linked. (Recall that the table command passes on only the named fields.)

Perhaps we need to refactor the queries.  The base search should include everything up to (but not including) the first table command.

The second search, which will populate the first panel, uses the base search and adds | table reqid.

The third search populates the second panel as it currently does. 

[kumarnis45]

I tried using subsearch for the same as below. For some reason its not working with sub query. 

The overall query for subsearch is,

query 2 is,

[search sourcetype=sample*-cloudwatch-logs file.txt | rex "RequestId: (?<reqid>[\S+]*)\s" | table reqid | dedup reqid] | rex "uploaded to: s3:\/\/sample.*?-test-.*?-us-east-1-s3/(?<json>.*)" | table json | where isnotnull(json)

query 1 is,

[search sourcetype=sample*-cloudwatch-logs file.txt | rex "RequestId: (?<reqid>[\S+]*)\s" | table reqid | dedup reqid]

Its not returning anything in splunk search.

[richgalloway]

Where did the subsearches come from?  We never mentioned those in this thread.  Subsearches behave differently and take this thread in a new direction.

FTR, there is nothing gained by starting a query with a subsearch (subsearches run first, anyway) nor by making the entire query a subsearch.

[kumarnis45]

I came across this approach when i was looking for solution. So i just gave a try to check if it works. I am still looking for the solution for this issue. I am really not sure what else i can try