Splunk Search

Splunk - Output of one query as an input to another query

kumarnis45
Path Finder

Hi,

   I have two different queries running on same dashboard but a different panel.  Below is the query one which results the "reqid" as the output (ex:123456)

Query 1: sourcetype=test*-cloudwatch-logs file.txt | rex "RequestId: (?<reqid>[\S+]*)\s" | table reqid | dedup reqid

Output return as ex : 123456

 

Then, i would like feed the output of query1 as input of query2.

 

Query 2:  $query1_output$ | rex "uploaded to: s3://sample-us-east-1-s3/transmit-os/(?<filename>.*)" | table filename

 

  Can someone suggest me is this right way of passing? How can i update the source code (XML) for the changes?

 

Thanks.

Labels (1)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

I think this approach might work for you.

Step 1: Run your first search/query1 (which gives a single row/column result) and use approach from below link to capture the result as token. In this example, you'll be displaying the result as table as well as saving it as a token.

https://community.splunk.com/t5/Dashboards-Visualizations/How-do-you-store-search-results-in-a-token...

Step 2: Use the token generated in Step 1 in your second search/query2.

Now, you can do a text base search (like google search) in your query2 but it's better to specify the index/sourcetype you want to search against, it'll perform much better.

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

Does the first query always returns one row with one field?

Tags (1)
0 Karma

kumarnis45
Path Finder

@somesoni2 

       Yes, thats right. It always returns Only one row with one field. How can i make my queries better to get the result i need?

  Thank you so much for the response.

 

Thanks.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I think this approach might work for you.

Step 1: Run your first search/query1 (which gives a single row/column result) and use approach from below link to capture the result as token. In this example, you'll be displaying the result as table as well as saving it as a token.

https://community.splunk.com/t5/Dashboards-Visualizations/How-do-you-store-search-results-in-a-token...

Step 2: Use the token generated in Step 1 in your second search/query2.

Now, you can do a text base search (like google search) in your query2 but it's better to specify the index/sourcetype you want to search against, it'll perform much better.

kumarnis45
Path Finder

Thanks. It worked 👏

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try declaring the first search as a base search and base both panel on it, one with the straight results and the other with the additional search commands

0 Karma

kumarnis45
Path Finder

can you please share some examples i can try with?

 

Thanks.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

One does not "pass" results from one query to another in Splunk dashboards.  If you need to use the results of one query in another query then use post processing (https://docs.splunk.com/Documentation/Splunk/8.2.2/Viz/Savedsearches#Post-process_searches_2).  Post processing defines a base search and one or more post-processing searches that refine or enhance the results of the base search.

In your example, Query 1 would be the base search and Query 2 the post-processing search.  The code would look something like this

<panel>
  ...
  <search id="base">
    <query>sourcetype=test*-cloudwatch-logs file.txt | rex "RequestId: (?&lt;reqid&gt;[\S+]*)\s" | dedup reqid | table reqid</query>
  </search>
</panel>
<panel>
  ...
  <search base="base">
    <query>| rex "uploaded to: s3://sample-us-east-1-s3/transmit-os/(?&lt;filename&gt;.*)" | table filename</query>
  </search>
</panel>
---
If this reply helps you, Karma would be appreciated.
0 Karma

kumarnis45
Path Finder

@richgalloway ,

 

   The two queries are working individually great. Also I tried as you suggested , but its not returning anything. Even if running two queries together in splunk search not working. Can you please suggest in any another way i can pass the output of first query as an input to the second query?

I appreciate your help.

 

Thanks.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

This is the only way, AFAIK, to share results among panels in a dashboard.

Please share your dashboard code (sanitized as necessary).  Which panel is not returning results?  Have you tried running the queries in a search window?

---
If this reply helps you, Karma would be appreciated.
0 Karma

kumarnis45
Path Finder

@richgalloway ,

   I tried as you suggested. Below is the screen shot of running two commands as one in splunk search. It doesn't return anything. (same query runds through dashboard). Its taking the command as whole instaed of running first query and then pass it as an input to second query.

The code looks like this in xml file:

 

 

<row>
  <panel>
    <table>
     <search id="base">
       <query>sourcetype=sample*-cloudwatch-logs file.txt | rex "RequestId: (?&lt;reqid&gt;[\S+]*)\s" | table reqid | dedup reqid</query>
     <earliest>-7d@h</earliest>
    <latest>now</latest>
   </search>
   <option name="drilldown">none</option>
   <option name="refresh.display">progressbar</option>
   </table>
  </panel>
</row>
<row>
<panel>
  <table>
   <search base = "base">
    <query>| rex "uploaded to: s3:\/\/sample.*?-test-.*?-us-east-1-s3/transmit-os/(?&lt;filename&gt;.*)" | table filename</query>
   </search>
   <option name="drilldown">none</option>
   <option name="refresh.display">progressbar</option>
  </table>
 </panel>
</row>

 

 

Please check and let me know what else i can do to make it work. Also, please check the attached pic of splunk running in my UI.

 

Thanks.

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The query in the screenshot looks good to me, but I don't know your data so I can't tell what may need to be corrected.  In particular, does the reqid field contain text that matches what the rex command expects?  I'm guessing it does not, which means the two queries cannot be linked. (Recall that the table command passes on only the named fields.)

Perhaps we need to refactor the queries.  The base search should include everything up to (but not including) the first table command.

The second search, which will populate the first panel, uses the base search and adds | table reqid.

The third search populates the second panel as it currently does. 

---
If this reply helps you, Karma would be appreciated.
0 Karma

kumarnis45
Path Finder

I tried using subsearch for the same as below. For some reason its not working with sub query. 

The overall query for subsearch is,

query 2 is,

[search sourcetype=sample*-cloudwatch-logs file.txt | rex "RequestId: (?<reqid>[\S+]*)\s" | table reqid | dedup reqid] | rex "uploaded to: s3:\/\/sample.*?-test-.*?-us-east-1-s3/(?<json>.*)" | table json | where isnotnull(json)

query 1 is,

[search sourcetype=sample*-cloudwatch-logs file.txt | rex "RequestId: (?<reqid>[\S+]*)\s" | table reqid | dedup reqid]

 

Its not returning anything in splunk search.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Where did the subsearches come from?  We never mentioned those in this thread.  Subsearches behave differently and take this thread in a new direction.

FTR, there is nothing gained by starting a query with a subsearch (subsearches run first, anyway) nor by making the entire query a subsearch.

---
If this reply helps you, Karma would be appreciated.
0 Karma

kumarnis45
Path Finder

I came across this approach when i was looking for solution. So i just gave a try to check if it works. I am still looking for the solution for this issue. I am really not sure what else i can try 

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...