Dashboards & Visualizations

How to pass base search results to subsearch

dougburdan
Explorer

Hi Folks,

We receive several hundred files per day from 20 different sources. The filenames contain the source that we received the file from, and have a three digit sequence number as a suffix.

Occasionally a file gets lost in transit, so I have designed a dashboard with 20 panels (one for each source) to highlight missing files by doing a makeresults and then a streamstats to generate a list of sequence numbers, and then a join to a search which extracts the sequence numbers from the filenames received, and then any sequence numbers that are not 'joined' to a filename are flagged as missing files.

To make the dashboard more efficient, I'm trying to implement a base search to list the files from all sources, which I then want to pass to my subsearches - I have to use subsearches because of the makeresults which generates the full list of sequence numbers.

(please see a cut-down version of the code below)

However, it seems that the subsearches are unable to read my base search.

I see that this question has been asked a few times in this forum, but none of the questions I viewed have accepted answers, and none of them were trying to use the same technique.

So I just wanted to check . . . is there a way to pass base search results to subsearches?

If not, is there another strategy that I could use to detect missing files?

Thanks,
Doug.

<dashboard>
  <label>Base Post Question</label>
  <search id="filelist">
    <query>           
      my base search which extracts filenames and the times that they arrived
      | eval source=substr(filename,1,3)
      | eval seq=ltrim(substr(filename,14,3),"0")
      | table _time filename source seq
    </query>
    <earliest>-24h</earliest>
    <latest>now</latest>
    <sampleRatio>1</sampleRatio>
    <refresh>1h</refresh>
    <refreshType>interval</refreshType>
  </search>
  <row>
    <panel>
      <table>
        <search>
          <query>
            | makeresults count=99
            | streamstats count as seq
            | join type=left seq
              [ | search base="filelist" source="ABC"
                | table _time filename source seq ]
            | eval filename=if(isnull(filename),"Missing File!",filename)
            | table _time filename
          </query>
        </search>
      </table>
    </panel>
  </row>
</dashboard>
0 Karma

cmerriman
Super Champion

base searches do not work like that. you can't add it as an identifier inside of a search string, it only works inside of the node. You'll probably want to use |loadjob

create a token with the sid from your base search, something like

<done>
<set token="sid">$job.sid$</set>
</done>

and then within your subsearch, something like

|loadjob $sid$.....
0 Karma

dougburdan
Explorer

Thanks cmerriman, I did see a similar answer in this forum, but I couldn't get it to work.

It used index=_internal, which I didn't have access to (I'm just a user - not admin), so I applied for and got access, but it still didn't work, so maybe the _internal index was just because it was a 'run anywhere' example?

Here is an example where I've removed index=_internal and added some code to show that the SID has been assigned to the token, but my loadjob still shows no results - are you able to tell me where I am going wrong?

(I'm relatively new to Splunk so it's probably some rookie mistake ;o)

Thanks,
Doug.

<dashboard>
<label>Test SID</label>
  <search>
    <query>
      | makeresults count=99
      | streamstats count as seq
    </query>
    <earliest>-24h@h</earliest>
    <latest>now</latest>
    <done>
      <condition>
        <set token="subsearch_sid">$job.sid$</set>
      </condition>
    </done>
  </search>
  <row>
    <panel>
      <table>
        <title>Token Values</title>
        <search>
          <query>
            | makeresults
            | eval subsearch_sid = "$subsearch_sid$"
            | table subsearch_sid
          </query>
        </search>
        <option name="drilldown">none</option>
      </table>
    </panel>
  </row>
<row>
  <panel>
    <event>
      <search>
        <query>
          | loadjob $subsearch_sid$
          | table seq
        </query>
        <earliest>-24s@s</earliest>
        <latest>now</latest>
        <sampleRatio>1</sampleRatio>
      </search>
      <option name="list.drilldown">none</option>
    </event>
  </panel>
</row>
</dashboard>
0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

&#x1f342; Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...