How to pass base search results to subsearch

[dougburdan]

Hi Folks,

We receive several hundred files per day from 20 different sources. The filenames contain the source that we received the file from, and have a three digit sequence number as a suffix.

Occasionally a file gets lost in transit, so I have designed a dashboard with 20 panels (one for each source) to highlight missing files by doing a makeresults and then a streamstats to generate a list of sequence numbers, and then a join to a search which extracts the sequence numbers from the filenames received, and then any sequence numbers that are not 'joined' to a filename are flagged as missing files.

To make the dashboard more efficient, I'm trying to implement a base search to list the files from all sources, which I then want to pass to my subsearches - I have to use subsearches because of the makeresults which generates the full list of sequence numbers.

(please see a cut-down version of the code below)

However, it seems that the subsearches are unable to read my base search.

I see that this question has been asked a few times in this forum, but none of the questions I viewed have accepted answers, and none of them were trying to use the same technique.

So I just wanted to check . . . is there a way to pass base search results to subsearches?

If not, is there another strategy that I could use to detect missing files?

Thanks,
Doug.

<dashboard>
  <label>Base Post Question</label>
  <search id="filelist">
    <query>           
      my base search which extracts filenames and the times that they arrived
      | eval source=substr(filename,1,3)
      | eval seq=ltrim(substr(filename,14,3),"0")
      | table _time filename source seq
    </query>
    <earliest>-24h</earliest>
    <latest>now</latest>
    <sampleRatio>1</sampleRatio>
    <refresh>1h</refresh>
    <refreshType>interval</refreshType>
  </search>
  <row>
    <panel>
      <table>
        <search>
          <query>
            | makeresults count=99
            | streamstats count as seq
            | join type=left seq
              [ | search base="filelist" source="ABC"
                | table _time filename source seq ]
            | eval filename=if(isnull(filename),"Missing File!",filename)
            | table _time filename
          </query>
        </search>
      </table>
    </panel>
  </row>
</dashboard>

[cmerriman]

base searches do not work like that. you can't add it as an identifier inside of a search string, it only works inside of the node. You'll probably want to use |loadjob

create a token with the sid from your base search, something like

<done>
<set token="sid">$job.sid$</set>
</done>

and then within your subsearch, something like

|loadjob $sid$.....

[dougburdan]

Thanks cmerriman, I did see a similar answer in this forum, but I couldn't get it to work.

It used index=_internal, which I didn't have access to (I'm just a user - not admin), so I applied for and got access, but it still didn't work, so maybe the _internal index was just because it was a 'run anywhere' example?

Here is an example where I've removed index=_internal and added some code to show that the SID has been assigned to the token, but my loadjob still shows no results - are you able to tell me where I am going wrong?

(I'm relatively new to Splunk so it's probably some rookie mistake ;o)

Thanks,
Doug.

<dashboard>
<label>Test SID</label>
  <search>
    <query>
      | makeresults count=99
      | streamstats count as seq
    </query>
    <earliest>-24h@h</earliest>
    <latest>now</latest>
    <done>
      <condition>
        <set token="subsearch_sid">$job.sid$</set>
      </condition>
    </done>
  </search>
  <row>
    <panel>
      <table>
        <title>Token Values</title>
        <search>
          <query>
            | makeresults
            | eval subsearch_sid = "$subsearch_sid$"
            | table subsearch_sid
          </query>
        </search>
        <option name="drilldown">none</option>
      </table>
    </panel>
  </row>
<row>
  <panel>
    <event>
      <search>
        <query>
          | loadjob $subsearch_sid$
          | table seq
        </query>
        <earliest>-24s@s</earliest>
        <latest>now</latest>
        <sampleRatio>1</sampleRatio>
      </search>
      <option name="list.drilldown">none</option>
    </event>
  </panel>
</row>
</dashboard>