Re: How to display all events of a source if the s...

mawomommoh · ‎03-22-2018

I am getting my events from an xml file. In that xml file, there is tag which gives the BuildID of a software. For instance: <BuildID>SoftwareBuildVer.2.0.6</BuildID>. I have been able to extract the SoftwareBuildVer.2.0.6 as a BuildID field in Splunk using regex. I want to be able to use BuildID + DateTime of the file creation to uniquely identify the source in Splunk. I have been able to create a final field, BuildAndDate, which has values that looks like this: SoftwareBuildVer.2.0.6 March 22, 2018 12:05PM. I plan on using the values in this field in a dashboard dropdown menu so that when a user selects a particular BuildAndDate value from the dropdown, the source which contains that particular BuildID is returned, and a new dashboard is generated based on the newly determined source value. Here is my simple xml for the dashboard (changes in dropdown to use BuildAndDate field has not been added yet):

<form>
  <label>XXXX_Dashboard</label>
  <fieldset submitButton="false" autoRun="true">
    <input type="dropdown" token="source_tok" searchWhenChanged="true">
      <label>Selected BuildID/DateTime</label>
      <fieldForLabel>source</fieldForLabel>
      <fieldForValue>source</fieldForValue>
      <search>
        <query>| tstats max(_time) AS mostRecent where host=XXXX sourcetype=XXXX groupby source, sourcetype, host| sort -mostRecent
| fields source 
| rex mode=sed field=source "s/(\\\)/\1\1\1/g"</query>
        <earliest>0</earliest>
        <latest></latest>
      </search>
      <prefix>source="</prefix>
      <suffix>"</suffix>
    </input>
  </fieldset>
  <row>
    <panel>
      <html depends="$alwaysHideCSSOverride$">
        <style>
          .select2-container .select2-choice {width: 520px;} 

          .select2-drop .select2-search {width: 528px; background-color: white;} 

          .select2-drop .select2-results {width: 528px;}

          .select2-drop-active {width: 530px !important;}

          .select2-container-active {width: 530px !important;}

          .splunk-status-indicator {
            border-radius: 15px !important;
            border: 2px solid white;
            font-size: 10px !important;
            line-height: normal !important;
            white-space: normal !important;
            <!--word-spacing: 100vw !important;-->
            overflow-x: hidden !important;
            display: flex;
            justify-content: center;
            align-items: center;
            word-break: break-word;
            padding: 3px;
          }

          div.facet-label {
            visibility: hidden !important;
          }

          div.viz-facet {
            <!--border: 2px solid red !important;-->
            padding:0 !important;
            height: 30% !important;
            width: 5% !important;
            margin: 0 15px 15px 0 !important;
          }

          div.facets-container {
            <!--background-color: lightgrey;-->
            background-image: url("/static/app/status_indicator_app/images/arrow.png");
            background-repeat: no-repeat;
            background-size: 100% 45%;
            margin-top: 55px;
          }
        </style>
      </html>
    </panel>
  </row>
  <row>
    <panel>
      <viz type="status_indicator_app.status_indicator">
        <search>
          <query>host=XXXX index=XXXX $source_tok$ | eval NewTime=strptime(StartTime,"%Y-%m-%dT%H:%M:%S.%3N") | eval _time=NewTime | eventstats max(_time) AS latestScan by Description  | where _time=latestScan | stats  count by StepResult, Description | eval  color=case(StepResult=="Passed","green",StepResult=="Skipped","gold", StepResult=="Failed","red")   | eval  StepResult = Description</query>
          <earliest>$earliest$</earliest>
          <latest>$latest$</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="height">275</option>
        <option name="refresh.display">progressbar</option>
        <option name="status_indicator_app.status_indicator.colorBy">field_value</option>
        <option name="status_indicator_app.status_indicator.fillTarget">background</option>
        <option name="status_indicator_app.status_indicator.fixIcon">check</option>
        <option name="status_indicator_app.status_indicator.icon">field_value</option>
        <option name="status_indicator_app.status_indicator.precision">0</option>
        <option name="status_indicator_app.status_indicator.showOption">1</option>
        <option name="status_indicator_app.status_indicator.staticColor">#555</option>
        <option name="status_indicator_app.status_indicator.useColors">true</option>
        <option name="status_indicator_app.status_indicator.useThousandSeparator">true</option>
        <option name="trellis.enabled">1</option>
        <option name="trellis.size">small</option>
        <option name="trellis.splitBy">Description</option>
      </viz>
    </panel>
  </row>
</form>

The dashboard currently works, but it directly makes use of the source in the dropdown. I want the BuildAndDate to be what shows up in the dropdown, and it should be used to determine the source which is then used to produce the new dashboard.

The issue I am having is mapping the BuildAndDate to its source, and the source then displaying all events in the source based on that. RIght now, clicking on a BuildAndDate value only returns one event, i.e. one line in the xml file ( <BuildID>SoftwareBuildVer.2.0.6</BuildID> ) instead of returning all the events in the xml in which <BuildID>SoftwareBuildVer.2.0.6</BuildID> is contained in.

This query produces the BuildAndDate field and value as expected:

host=XXXX sourcetype=XXXX | eval  mostRecent=strftime(_time, "%B %d, %Y %I:%M%p")
| eval BuildAndDate = BuildID+" "+mostRecent 
| stats values(BuildAndDate ) as BuildAndDate

Goal: Click on BuildAndDate value in dropdown and generate new dashboard via that selection.

Any help would be appreciated.

Thanks in advance.

mawomommoh · ‎03-23-2018

Yes, this works:

host=XXXX sourcetype=XXX| eval  mostRecent=strftime(_time, "%B %d, %Y %I:%M%p")
| eval BuildAndDate  = BuildID+" "+mostRecent 
| stats count by source

but whenever I put in the BuildAndDate or BuildID field in the stats command, it doesn't work. For instance, this doesn't work:

host=WP3L2M6VNX1 sourcetype=SAMBAXMLTEST | eval  mostRecent=strftime(_time, "%B %d, %Y %I:%M%p")
| eval BuildAndDate  = BuildID+" "+mostRecent 
| stats count by source, BuildAndDate

This also doesn't work:

host=WP3L2M6VNX1 sourcetype=SAMBAXMLTEST | eval  mostRecent=strftime(_time, "%B %d, %Y %I:%M%p")
    | eval BuildAndDate  = BuildID+" "+mostRecent 
    | stats count by source, BuildID

These two queries (which don't work) work fine in the Search & Reporting App, it only fails to work when I insert it in the Search criteria in the dropdown.

maciep · ‎03-23-2018

are you building this dashboard in the search&reporting app as well? Is it possible that your BuildID field extraction isn't shared in the same app as the dashboard your building?

mawomommoh · ‎03-23-2018

I am building the dashboard in a different app, but the BuildID is globally shared.

I discovered where the issue was coming from. It had to do with the _time field. When I changed mostRecent=strftime(_time, "%B %d, %Y %I:%M%p") to mostRecent=strftime(mostRecent, "%B %d, %Y %I:%M%p") it kind of works (It returns the values for the BuildID in the dropdown). So this works: | stats count by source, BuildID but this still doesn't work: | stats count by source, BuildAndDate.

Any thoughts?

maciep · ‎03-26-2018

is there a base search in play here? How do you already have a mostRecent field? This is one of those times I wish i could just get hands on keyboard, because I feel like something's getting lost in translation here. Building a dropdown should't take this much effort 🙂

mawomommoh · ‎03-26-2018

I tried this query again this morning and it now works in the dropdown:

host=XXXX sourcetype=XXXX | eval  mostRecent=strftime(_time, "%B %d, %Y %I:%M%p")
| eval BuildAndDate  = BuildID+" "+mostRecent 
| stats count by source,BuildAndDate

I can't really pinpoint what exactly the solution was, but before now I deleted the BuildID field in Field Extractions, and re-extracted it from the xml file. I still tested out the query above after doing this but it did not work, but it surprisingly works this morning. The dropdown now produces the BuildAndDate field values as expected, and the new dashboard is being generated when a value is selected from the dropdown.

Side note: At some point I opened two tabs of the Search & Reporting app and ran the following query:

host=WP3L2M6VNX1 sourcetype=SAMBAXMLTEST | eval  mostRecent=strftime(_time, "%B %d, %Y %I:%M%p")
| eval BuildAndDate  = BuildID+" "+mostRecent 
| stats count by source,BuildAndDate

In one of the tabs, the BuildID field did not show up in the fields area (it was also absent is the More fields area), while in the other tab, the BuildID field showed up. Both queries were ran using the same criteria (All time) & Verbose Mode. It was strange to see that the same query ran in the same app worked in one tab and not the other. It seems part of the app was not detecting the extracted field (don't know if that's possible), but I think this may be related to the dropdown issue.

Thanks so much @maciep for your assistance. 🙂 I believe you helped me drilldown to the potential cause of the issue.

Thanks @Sukisen1981 for the suggestion.

Sukisen1981 · ‎03-24-2018

well, you say - These two queries (which don't work) work fine in the Search & Reporting App, it only fails to work when I insert it in the Search criteria in the dropdown.

I am assuming you have a dashboard panel for this (or create a dashboard panel to test), now when you run these 2 queries from the dashboard, click on the magnifying glass (search) button for the panel, append your query (remove stats command) with something like |table BuildAndDate,BuildID,mostRecent.

This should show you how the dropdown actually is interpreted by Splunk, results should open up in a new window (along with the search as well). Invesitgate both the search and the table, do you see any issues in the way the tokens are passed and is being interpreted in the dashboard search and of course your table results?

maciep · ‎03-22-2018

I'm not going to lie, I'm not sure I completely follow what you're trying to accomplish. But it sounds like you want to display to the user in the dropdown a BuildID/Date field to select. But you when they select one, you want to populate the token with source associated with just the BuildID.

If that's the case, can you just replace that last stats with this?

| stats count by BuildAndDate, source

Then use BuildAndDate for the "label" field of your dropdown, and source as the "value" field. So then users are shown the BuildAndDate, but the token is populated with the associated source?

mawomommoh · ‎03-23-2018

Yes, you're right. That's basically what I am trying to accomplish.

I replaced the last stats as you said, and it returns the expected result in the Search app, however when I insert the same search query in the Search found in the dropdown, I get a message saying 'Search produced no results', so the dropdown is not being populated. Any idea why the search does not work in the dropdown?

maciep · ‎03-23-2018

Hmm, not sure. Is the time scope for the dropdown search the same as what you used in the search app?

And i don't think it matters, but might be worth trying to drop the count field. Just add | fields - count after the stats.

mawomommoh · ‎03-23-2018

Thanks, but it still doesn't work. Quite strange.

maciep · ‎03-23-2018

does any search work?

How to display all events of a source if the source contains a particular field?

ICYMI - Check out the latest releases of Splunk Edge Processor

Introducing the 2024 SplunkTrust!

Introducing the 2024 Splunk MVPs!