Splunk Search

How do I create a search that dynamically finds a targeted variable in the results, then uses it to find more results?

Explorer

I've been struggling with this one for about a week now.

I would like to create a search on a dashboard that shows all events related to a known variable (REF-ID_xxxxxx) and an otherwise undefined variable we will call (CON-ID). The catch is that the CON-ID number is sometimes referenced as CON-ID=, or CO=, or IDCoN, and a handful of others. The trailing number for that second variable is unique, even though the variable name isn't. So it comes down to you doing a manual search like this.

REF-ID_234d23dd23f

Which gives 30 some odd results. You then have to by eye look for something like 'CON-ID=Ct774235fffrf4345gf' in the first few records. Take all the alpha numeric characters behind the '=', in this case 'Ct774235fffrf4345gf'. Then do a second search like this.

REF-ID_234d23dd23f OR Ct774235fffrf4345gf

That gives you all 150+ events to give you a full view of related logs to investigate.

I've tried something like this below, but am not having much luck. Tried regex extraction, sub searches. I just am not sure what the best way is to proceed. Or even if I'm doing them correctly.

The manual fancy version I attempted... and failed at...

REF-ID_234d23dd23f OR [search REF-ID_234d23dd23f | rex field=_raw "CON-ID=(?<ConID>[a-zA-Z0-9].+)" | eval foundConID=ConID]

This is an example, based on above, of the dashboard version that uses an input variable, eventually what I want this to be something like.

$REF-ID$ OR [search $REF-ID$ | rex field=_raw "CON-ID=(?<ConID>[a-zA-Z0-9].+)" | eval foundConID=ConID]

None of these seem to give me the results I'm looking for past the first search. But if all works out, I will have a dashboard, where a guy puts a REF-ID number in, and gets back all kinds of correlated data. Showing an entire incident or series of events from start to finish, as it jumps across multiple systems and log sources.

Any suggestions on how to make this search work? I'm at a loss.

1 Solution

Motivator

The | format command is good when you want to see what a subsearch will return to the parent search. I've never used it like jeffland suggests though from reading the documentation a bit it might work the same as what I am about to suggest. The way I've typically used it is as a sanity check for subsearch results.

I think what you are looking for though is the raw data from your subsearch. For example what you aren't looking for is ConID=foo OR ConID=bar but "foo" OR "bar". For that you have to be a little sneaky and rename your field to search (I think you can do query as well)

REF-ID_234d23dd23f OR [search REF-ID_234d23dd23f | rex field=_raw "CON-ID=(?<ConID>[a-zA-Z0-9].+)" | rename ConID as search | fields search]

I'm going from memory so you might have to play with it just a bit. I feel like Splunk has introduced return and format in place of renaming fields search or query but not sure which proceeded which and which might be more elegant/appropriate. Obviously if you are able to specify metadata fields as part of your search it will make it that much quicker (index, sourcetype, etc) especially as subsearches time out after 60 sec.

View solution in original post

Motivator

The | format command is good when you want to see what a subsearch will return to the parent search. I've never used it like jeffland suggests though from reading the documentation a bit it might work the same as what I am about to suggest. The way I've typically used it is as a sanity check for subsearch results.

I think what you are looking for though is the raw data from your subsearch. For example what you aren't looking for is ConID=foo OR ConID=bar but "foo" OR "bar". For that you have to be a little sneaky and rename your field to search (I think you can do query as well)

REF-ID_234d23dd23f OR [search REF-ID_234d23dd23f | rex field=_raw "CON-ID=(?<ConID>[a-zA-Z0-9].+)" | rename ConID as search | fields search]

I'm going from memory so you might have to play with it just a bit. I feel like Splunk has introduced return and format in place of renaming fields search or query but not sure which proceeded which and which might be more elegant/appropriate. Obviously if you are able to specify metadata fields as part of your search it will make it that much quicker (index, sourcetype, etc) especially as subsearches time out after 60 sec.

View solution in original post

Explorer

Runals,
This is spot on! Thank you so much!

I was even able to add to it at the end
| timechart count by host | sort -count
Which made it possible for me to graph the results as well. Giving me two searches in the dashboard that look like this.

To see Log Details for all related ID's, if I run it manually

REF-ID_234d23dd23f OR [search REF-ID_234d23dd23f | rex field=_raw "CON-ID=(?<CON-ID>[a-zA-Z0-9].+)" | rename CON-ID as search | fields search]

If I want it in a dashboard
$REF-ID$ OR [search $REF-ID$ | rex field=_raw "CON-ID=(?<CON-ID>[a-zA-Z0-9].+)" | rename CON-ID as search | fields search]

Then I did this to get a pretty graph for the second search at the top if I do a manual search

REF-ID_234d23dd23f OR [search REF-ID_234d23dd23f | rex field=_raw "CON-ID=(?<CON-ID>[a-zA-Z0-9].+)" | rename CON-ID as search | fields search] | timechart count by host | sort -count

if I put it in a dashboard it looks like this

$REF-ID$ OR [search $REF-ID$ | rex field=_raw "CON-ID=(?&lt;CON-ID&gt;[a-zA-Z0-9].+)" | rename CON-ID as search | fields search] | timechart count by host | sort -count

That second one there with the timechart at the end allows me to see on a bar graph, at a glance, the events jump back and forth between systems.

Hi, You can solve you problem by creating a dashboard as follow:

1- your dashboard should have two input(may be dropdown)
- one for populated by a search that produce the values of REF-XXXX XXXXXX

  • the other populated by a search that gives you all the appropriate value of CON-ID,CO an IDcon.

2- your dashboard should have a main query (or more than one ) filter by the tokens of the two inputs.

3- Note that if there is a dependency between REF-XXXXXXXXX and your CONIDs, you can use the token of one to filter the populating search of the other.

BY DOING SO IT WILL BE OK IF you have you have goods populating searches and good query.

As you are new you can provide a sample of you data to have more help with the queries.

Explorer

stephane_cyrille,
Thanks for the suggestion, and to use token's. I did put that into my dashboard based on the input. I do have two searches running, each for their own part of the dashboard. I haven't played with a main query, but I'll defiantly check that out.

0 Karma

Hi sethrice, just to tell you that to manifest you gratitude as you did by awarding me a point, you can vote. It cost nothing.
Thanks to Runals.

0 Karma

Champion

Your first attempt wasn't far off I believe. For what you want, you could use the format command in your subsearch, which returns the results of your subsearch as a list of your results OR'ed together like ((CON-ID = Ct774235fffrf4345gf) OR (CON-ID = ...)). This means you can form your search like this:

REF-ID_234d23dd23f OR [search REF-ID_234d23dd23f | rex field=_raw "CON-ID=(?<CON-ID>[a-zA-Z0-9].+)" | table CON-ID | format]

See http://docs.splunk.com/Documentation/Splunk/6.2.2/SearchReference/Format for further information. It might help you understand what's happening if you look at the result from the subsearch alone.
Notice that in order for your subsearch to work like this, the list of key=value that format returns has to use the same key as your data, i.e. "CON-ID". If your subsearch produces something like "FoundConID", then you can't directly correlate that to your data if there is no "FoundConID" in it.

Explorer

Jeffland,

Thanks for your quick reply, I think you helped lay the ground work for Runals a bit. Your suggestion didn't do exactly what I was hoping, but was still a great help, and reference. 🙂

0 Karma