I have a search that finds failed jobs from my logs. Each of those failed jobs has a job number. I'd like to then take those job numbers and get all the log lines that contain one of those job numbers. Whats the best way to do this?
Assuming that something like this is your search to find the failed jobs:
index=myindex "Failed Job"
You can use it as a subsearch to provide a key (e.g. the Job Number) for another search. Then you can use transaction to group the events together by Job Number, if desired. So assuming the above, and assuming that your Job Number field in Splunk is something like jobNumber, try a search like the following:
index=myindex [search index=myindex "Failed Job" | fields jobNumber | dedup jobNumber] | transaction jobNumber
The sub search provides a list of values to your main search that are the equivalent of:
(jobNumber=000001 OR jobNumber=000002 OR jobNumber=000003 etc...)
Hope this helps!
Edit: Can you post your search commands? I'm wondering if the outer search is somehow excluding the results. Also, I can't add comments from my workplace for some reason, I can only edit my response, all my suggestions will be added here as edits.
Assuming that something like this is your search to find the failed jobs:
index=myindex "Failed Job"
You can use it as a subsearch to provide a key (e.g. the Job Number) for another search. Then you can use transaction to group the events together by Job Number, if desired. So assuming the above, and assuming that your Job Number field in Splunk is something like jobNumber, try a search like the following:
index=myindex [search index=myindex "Failed Job" | fields jobNumber | dedup jobNumber] | transaction jobNumber
The sub search provides a list of values to your main search that are the equivalent of:
(jobNumber=000001 OR jobNumber=000002 OR jobNumber=000003 etc...)
Hope this helps!
Edit: Can you post your search commands? I'm wondering if the outer search is somehow excluding the results. Also, I can't add comments from my workplace for some reason, I can only edit my response, all my suggestions will be added here as edits.
I got it working here's the modified search:
index=myindex | rex "(?i)^[^[][(?P
Here's my current search
index=myindex sourcetype=jobLogs [search index=myindex sourcetype=jobLogs "Job Failed" | rex "(?i)^[^[]*[(?P
I think I found the problem somewhere in the fact that I'm using rex to obtain the jobNubmer and its not using jobNumber=###### once the subsearch is evaluated.
The sub-search by itself produces the jobNumber's as expected but for some reason adding the outer search produces zero results.
You will figure it out.Please use this for reference:
http://docs.splunk.com/Documentation/Storm/Storm/User/Useasubsearch
That doesn't work I don't think you can have jobid = and then a bracket.
index=myindex source=jobs jobid=[|search index=myindex source=jobs "Failed Job"|fields jobid]
It won't let me do jobid=[
Error in 'search' command: Unable to parse the search: Comparator '=' has an invalid term on the right hand side.
Yes that means you will get the job id in your events, put the search in the subsearch index=myindex source=jobs jobid=[index=myindex source=jobs "Failed Job"|fields jobid]
I search: index=myindex source=jobs "Failed Job"
one of the fields that gets extracted is jobId but this obviously only gets me that one line with Failed Job not all the lines for that job.
According to your post "I have a search that finds failed jobs from my logs." what does that mean? how do you know that it's a failed job event, could you tell us?
I don't want to manually have to type the failed jobid I'd like to see it be dynamic off of the jobid's that were returned by looking for ones that have failed.
As you mentioned in the mail post your search gives you the failed job list.
Just write your search which gives you the all the events
your search jobid=[|search failed jobid]. this will only give you the failed job events.
I only want the failed job events how do I achieve that with what linu1988 suggested?
Yeah I am have problems getting the subsearch to only give the job numbers that failed. linu1988 has the best idea I can think of.
Append a subsearch to the search then sort by job number. I'll try and make an example.
Need more details then that...
Actually I would say that append might be better.
First extract the job number fields and then use a subsearch, filter out the failed job events.