I have a search that finds failed jobs from my logs. Each of those failed jobs has a job number. I'd like to then take those job numbers and get all the log lines that contain one of those job numbers. Whats the best way to do this?
Assuming that something like this is your search to find the failed jobs:
index=myindex "Failed Job"
You can use it as a subsearch to provide a key (e.g. the Job Number) for another search. Then you can use transaction to group the events together by Job Number, if desired. So assuming the above, and assuming that your Job Number field in Splunk is something like jobNumber, try a search like the following:
index=myindex [search index=myindex "Failed Job" | fields jobNumber | dedup jobNumber] | transaction jobNumber
The sub search provides a list of values to your main search that are the equivalent of:
(jobNumber=000001 OR jobNumber=000002 OR jobNumber=000003 etc...)
Hope this helps!
Edit: Can you post your search commands? I'm wondering if the outer search is somehow excluding the results. Also, I can't add comments from my workplace for some reason, I can only edit my response, all my suggestions will be added here as edits.
perhaps some sort of IF statement?
Thats correct
This means you have more than one log line with the job number for each failed job right?