Splunk Search

Subsearch filtering not working between sourcetypes

jcarlock
Explorer

Just started getting data flowing from a new machine that produces data which is similar in content, but different in structure to data that I've previously been working with for some time.  I want to use values in the data produced by the new machine to filter a search through logs from the old machine, and I'm trying to do so as follows:

 

index=index1 ... [ search index=index2 ... 
    | rename message.callId as CallID
    | fields CallID]

 

However, this query finds no events. message.callId (new machine) and CallID (Old Machine) are functionally identical and I am baffled as to why this does not work.  I have no problems filtering index=index1 searches with index1 CallID's, and I have no problem filtering index=index2 searches with index2 message.callId's ie:

 

index=index1 ...[ search index=index1 ...
    | fields CallID]

and

index=index2 ... [ search index=index2 ...
    | fields message.callId]

both correctly return results as I would expect.  My current suspicion is that this has something to do with the sourcetypes/extractions.   The new machine output is JSON, while the old machine output is it's own thing.  I wrote the extractions for the old machine but I've relied on Splunk's JSON sourcetype to handle the new  machine's data.  I've not worked with JSON data before so it would not surprise me to learn there are some tricks necessary to get the subsearch output from index2 to jive with index1.  Any help or ideas are appreciated.

 

Labels (2)
0 Karma
1 Solution

rupkumar4sec
Path Finder

Your search looks fine to me. It should work if both fields are being extracted. Some of the things I would check

  1. Check if values for both the fields are consistent (maybe change both fields to lower case)
  2. Check if there are any additional spaces at the end
  3. Confirm manually if there are any common values or not.

 

View solution in original post

rupkumar4sec
Path Finder

Your search looks fine to me. It should work if both fields are being extracted. Some of the things I would check

  1. Check if values for both the fields are consistent (maybe change both fields to lower case)
  2. Check if there are any additional spaces at the end
  3. Confirm manually if there are any common values or not.

 

jcarlock
Explorer

After a nights sleep and a bit of review this morning, 3 turned out to be the correct troubleshooting step and revealed the solution.  The values should have been common,  but manually checking individual values revealed a bug in the underlying code causing the IDs I'm using as a filter to be reused from a previous interaction and so there were just no matches between the two datasets. 

Thank you for the sanity check!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...