Just started getting data flowing from a new machine that produces data which is similar in content, but different in structure to data that I've previously been working with for some time. I want to use values in the data produced by the new machine to filter a search through logs from the old machine, and I'm trying to do so as follows:
However, this query finds no events. message.callId (new machine) and CallID (Old Machine) are functionally identical and I am baffled as to why this does not work. I have no problems filtering index=index1 searches with index1 CallID's, and I have no problem filtering index=index2 searches with index2 message.callId's ie:
both correctly return results as I would expect. My current suspicion is that this has something to do with the sourcetypes/extractions. The new machine output is JSON, while the old machine output is it's own thing. I wrote the extractions for the old machine but I've relied on Splunk's JSON sourcetype to handle the new machine's data. I've not worked with JSON data before so it would not surprise me to learn there are some tricks necessary to get the subsearch output from index2 to jive with index1. Any help or ideas are appreciated.
After a nights sleep and a bit of review this morning, 3 turned out to be the correct troubleshooting step and revealed the solution. The values should have been common, but manually checking individual values revealed a bug in the underlying code causing the IDs I'm using as a filter to be reused from a previous interaction and so there were just no matches between the two datasets.