Splunk Search only works if Sourcetype is specified

[amtoyo]

So I have this very strange problem. We have 2 SearchHead environments. 1 SearchHead Cluster(7) and a Standalone Dev SearchHead. They both are connected to the same indexer Cluster.

A new problem appeared today. When users are searching for a key-value pair, Splunk stalls and does not bring up any results. Splunk does not terminate the search either, just stalls until I close the tab.

  See below:

However, the moment sourcetype is mentioned, splunk brings back searches:

The issue only appears on the firewall index which has several sourcetypes in there. This issue does not appear in other indexes that contain multiple sourcetypes. Just the index=firewalls

I was curious if the problem resided in the search head layer or indexers. So I ran the same vague search  ("index=firewalls src="*" ) on our standalone SH and indexers which both returned normal results. So the issue seems to reside on the specific Search Head Cluster.

Any idea?

[amtoyo]

I forgot to mention that this problem is only seen in Smart and Verbose Mode. Fast mode seems to pull the data fine with both searches from the original post

[diogofgm]

Have checked the job inspector for what's being added to the search?

[The_Simko]

Track down where the field is being set. It's likely a sourcetype specific field extraction or alias. 

[amtoyo]

How does Alias and renames affect data from being searched when sourcetype is not specified?  When i search with no specific sourcetype, it should pull everything, right?  This index collects data from 2 different firewalls ( Juniper and Palo Altos) . The Palo Altos undergo a sourcetype rename at ingestion to index properly.

But it shouldnt impact a vauge search as Splunk is supposed to pull everything from that index regardless of souretype. Any specific ideas I should look for aliases and renames?