Hello,
Our goal is to define some alerts based on some custom searches from our indexed data. We wrote the search query and we are able to validate it against data - it matches correctly when using preset time intervals and we are able to see, for instance, "14 events matched" in the summary and also the contents of those events.
When we move to All time (real-time) and inject events into the system, we are able to see something like "0 of 2 events matched" in the summary (for 2 events that we injected), however there is nothing displayed in the search results (and as a consequence the alert that uses this search isn't triggered). When we search again with a preset time interval (e.g. Last 15 minutes) and with the same query as before we are able to see those newly injected events.
Our search query normally looks something like this:
```
index="snmp_data" SNMPv2-SMI::enterprises.3317.1.2.2.0.1
| rex field=_raw "UDP:\s\[(?<source_ip_address>[^\]]+)\].*SNMPv2-MIB::snmpTrapOID\.0\s=\sOID:\sSNMPv2-SMI::enterprises\.3317\.1\.2\.2\.0\.1.*RFC1269-MIB::bgpPeerState\.(?<peer_ip_address>[^\s]+)\s=\s[^:]+:\s"
| lookup dnslookup clientip as peer_ip_address outputnew clienthost as "peer_hostname"
| rex field="peer_hostname" mode=sed "s/.my.domain.net//"
| lookup dnslookup clientip as source_ip_address outputnew clienthost as "source_hostname"
| rex field="source_hostname" mode=sed "s/.my.domain.net//"
| eval time=strftime(_time, "%H:%M:%S %d-%m-%y")
```
But we've simplified it to just `index="snmp_data" SNMPv2-SMI::enterprises.3317.1.2.2.0.1`, in order to rule out regex parsing time and DNS lookup time, but we still get the same behavior.
What's more puzzling is that for other searches which look similar in complexity, the real-time alerting and searching works just fine...
How can we further troubleshoot what's going on?
Thanks
