Why do I see log entries monitoring splunkd.log via search that do not appear in splunkd.log?

[wrangler2x]

I was under the impression that if I did index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" realtime that it would be the same as doing a tail -f /opt/splunk/var/log/splunk/splunkd.log (in Linux). That seems to not be quite so.

I was explaining this to a co-worker and showed a tail -f while on another screen running the search in real-time. Yes, I saw everything showing in the tail -f in the search window, but in the search window I saw two other log entry types that were not showing on the other screen: INOFO HttpPubSubConnection and ERROR DiskMon. Here are a couple of samples (with IP redacted):

1. 07-12-2016 10:38:47.075 -0700 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_xxx.xxx.xxx.xxx_8089_oda1b.oit.uci.edu_oda1b_18CB2BD0-0207-47E0-B4E7-C62BAC751304
2. 07-12-2016 10:38:42.159 -0700 ERROR DiskMon - None such on disk: /opt/splunk/var/run/splunk/dispatch

And, if I grep for HttpPubSubConnection or DiskMon in /opt/splunk/var/log/splunk/splunkd.log I get nothing back. So where are these log entries coming from, and why do I not see exactly the same thing on both screens?

[ryanoconnor · Accepted solution]

How many Splunk systems are in your environment?

The following search will return events from any host that is monitoring the file /opt/splunk/var/log/splunk/splunkd.log

index=_internal source="/opt/splunk/var/log/splunk/splunkd.log"

If, for example, you have two different search heads, or a search head and an indexer, than your splunk search might be returning data from multiple hosts.

[wrangler2x]

Yes, that is what it was. And as @somesoni2 suggested, I added a host filter and the results now match. Thanks guys.

[somesoni2]

Were you running your query for the exact same host (host filter explicitly specified)?