The issue was that the source type was actually webbyLogs with a capital L. and in the props file I had it all in lower case (like every other sourcetype). thanks to MuS for all your pointers and help ... View more

Splunk support helped me with this issue. Their solution: putty into server, navigate to where the splunk data is and run find . -name Hosts.data |xargs grep --color='auto' -P -n "[\x80-\xFF]" this will return some of the junk host names you have e.g. in my case "host::#000#000#000#000#000#000#000#000#000#000#000#000#000#000#000#000#000#000#000#000#000#000#000#000!#000#000#000#000#000#000#000imudp#000#000#000?#013z#001#000#000#000#000 " I could search for this in the splunk GUI (host=#000*) but the data would never show, but the interesting fields would. Then try adding |delete to get rid of this data and if your lucky you fix your issues. I would recommend working with support, this was a nightmare that took a few weeks to get to the bottom of. ... View more

I changed the search to | `host_eventcount(30,2)` | search is_expected=true which gave me the field names of host_owner and host_bunit which i added happily into my alerts and they now work ... View more

Best approach will be have your forwarder configured to send in one format. If its not easy, then you can use dns lookup feature in splunk to convert all to one type, either hostname or IP, up to you. For reference see this: http://answers.splunk.com/answers/8051/dns-lookup-via-splunk ... View more