As you mentioned: Field Discovery was set to "OFF" Field discovery is the process Splunk uses to extract fields aside from default fields such as host, source, and sourcetype. This means that Splunk only returns information on default fields and fields that are required to fulfill your search (if you are searching on certain fields, it will extract those fields). so you must include | fields httpTimeStamp, httpClientIp, httpMethod, httpUri, httpResponseCode, httpBytesDownloaded, httpDownloadTime or mention them in some other explicit way in order for Splunk to extract them when Field Discovery is off. If you were testing with just the sourcetype or with * you wouldn't have seen them. ... View more

It’s described in the documentation that the search peers don’t use the mounted directories for other purposes not directly related to search i.e. the geoip script. Important: The search peers use the mounted directories only when fulfilling the search head's search requests. For indexing and other purposes not directly related to distributed search, the search peers will use their own, local apps, users, and system directories, the same as any other indexer. So, I had to copy the maps application to the local apps folder in each Splunk indexer (search peer). Next I had to restart each Splunk indexer so that the new application gets installed and enabled. I didn’t create a symbolic link because we would have to do it for all shared bundles. Check the documentation here. http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Mounttheknowledgebundle ... View more

Here is the final result: "failed to route request" sourcetype="egw" | rex field=_raw "for \[(?<service>.?):(?<method>.?)\]" | timechart span=1d count by service and then "failed to route request" sourcetype="egw" | rex field=_raw "for \[(?<service>.?):(?<method>.?)\]" | timechart span=1d count by method I will try to combine both graphs. ... View more