I'm trying to run the splunk CLI client using a remote server uri. The remote server is live and listening on port 8089, and the client has access to port 8089 on the server (verified using telnet). However, when I tried to run the splunk client using the -uri http://[my splunk server]:8089 option it immediately returns with the error message " Splunk is not running, and it must be for this operation. To start splunk, run "splunk start" ". I've also tried adding allowRemoteLogin = always to server.conf without success. ... View more

I'm running a search against about 1.2 million log records. Each record contains some geo tags and numeric values representing performance metrics. There are a total of about 45 key/values per record including the following: id: the service id type: the service type testId: the type of test (e.g. latency, throughput) region: the user's geographical region median: the median performance metric value ip: the user's IP address The search query I'm running calculates a 90th percentile median performance value grouped by service id within a specific geographical region, service type and test ID. Here is an example query: type="CDN" (testId="tl" OR testId="l") region="us" | eventstats perc90(median) as median90 | where median <= median90 | stats mean(median) as mean median(median) as median stdev(median) as stdev avg(stdDev) as avg_stdev count(median) as num_tests dc(ip) as num_ips by id | eval rel_stdev=100*(stdev/median) | table id, mean, median, avg_stdev, stdev, rel_stdev, num_tests, num_ips | sort median To my disappointment, this query is taking about 5 minutes to run completely on a fairly high end dedicated server (quad core X5570 2.93 GHz, 128GB memory, Raid 0 15K SAS + SSD cache) and much longer on the new hosted splunkstorm service. My question is if this level of performance should be expected for this amount of data and this type of search query. Are there any optimizations that could be made at index or search time in order to improve performance? Is there a significant hit on performance when applying | stats or | eventstats to a search? I've been using splunk for 5 days now... any help would be greatly appreciated. ... View more

I'm new to splunk. We pre-process http logs and assign geo tags so those tags can be included in the index. One of the geo tags we are adding is called 'region'. This tag describes the users' regions using different levels of granularity. For example, here is a sample log entry: asn=19262 location="MD, US" region=us region=us_east region=us_south region=us_south_southatlantic region=america_north The number of regions is arbitrary depending on the users location (e.g. US regions have more granularity than other countries). We need to be able to both search and group on region. For example: region="us" currently works, but region="us_east" does not because the index appears to only use the first value. However, we can do a fulltext search "region=us_east", but this has the undesirable effect of also picking up us_east_coast. We also want to be able to group by region using the stats function => stats median(rate) by region.... but this again only picks up the first instance of the region key. There are no parenthesis around the region value in the logs. I control the log input, so if it would be easier to accomplish this using delimited values like "region=us,us_east,us_south,us_south_southatlantic,america_north" I could change the format. I've read other posts suggesting use of MV_ADD or REPEAT_MATCH but haven't had much luck getting either to work. I would like the region to be added to the index, not applied at search time. ... View more