thanks, i too am of the same opinion, i will implement option 1 as a bug fix and scale the HF ... View more

I think (i might be wrong though) that you have a multi-tenant splunk environment, there are chances that some other team is using those ITSI searches. My suggestion would be to reach out to Splunk support, they will surely assist you. ... View more

check this page if it helps. If not, could you please share some sample data. Please ensure to mock/anonymize any sensitive information in your data/code before posting on Splunk community. ... View more

So your SHs are running real-time searches which are really resource intensive, as a quick fix try to convert them into scheduled searches, in "most" cases a real time search is never required. Regarding the Skipped concurrent searches you can follow the below steps: 1. Detect which searches are being skipped index=_internal earliest=-24h status=skipped sourcetype=scheduler 2. Which apps the Skipped Searches are coming from index=_internal earliest=-24h status=skipped sourcetype=scheduler | stats count by host app | sort - count 3. Identify the bad searches (change the timeframe and host as per your needs) index=_internal sourcetype=scheduler status!=queued earliest=-30d@d host=<server_name> | eval is_realtime=if(searchmatch("sid=rt* OR concurrency_category=real-time_scheduled"),"yes","no") | fields run_time is_realtime savedsearch_name user | stats avg(run_time) as average_run_time max(run_time) as max_run_time min(run_time) as min_run_time max(is_realtime) as is_realtime by savedsearch_name user | eval average_run_time = average_run_time/60| eval min_run_time = min_run_time/60 | eval max_run_time = max_run_time/60 | sort - average_run_time | join savedsearch_name [|rest /servicesNS/-/-/saved/searches splunk_server=<server_name> | search is_scheduled=1 | rename title AS savedsearch_name | rename eai:acl.app as title| table splunk_server title savedsearch_name cron_schedule search ] | rename title as "App Name" | fields splunk_server savedsearch_name user average_run_time max_run_time min_run_time cron_schedule is_realtime search "App Name" | sort splunk_server, -average_run_time   4. Modify scheduler limits In case the above did not remediate your situation the last option is to increase limits on a system by modifying limits.conf (make sure you do it on a custom app and not the one in $Splunk_HOME$/etc/default) Out of the box with a Splunk 16 core system, Splunk can run 22 searches at any one time. That is calculated using the following formula: max_hist_searches = max_searches_per_cpu ( default of 1) x number_of_cpus (16) + base_max_searches (default of 6). Of those 22 searches, the scheduler is allocated 50 percent of that number by default (so 11 searches) according to the setting max_searches_perc. Of those 11 searches that the scheduler can run, the auto summarization (things like Data Models) are allowed 50% of the number of scheduled searches. Taking 50% of the number of searches that the scheduler can run, we end up with about 6 searches that can be run at a time for your Data Models according to the setting for auto_summary_perc. This all being said, if your data models are taking an extremely long time to run, or not completing and consistently skipping, increasing your base max searches from 6 to 12, will only get you to 7 auto summary searches at one time. You may want to alter auto_summary_perc in order to allow more searches to be dedicated to your Data Models. If you have the CPU resources, you may want to up the max_searches_per_cpu to something like 2. This would allow you to run 10 Data Model summarizations at any one time and give an overall number of 38 maximum historical searches. The above info is derived from here ... View more

I am guessing you have used the Color settings with mode "scale" set to the Presets - divergent  something like this: so this applies the settings dynamically as you move to the later pages - page 1 vs page 13 in your screenshots. So this color scale maps the colors as  per the values in each page. What you could do is: option 1. Set some static ranges and associated color for them like below option 2.  you could define a "custom" scale (as you did already) but define some sort of static range by specifying min, midpoint and max values for the logcount number   ... View more

Thanks @richgalloway, i could not find any issues with any search in particular (yes there were users with badly written searches but that should not impact so much)  as a test i disabled the realtime metadata search that populates the search summary page (disabled it globally so that no apps have that search running) and looks like it solved the issue. ... View more

Hi @zooky92 try something like the below query. index=myindex host=myhost source="*access_log" duration!="" NOT "status" earliest=-24h latest=-1h (date_hour > 00 AND date_hour < 14) |eval apacheDuration_today=apacheDuration/1000 |stats avg(apacheDuration_today) as avg_apacheDuration_today by host | appendcols [search index=myindex host=myhost source="*access_log" duration!="" NOT "status" earliest=-8d latest=-1d (date_hour > 00 AND date_hour < 14) |eval apacheDuration_week=apacheDuration/1000 | stats avg(apacheDuration_week) avg_apacheDuration_week by host] | eval is_spike=if(avg_apacheDuration_today >1.2 * avg_apacheDuration_week, avarageToday, 0) | stats values(spike_value) as spike_value values(apacheDuration_today) as apacheDuration_today values(avg_apacheDuration_week) as avg_apacheDuration_week values(is_spike) by _time you can try to run it as a scheduled search at 15:00. ... View more

As far as I know you need to have purchased a valid license for the data stream processor before you can use splunk DSP. Take a look here ... View more

On your question about the performance impact, you might end up running into OOM issues and your search performance might also get degraded, as suggested by @to4kawa there might be alternatives available, if you can post your query and some mockup data and the desired output we can try to help you out ... View more

Can you check web.conf in /etc/system/default and check for the port that is set for splunkweb see attribute "httpport". So if it is set to port 8002 you use http://localhost:8002 ... View more

when you say you have configured re-balancing, did you configure your UF outputs.conf with target groups? DO you use indexer discovery configured? ... View more

You can create a DELIMS based extraction (transforms.conf) to extract the subfields: [your_transform_rule] SOURCE_KEY = _raw DELIMS = "|" FIELDS = Extracted_time, Error, ..., General_Agency Then, you'd call that rule from the props.conf of your sourcetype, like this: [your_sourcetype] REPORT-extracted_fields = your_transform_rule ... View more

it is quite difficult to tell the exact message that splunk will throw when an indexer goes down since it might go down for a variety of factors (maybe the disk/memory/cpu utilization had spiked), but you should be able to figure it out from the splunkd logs just look into the error logs (index=_internal source=*splunkd.log log_level=ERROR host=). ... View more