I am seeing an issue on our Splunk server where we seem to be hitting a performance bottleneck. When generating charts from searches it takes a long time to render the results. This is especially noticable on dashboards using where multiple graphs may be rendered. As our server is limited in resources we have restricted the concurrent searches to 5 (we patient enough to wait for the results) there it appears that the search itself completes relatively quickly and the jobs report indicates that 98% of these searches take between 0 and 3 seconds to complete. When loading a dashboard it can be seen that multiple splunkd processes are spawned, which I assume are carrying the concurrent searches. These splunkd processes disappear at times which would seem to correspond to the completion times shown in the job report. Around the same time the main splunkd process will rise up to as high at 300% and generate load average of up to 16. This increased CPU usage and corresponding load average remain high until all the the panels within the dashboard have been rendered and this rendering can take a long time. I would be prepared to accept this is delay is reasonable when the search is a long running and retrieves a lot of data, but this lag seems excessive for searches that only take between 0 and 3 seconds. So, it would appear that we are hiiting a bottleneck at the point at which the panels or data used to generate the panels are rendered, maybe queuing or backing up a process which is single threaded. Has anyone else observed this behaviour and is there any way to improve this performance other than increasing the servers CPU? Server specifications are: CPU: 2x3GHz (hyper threaded processors) cache size : 512 KB memory: 5GB Ram Storage: 1TB SAN Example of the diffrence between normal and higher load: Normal load =========== top - 20:02:26 up 90 days, 10:25, 12 users, load average: 0.50, 1.04, 0.94 Tasks: 208 total, 1 running, 207 sleeping, 0 stopped, 0 zombie Cpu(s): 7.5% us, 2.2% sy, 0.0% ni, 90.1% id, 0.2% wa, 0.0% hi, 0.0% si Mem: 4933824k total, 4896456k used, 37368k free, 27384k buffers Swap: 4192956k total, 10372k used, 4182584k free, 3867576k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17878 syslogng 15 0 8776 7360 940 S 27 0.1 3508:47 syslog-ng 31911 root 16 0 698m 609m 9.9m S 10 12.7 260:36.48 splunkd High load ========= top - 19:55:07 up 90 days, 10:17, 12 users, load average: 4.36, 1.92, 0.97 Tasks: 217 total, 1 running, 216 sleeping, 0 stopped, 0 zombie Cpu(s): 66.7% us, 7.2% sy, 0.0% ni, 13.4% id, 12.5% wa, 0.1% hi, 0.0% si Mem: 4933824k total, 4900944k used, 32880k free, 23892k buffers Swap: 4192956k total, 10372k used, 4182584k free, 3842728k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24262 root 19 0 40900 15m 8196 S 127 0.3 0:21.68 splunkd 24131 root 18 0 56096 25m 8176 S 73 0.5 0:13.31 splunkd 31911 root 15 0 698m 609m 9.9m S 58 12.7 259:23.29 splunkd 17878 syslogng 15 0 8776 7360 940 S 27 0.1 3507:15 syslog-ng ... View more

Is there way to add a time picker in a form which allows the user to select a start and end time and date, without having to go through the drop down list and choose custom? ... View more

I have a question regarding the population of dropdowns via saved searches. The examples in the Splunk documentation show a search similar to the following: <populatingSearch fieldForValue="suser" fieldForLabel="suser"><!CDATA[sourcetype=p4change | rex "user=(?<suser>\w+)@" | stats count by suser]]></populatingSearch> However, I am slightly confused (maybe because the search in the examples is quite complex) on how this is carried out in practice. I wish to populate the dropdown with the contents of the partner field which has up to 200 different values within the indexed data. The string being as follows: 2010/12/13@13:31:22,billstats,partner=XXXX,cde=XX,usd=XX How would I write the example population string in a way which can parse my indexed data in a way that could populate the dropdown? ... View more

Is there anyway of increasing the row limit in the dashboard? I need to be able to create 50 rows. Thanks ... View more

Hi, I have created a dashboard which is too intensive with it's searches. Each search is effectively reading fields form the same line. In order to alleviate this loading I have created form with a search template and related postprocessing searches, which works well and reduces the load time of the dashboard. However this creates some strange behaviour in that the <earliestTime> tag seems to be ignored and graphs only show the last 6 hours. Also is it possible to use postprocessing searches (preferably non-dynamic) within a dashboard? Replacing the <form> tags with <dashboard> results in blank graphs, which I assume is caused by the lack of user input in the <fieldset> component. Excerpt from the form is as follows: <form> <label>order dashboard</label> <fieldset autoRun="true"> <input type="time"> <default>Last 30 days</default> <seed>Last 30 days</seed> </input> </fieldset> <searchTemplate>index="c3" source="*submitted_order_count.log" OR source="*failed_order_count.log" | fields source, _time, orderType, paymentType, count</searchTemplate> <row> <single> <searchPostProcess>search source="*submitted_order_count.log" AND (paymentType="Postpay" OR paymentType="Prepay") | head 2 | stats sum(count) as total | rangemap field=total low=0-199 severe=400-4000 elevated=200-399</searchPostProcess> <title>Current number of 'Submitted' orders</title> <earliestTime>-30m</earliestTime> <option name="afterLabel">'Submitted' orders</option> <option name="classField">range</option> <option name="field">total</option> </single> </row> <row> <chart> <title>Submitted orders by type</title> <searchPostProcess>search source="*submitted_order_count.log" orderType!="" | timechart avg(count) by orderType limit=0</searchPostProcess> <option name="charting.chart">area</option> <earliestTime>-28d</earliestTime> <option name="Height">400px</option> <!--<option name="charting.legend.placement">bottom</option>--> <option name="charting.legend.labelStyle.maximumWidth">75</option> <option name="charting.primaryAxisTitle.text">time</option> <option name="charting.secondaryAxisTitle.text">Number of orders in 'Submitted' state</option> <option name="charting.chart.stackMode">stacked</option> </chart> </row> </form> ... View more

I have the following search which displays amounts of records by month (X-axis). index="billing" suspededrecords | eval monthnum = if(month="JAN",1,if(month="FEB",2,if(month="MAR",3,if(month="APR",4,if(month="MAY",5,if(month="JUN",6,if(month="JUL",7,if(month="AUG",8,if(month="SEP",9,if(month="OCT",10,if(month="NOV",11,if(month="DEC",0,0)))))))))))) |sort num(monthnum) | chart max(monthsuspense) as "Suspense" by month However the X-axis is always sorted month name alaphabetically which is confusing. I put in the the above eval and sort statements to try and create fields that would allow the months to be displayed in the correct order, but it doesn't work. I'm starting to think that I starting to complicate the issue and wondered if there is an easy way to achieve a correct listing of months. ... View more

I would like to see some clear rention times for the results from saved queries. Looking through the manual (http://www.splunk.com/base/Documentation/4.1.4/User/SchedulingSavedSearches) I have found this is possible via the "Retention" Field. I am unsure how this is field is actually used? Does anyone have an example of how this is used within a scheduled saved search? ... View more

Hi, I would like to rewrite bogus field values that are negative to 0. For example I would like to run the following search and plot a timechart of all 3 values, but change any that are negative to 0. timechart max(mms_run_time_hour) as "MMS Run Time" max(vfn_run_time_hour) as "VFN Run Time" max(sms_run_time_hour) Thanks ... View more

Hi, With the following code in a dashboard I can get the background of a single value to change depending on the range of a field. However the value within the single value box also changes to the value of the ranges i.e low, severe, good. How can I keep the single value as the original numeric value and still retain the color coding? <searchString>midas tablespace| table DATA_STAGE_percentage_used | rangemap field=DATA_STAGE_percentage_used low=0-89 severe=95-100 elevated=90-94</searchString> <title>Tablespace DATA_STAGE Percentage Used</title> <option name="classField">range</option> <earliestTime>-1h</earliestTime> Thanks ... View more