hi Thanks for the replay, i am not sure how to check the size of this, is it below? [tcpout:my_LB_indexers] server=hp737srv:9997 maxQueueSize=500MB Cheers Rob ... View more

Hi I am using LINUX - Red Hat. We have one application folder, but we have multiple sourcetypes all over the application. so i had to set up multiple lines to take in multiple sourcetypes. [monitor:///dell873srv/apps/UBS_QCST_SEC3/logs.../.log] disabled = false host = UBS-RC_QCST_MASTER index = mlc_live sourcetype = sun_jvm crcSalt = whitelist = .*gc..log$ blacklist=logs_|fixing_|tps-archives [monitor:///dell873srv/apps/UBS_QCST_SEC3.../.tps] disabled = false host = UBS-RC_QCST_MASTER index = mlc_live sourcetype = tps crcSalt = whitelist = ..tps$ blacklist=logs_|fixing_|tps-archives [monitor:///dell873srv/apps/UBS_QCST_SEC3/logs/monitoring/vmstat/.log] disabled = false host = UBS-RC_QCST_MASTER index = mlc_live sourcetype = vmstat-linux crcSalt = whitelist = vmstat..log$ blacklist=logs_|fixing_|tps-archives [monitor:///dell873srv/apps/UBS_QCST_SEC3/logs/monitoring/nicstat/] disabled = false host = UBS-RC_QCST_MASTER index = mlc_live sourcetype = nicstat crcSalt = whitelist = nicstat..log$ blacklist=logs_|fixing_|tps-archives [monitor:///dell873srv/apps/UBS_QCST_SEC3.../.log] disabled = false host = UBS-RC_QCST_MASTER index = mlc_live whitelist=mxtiming..log$ blacklist=logs_|fixing_|tps-archives|mxtiming_crv_nr.* crcSalt = sourcetype = MX_TIMING2 [monitor:///dell873srv/apps/UBS_QCST_SEC3.../service.log] disabled = false host = UBS-RC_QCST_MASTER index = mlc_live whitelist = (?\d)-\d*-service.log blacklist=logs_|fixing_|tps-archives crcSalt = sourcetype = service [monitor:///dell873srv/apps/UBS_QCST_SEC3/.log] disabled = false host = UBS-RC_QCST_MASTER index = mlc_live whitelist=mxtiming_crv_nr..log$ blacklist=logs_|fixing_|tps-archives crcSalt = sourcetype = MX_TIMING_RATE_CURVE [monitor:///dell873srv/apps/UBS_QCST_SEC3/logs/traces/.log] disabled = false host = UBS-RC_QCST_MASTER index = mlc_live whitelist=mxtiming_crv_nr..log$ blacklist=logs_|fixing_|tps-archives crcSalt = sourcetype = MX_TIMING_RATE_CURVE ... View more

Hi In The end was a version issues, in the later version you can overwrite a field "context" in CALCULATED, but in the earler version you cant. So i had to change the name of "context" to "context2" in the CALCULATED and it worked - Thanks For your help on this Robert ... View more

Hi ![alt text][1]I cant import on the old version, but i can on Splunk 7.0.2. Without issue. So now i need a workaround, as i need to get this into the app. C:\Users\rlynch\Desktop\Quick_Work\Greenhorn\2018-05-15+15_53_13-Edit+Objects_+MXTIMING1_RATE_CURVE_CLONE_TOBE_DELETED+_+Splunk+7.1.0.png ... View more

Hi I do a check for THP and to me it's off. Why do you think it is on? I check all the process and it returns nothing. grep -e AnonHugePages /proc/*/smaps | awk '{ if($2>4) print $0} ' | awk -F "/" '{print $0; system("ps -fp " $3)} ' | grep splunk ... View more

Cheers - This is perfect. ... View more

Hi THP is off. grep -e AnonHugePages /proc/*/smaps | awk '{ if($2>4) print $0} ' | awk -F "/" '{print $0; system("ps -fp " $3)} ' | grep splunk However when i run (ALL Time) index=_internal ulimit i don't get any answers back below, so i am not sure what i am looking at? 09/05/2018 13:51:41.284 05-09-2018 13:51:41.284 +0200 INFO ulimit - Linux transparent hugetables support, enabled="always" defrag="always" host = dell425srv index = _internal linecount = 1 source = /dell845srv/apps/splunk_forwarder/splunkforwarder_NT_PAC/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = dell425srv 2 09/05/2018 13:51:41.284 05-09-2018 13:51:41.284 +0200 INFO ulimit - Limit: cpu time: unlimited host = dell425srv index = _internal linecount = 1 source = /dell845srv/apps/splunk_forwarder/splunkforwarder_NT_PAC/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = dell425srv 3 09/05/2018 13:51:41.284 05-09-2018 13:51:41.284 +0200 INFO ulimit - Limit: user processes: 790527 processes host = dell425srv index = _internal linecount = 1 source = /dell845srv/apps/splunk_forwarder/splunkforwarder_NT_PAC/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = dell425srv 4 09/05/2018 13:51:41.284 05-09-2018 13:51:41.284 +0200 INFO ulimit - Limit: open files: 65536 files host = dell425srv index = _internal linecount = 1 source = /dell845srv/apps/splunk_forwarder/splunkforwarder_NT_PAC/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = dell425srv 5 09/05/2018 13:51:41.284 05-09-2018 13:51:41.284 +0200 INFO ulimit - Limit: data file size: unlimited host = dell425srv index = _internal linecount = 1 source = /dell845srv/apps/splunk_forwarder/splunkforwarder_NT_PAC/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = dell425srv 6 09/05/2018 13:51:41.284 05-09-2018 13:51:41.284 +0200 INFO ulimit - Limit: core file size: unlimited host = dell425srv index = _internal linecount = 1 source = /dell845srv/apps/splunk_forwarder/splunkforwarder_NT_PAC/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = dell425srv 7 09/05/2018 13:51:41.284 05-09-2018 13:51:41.284 +0200 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited] host = dell425srv index = _internal linecount = 1 source = /dell845srv/apps/splunk_forwarder/splunkforwarder_NT_PAC/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = dell425srv 8 09/05/2018 13:51:41.284 05-09-2018 13:51:41.284 +0200 INFO ulimit - Limit: resident memory size: unlimited host = dell425srv index = _internal linecount = 1 source = /dell845srv/apps/splunk_forwarder/splunkforwarder_NT_PAC/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = dell425srv 9 09/05/2018 13:51:41.284 05-09-2018 13:51:41.284 +0200 INFO ulimit - Limit: data segment size: unlimited host = dell425srv index = _internal linecount = 1 source = /dell845srv/apps/splunk_forwarder/splunkforwarder_NT_PAC/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = dell425srv 10 09/05/2018 13:51:41.284 05-09-2018 13:51:41.284 +0200 INFO ulimit - Limit: virtual address space size: unlimited host = dell425srv index = _internal linecount = 1 source = /dell845srv/apps/splunk_forwarder/splunkforwarder_NT_PAC/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = dell425srv ... View more

So, how do i check if i am running out of sockets? ... View more

Hi Thanks for the answer. As i have one indexer and one search head is this setting usable for me. In fact this is what i am asking, should i increase my no of indexers and search heads to 2 http://docs.splunk.com/Documentation/Splunk/7.1.0/Capacity/Summaryofperformancerecommendations If i am reading this doc i think i should go to 2 indexers and 1 search head. I have up to 100GB a day and 20 users Thanks Robert ... View more

Hi Woodcock (Thanks for your help) I am not sure what cim_DM_index is - how do i set this? I have also added in the setting from http://docs.splunk.com/Documentation/Splunk/latest/Capacity/Parallelization However i did not really see any improvement. server.conf [general] parallelIngestionPipeline = 2 limits.conf [search] batch_search_max_pipeline = 4 batch_search_max_results_aggregator_queue_size = 300 batch_search_max_serialized_results_queue_size = 300 In datamodels.conf [MXTIMING_V8_5_Seconds] acceleration = 1 acceleration.earliest_time = -3mon acceleration.hunk.dfs_block_size = 0 acceleration.manual_rebuilds = 0 acceleration.poll_buckets_until_maxtime = 0 acceleration.max_concurrent = 3 Example of the tstats | tstats summariesonly=true max(MXTIMING.Elapsed) AS Elapsed max(MXTIMING.CPU) AS CPU max(MXTIMING.CPU_PER) AS CPU_PER values(MXTIMING.RDB_COM1) AS RDB_COM values(MXTIMING.RDB_COM_PER1) AS RDB_COM_PER max(MXTIMING.Memory) AS Memory max(MXTIMING.Elapsed_C) AS Elapsed_C values(source) AS source_MXTIMING avg(MXTIMING.Elapsed) AS average, count(MXTIMING.Elapsed) AS count, stdev(MXTIMING.Elapsed) AS stdev, median(MXTIMING.Elapsed) AS median, exactperc95(MXTIMING.Elapsed) AS perc95, exactperc99.5(MXTIMING.Elapsed) AS perc99.5, min(MXTIMING.Elapsed) AS min,earliest(_time) as start, latest(_time) as stop FROM datamodel=MXTIMING_V8_5_Seconds WHERE host=QCST_RSAT_40 AND MXTIMING.Elapsed > 5 GROUPBY _time MXTIMING.Machine_Name MXTIMING.Context+Command MXTIMING.NPID MXTIMING.Date MXTIMING.Time MXTIMING.MXTIMING_TYPE_DM source MXTIMING.UserName2 MXTIMING.source_path MXTIMING.Command3 MXTIMING.Context3 span=1s | rename MXTIMING.Context+Command as Context+Command | rename MXTIMING.NPID as NPID | rename MXTIMING.MXTIMING_TYPE_DM as TYPE | rename MXTIMING.Date as Date | rename MXTIMING.Time as Time | rename MXTIMING.Machine_Name as Machine_Name | rename MXTIMING.UserName2 as UserName | rename MXTIMING.source_path as source_path | eval Date=strftime(strptime(Date,"%Y%m%d"),"%d/%m/%Y") | eval Time = Date." ".Time | eval FULL_EVENT=Elapsed_C | eval FULL_EVENT=replace(FULL_EVENT,"\d+.\d+","FULL_EVENT") | join Machine_Name NPID type=left [| tstats summariesonly=true count(SERVICE.NPID) AS count2 values(source) AS source_SERVICES FROM datamodel=SERVICE_V5 WHERE ( host=QCST_RSAT_40 earliest=1525770000 latest=1525859918) AND SERVICE.NICKNAME IN () GROUPBY SERVICE.Machine_Name SERVICE.NICKNAME SERVICE.NPID | rename SERVICE.NPID AS NPID | rename SERVICE.NICKNAME AS NICKNAME | rename SERVICE.Machine_Name as Machine_Name | table NICKNAME NPID source_SERVICES Machine_Name ] | lookup MXTIMING_lookup_Base Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | appendpipe [ | where isnull(Threshold) | rename TYPE AS BACKUP_TYPE | eval TYPE="" | lookup MXTIMING_lookup_Base Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | rename BACKUP_TYPE AS TYPE] | dedup Time, NPID,Context+Command | where Elapsed > Threshold OR isnull('Threshold') | fillnull Tags | eval Tags=if(Tags=0,"PLEASE_ADD_TAG",Tags) | makemv Tags delim="," | eval Tags=split(Tags,",") | search Tags IN (*) | eval source_SERVICES_count=mvcount(split(source_SERVICES, " ")) | eval NICKNAME=if(source_SERVICES_count > 1, "MULTIPLE_OPTIONS_FOUND",NICKNAME) | search ... View more

Ok - am seeing "waiting on available sockets" When i click on "inspect" - this is the first time i have spotted this.. ... View more

no transformation command | tstats summariesonly=true max(MXTIMING.Elapsed) AS Elapsed max(MXTIMING.CPU) AS CPU max(MXTIMING.CPU_PER) AS CPU_PER values(MXTIMING.RDB_COM1) AS RDB_COM values(MXTIMING.RDB_COM_PER1) AS RDB_COM_PER max(MXTIMING.Memory) AS Memory max(MXTIMING.Elapsed_C) AS Elapsed_C values(source) AS source_MXTIMING avg(MXTIMING.Elapsed) AS average, count(MXTIMING.Elapsed) AS count, stdev(MXTIMING.Elapsed) AS stdev, median(MXTIMING.Elapsed) AS median, exactperc95(MXTIMING.Elapsed) AS perc95, exactperc99.5(MXTIMING.Elapsed) AS perc99.5, min(MXTIMING.Elapsed) AS min,earliest(_time) as start, latest(_time) as stop FROM datamodel=MXTIMING_V8_5_Seconds WHERE host=QCST_RSAT_40 AND MXTIMING.Elapsed > 5 GROUPBY _time MXTIMING.Machine_Name MXTIMING.Context+Command MXTIMING.NPID MXTIMING.Date MXTIMING.Time MXTIMING.MXTIMING_TYPE_DM source MXTIMING.UserName2 MXTIMING.source_path MXTIMING.Command3 MXTIMING.Context3 span=1s | rename MXTIMING.Context+Command as Context+Command | rename MXTIMING.NPID as NPID | rename MXTIMING.MXTIMING_TYPE_DM as TYPE | rename MXTIMING.Date as Date | rename MXTIMING.Time as Time | rename MXTIMING.Machine_Name as Machine_Name | rename MXTIMING.UserName2 as UserName | rename MXTIMING.source_path as source_path | eval Date=strftime(strptime(Date,"%Y%m%d"),"%d/%m/%Y") | eval Time = Date." ".Time | eval FULL_EVENT=Elapsed_C | eval FULL_EVENT=replace(FULL_EVENT,"\d+.\d+","FULL_EVENT") | join Machine_Name NPID type=left [| tstats summariesonly=true count(SERVICE.NPID) AS count2 values(source) AS source_SERVICES FROM datamodel=SERVICE_V5 WHERE ( host=QCST_RSAT_40 earliest=1525269600 latest=1525357584) AND SERVICE.NICKNAME IN () GROUPBY SERVICE.Machine_Name SERVICE.NICKNAME SERVICE.NPID | rename SERVICE.NPID AS NPID | rename SERVICE.NICKNAME AS NICKNAME | rename SERVICE.Machine_Name as Machine_Name | table NICKNAME NPID source_SERVICES Machine_Name ] | lookup MXTIMING_lookup_Base Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | appendpipe [| where isnull(Threshold) | rename TYPE AS BACKUP_TYPE | eval TYPE="" | lookup MXTIMING_lookup_Base Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | rename BACKUP_TYPE AS TYPE] | dedup Time, NPID,Context+Command | where Elapsed > Threshold OR isnull('Threshold') | fillnull Tags | eval Tags=if(Tags=0,"PLEASE_ADD_TAG",Tags) | makemv Tags delim="," | eval Tags=split(Tags,",") | search Tags IN (*) | eval source_SERVICES_count=mvcount(split(source_SERVICES, " ")) | eval NICKNAME=if(source_SERVICES_count > 1, "MULTIPLE_OPTIONS_FOUND",NICKNAME) ... View more

vmstat and top procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 19 0 0 907140 4304 375184512 0 0 28 104 0 3 4 1 95 0 0 15 0 0 3216580 4304 374960896 0 0 336 462 518492 24231 22 6 72 0 0 21 0 0 3744904 4304 374987200 0 0 0 29153 436836 30985 22 7 71 0 0 14 0 0 3243016 4304 375016384 0 0 0 17118 450371 29062 23 7 69 0 0 17 0 0 3439856 4304 375056768 0 0 0 427 517405 22007 24 6 70 0 0 15 0 0 3589468 4304 375072640 0 0 0 1417 499387 40596 24 5 71 0 0 15 0 0 3413696 4304 375063040 0 0 0 11410 473863 25533 23 5 72 0 0 25 0 0 3953944 4304 375152448 0 0 0 21670 492277 35373 23 6 70 0 0 13 0 0 4148080 4304 375221440 0 0 0 16422 373584 40051 20 5 75 0 0 6 0 0 4738224 4304 375207680 0 0 0 66 52522 22534 11 3 86 0 0 11 1 0 4355948 4304 375392992 0 0 0 78 54571 23997 10 4 86 0 0 10 0 0 4507112 4304 375828352 0 0 0 217 51837 17508 10 4 86 0 0 8 0 0 3708840 4304 375964480 0 0 0 46526 60016 17598 11 4 85 0 0 7 0 0 3552936 4304 376003744 0 0 0 87206 40021 10533 9 3 87 0 0 12 0 0 1958064 4304 376103168 0 0 0 806 56752 16889 23 6 71 0 0 10 0 0 2568120 4304 376076704 0 0 0 1843 60440 16156 15 4 81 0 0 9 0 0 2318016 4304 376160064 0 0 0 2604 51968 15052 14 3 82 0 0 8 0 0 2563528 4304 376129504 0 0 0 282 40177 15186 12 3 85 0 0 10 0 0 3183756 4304 376142976 0 0 0 37471 55299 14889 11 4 85 0 0 top - 15:47:42 up 2 days, 7:39, 4 users, load average: 14.84, 12.97, 10.54 Tasks: 658 total, 2 running, 656 sleeping, 0 stopped, 0 zombie %Cpu(s): 15.5 us, 6.5 sy, 0.0 ni, 77.9 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem : 39595564+total, 977992 free, 54651488 used, 34032617+buff/cache KiB Swap: 67108860 total, 67108860 free, 0 used. 33537020+avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5254 autoeng+ 20 0 47.470g 0.036t 24340 S 680.1 9.7 777:05.55 splunkd 2274 autoeng+ 20 0 5913624 985.0m 12344 S 199.7 0.3 34:07.72 splunkd 661 autoeng+ 20 0 5807112 1.508g 12336 S 99.7 0.4 5:55.52 splunkd 40803 autoeng+ 20 0 5688312 1.542g 12412 S 99.7 0.4 7:18.46 splunkd 304 root 20 0 0 0 0 S 21.2 0.0 0:04.37 kswapd1 303 root 20 0 0 0 0 S 20.5 0.0 0:04.52 kswapd0 42315 autoeng+ 20 0 92588 41836 15156 R 8.3 0.0 0:00.25 splunkd 10272 autoeng+ 20 0 6034912 5.297g 12348 S 6.6 1.4 329:38.43 splunkd 19849 autoeng+ 20 0 2262496 1.902g 12376 S 6.3 0.5 274:43.51 splunkd 19818 autoeng+ 20 0 394720 152636 12348 S 4.3 0.0 138:57.36 splunkd 42264 autoeng+ 20 0 813372 19864 7740 S 4.3 0.0 0:00.13 java 41966 root 0 -20 0 0 0 S 2.0 0.0 1:49.34 kworker/6:0H 42261 autoeng+ 20 0 107900 13508 5096 S 1.7 0.0 0:00.05 python 19845 autoeng+ 20 0 292320 48484 12364 S 0.7 0.0 1:37.45 splunkd 25248 root 0 -20 0 0 0 S 0.7 0.0 2:06.24 kworker/9:2H 25469 root 20 0 155552 95620 95288 S 0.7 0.0 16:09.59 systemd-journal 39096 root 0 -20 0 0 0 S 0.7 0.0 0:03.88 kworker/10:2H 42073 autoeng+ 20 0 52772 2724 1440 R 0.7 0.0 0:00.14 top ... View more

Hi Woodcock - Hope you are well 🙂 The behavior can change. Sometime i cant log into any new screen when the ~2 minute big search is running - Chrome will just keep going around and around. Sometime i can use screens that are open, however they don't work that well. Then other times, it seems to work fine..ish...but slower.... Its a funny one. ... View more

HI I cant get the BOX over 20% ? It a big big box... but i am looking to push it. I think i might have to add indexers and SH - I have not done it before, but i will try ... View more

Hi 2 CPU 14 cores - With Turbo threading on : Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz- SO 56 Cores (). RAM size 384GB 6TB SSD Red Hat When you say increase you mean add on more - at the moment i have one of each. I think i need to add on more. ... View more

Hi The server is a total Beast. 2 CPU 14 cores - With Turbo threading on : Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz- SO 56 Cores (). RAM size 384GB 6TB SSD Red Hat At the moment i cant get it to go over 20 % CPU this is another issues i am looking to improve. I have one search head and one indexer - I am thinking i might need to increase. This is the base search... It's long and when i run it over 200 Million lines of data it can take 2 minutes to complete - this is when most other things freeze. | tstats summariesonly=true max(MXTIMING.Elapsed) AS Elapsed max(MXTIMING.CPU) AS CPU max(MXTIMING.CPU_PER) AS CPU_PER values(MXTIMING.RDB_COM1) AS RDB_COM values(MXTIMING.RDB_COM_PER1) AS RDB_COM_PER max(MXTIMING.Memory) AS Memory max(MXTIMING.Elapsed_C) AS Elapsed_C values(source) AS source_MXTIMING avg(MXTIMING.Elapsed) AS average, count(MXTIMING.Elapsed) AS count, stdev(MXTIMING.Elapsed) AS stdev, median(MXTIMING.Elapsed) AS median, exactperc95(MXTIMING.Elapsed) AS perc95, exactperc99.5(MXTIMING.Elapsed) AS perc99.5, min(MXTIMING.Elapsed) AS min,earliest(_time) as start, latest(_time) as stop FROM datamodel=MXTIMING_V8_5_Seconds WHERE host=QCST_RSAT_40 AND MXTIMING.Elapsed > 5 GROUPBY _time MXTIMING.Machine_Name MXTIMING.Context+Command MXTIMING.NPID MXTIMING.Date MXTIMING.Time MXTIMING.MXTIMING_TYPE_DM source MXTIMING.UserName2 MXTIMING.source_path MXTIMING.Command3 MXTIMING.Context3 span=1s | rename MXTIMING.Context+Command as Context+Command | rename MXTIMING.NPID as NPID | rename MXTIMING.MXTIMING_TYPE_DM as TYPE | rename MXTIMING.Date as Date | rename MXTIMING.Time as Time | rename MXTIMING.Machine_Name as Machine_Name | rename MXTIMING.UserName2 as UserName | rename MXTIMING.source_path as source_path | eval Date=strftime(strptime(Date,"%Y%m%d"),"%d/%m/%Y") | eval Time = Date." ".Time | eval FULL_EVENT=Elapsed_C | eval FULL_EVENT=replace(FULL_EVENT,"\d+.\d+","FULL_EVENT") | join Machine_Name NPID type=left [| tstats summariesonly=true count(SERVICE.NPID) AS count2 values(source) AS source_SERVICES FROM datamodel=SERVICE_V5 WHERE ( host=QCST_RSAT_40 earliest=1525269600 latest=1525357584) AND SERVICE.NICKNAME IN () GROUPBY SERVICE.Machine_Name SERVICE.NICKNAME SERVICE.NPID | rename SERVICE.NPID AS NPID | rename SERVICE.NICKNAME AS NICKNAME | rename SERVICE.Machine_Name as Machine_Name | table NICKNAME NPID source_SERVICES Machine_Name ] | lookup MXTIMING_lookup_Base Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | appendpipe [| where isnull(Threshold) | rename TYPE AS BACKUP_TYPE | eval TYPE="" | lookup MXTIMING_lookup_Base Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | rename BACKUP_TYPE AS TYPE] | dedup Time, NPID,Context+Command | where Elapsed > Threshold OR isnull('Threshold') | fillnull Tags | eval Tags=if(Tags=0,"PLEASE_ADD_TAG",Tags) | makemv Tags delim="," | eval Tags=split(Tags,",") | search Tags IN (*) | eval source_SERVICES_count=mvcount(split(source_SERVICES, " ")) | eval NICKNAME=if(source_SERVICES_count > 1, "MULTIPLE_OPTIONS_FOUND",NICKNAME) ... View more

HI Thanks for this. I am running Splunk 7. with splunk admin permissions still your dashboard-2 hang ? - Yes it hangs on most screens only 3 jobs are running an nothing is Q. This screen will not load http://splunk:8000/en-GB/app/murex_mlc/job_manager So to test, i run a very large search on dashboard 1 - about 5 mintes of heavy tstats. Then when it gets going, i try and open another dashboard, but its like as if it cant connect to SPLUNK. So i don't even get to click on dashboard 2, to drive difficult search. At this point Splunk is not really working at ll. Even running http://splunk:8000/en-GB/app/murex_mlc/job_manager does-not load Any help would be great 🙂 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 9 0 0 3628604 4292 379449056 0 0 233 195 1 1 3 1 96 0 0 6 0 0 4089044 4292 379941632 0 0 65061 301 113781 23623 9 4 87 0 0 8 0 0 2759252 4292 380373920 0 0 51228 436 75845 24211 14 4 81 0 0 7 0 0 2921780 4292 380504128 0 0 42862 325 75419 20671 13 4 83 0 0 7 0 0 2370636 4292 380812768 0 0 38909 55606 91095 21279 10 3 87 0 0 8 0 0 2458808 4292 380990176 0 0 50385 12916 88924 20976 10 3 86 0 0 8 0 0 3018360 4292 381120640 0 0 48407 286 94513 16208 11 3 86 0 0 11 0 0 1739156 4292 380724224 0 0 46630 286 338549 23542 12 4 84 0 0 11 0 0 1399432 4292 380125376 0 0 10394 7102 790517 23970 15 4 81 0 0 10 0 0 1481324 4292 380425664 0 0 0 111 628053 21174 15 4 81 0 0 11 0 0 2428852 4292 380682944 0 0 0 50158 955866 22223 13 5 82 0 0 11 0 0 1462988 4292 380900480 0 0 0 8618 799501 21775 13 5 82 0 0 9 0 0 1520836 4292 380361088 0 0 0 53 632406 16847 14 5 82 0 0 10 0 0 2000348 4292 380087680 0 0 0 2 815441 23082 14 5 81 0 0 30 0 0 1548704 4292 380093952 0 0 0 0 792144 20819 16 5 79 0 0 14 0 0 4115076 4292 376929664 0 0 6 46813 889905 24017 32 9 59 0 0 13 0 0 4267528 4292 376846592 0 0 3 4637 818489 25031 21 6 73 0 0 13 0 0 5025256 4292 376267840 0 0 3 4088 818270 25629 19 6 75 0 0 12 0 0 5800516 4292 375848128 0 0 0 0 796476 18840 19 5 76 0 0 11 0 0 6252184 4292 375760096 0 0 0 766 863946 18655 16 5 79 0 0 10 0 0 5578976 4292 376279008 0 0 0 48390 735564 23757 15 4 80 0 0 ... View more

HI Thanks for this, the first answer did work for me. But this is good to know for the future. Cheers Robert ... View more

hi Thanks for this this worked 🙂 ... View more