HI Thanks for the answer, but its not clear to me how to install a second indexer - this is about a forwarder. For example if i was installing a second indexer on a different machine, how do i do that? Then if i get that working i can copy it back to my main machine and change a port perhaps. Rob ... View more

HI I have time controls on search with, so the less time i look over the quicker it is. $time_selection.earliest$ $time_selection.latest$ Did you change something else in the query as well to improve performance? ... View more

Hi Thanks for your help. The SPL is large and only looks lover 24 hours (I cant go any longer + i need all the data it is returning). It is tstats and what i was told that if i have multiple indexers it will help to get the data quicker. This is the SPL | tstats summariesonly=true max(MXTIMING.Elapsed) AS Elapsed max(MXTIMING.CPU) AS CPU max(MXTIMING.CPU_PER) AS CPU_PER values(MXTIMING.RDB_COM1) AS RDB_COM values(MXTIMING.RDB_COM_PER1) AS RDB_COM_PER max(MXTIMING.Memory_V2) AS Memory max(MXTIMING.Elapsed_C) AS Elapsed_C values(source) AS source_MXTIMING avg(MXTIMING.Elapsed) AS average, count(MXTIMING.Elapsed) AS count, stdev(MXTIMING.Elapsed) AS stdev, median(MXTIMING.Elapsed) AS median, exactperc95(MXTIMING.Elapsed) AS perc95, exactperc99.5(MXTIMING.Elapsed) AS perc99.5, min(MXTIMING.Elapsed) AS min,earliest(_time) as start, latest(_time) as stop FROM datamodel=MXTIMING_V9 WHERE host=QCST_RSAT_V41 AND MXTIMING.Elapsed > 0 OR 1=1 GROUPBY _time MXTIMING.Machine_Name MXTIMING.Context+Command MXTIMING.NPID MXTIMING.Date MXTIMING.Time MXTIMING.MXTIMING_TYPE_DM source MXTIMING.UserName2 MXTIMING.source_path MXTIMING.Command3 MXTIMING.Context3 span=1s | rename MXTIMING.Context+Command as Context+Command | rename MXTIMING.NPID as NPID | rename MXTIMING.MXTIMING_TYPE_DM as TYPE | rename MXTIMING.Date as Date | rename MXTIMING.Time as Time | rename MXTIMING.Machine_Name as Machine_Name | rename MXTIMING.UserName2 as UserName | rename MXTIMING.source_path as source_path | eval Date=strftime(strptime(Date,"%Y%m%d"),"%d/%m/%Y") | eval Time = Date." ".Time | eval FULL_EVENT=Elapsed_C | eval FULL_EVENT=replace(FULL_EVENT,"\d+.\d+","FULL_EVENT") | join Machine_Name NPID type=left [| tstats summariesonly=true count(SERVICE.NPID) AS count2 values(source) AS source_SERVICES FROM datamodel=SERVICE_V6 WHERE ( host=QCST_RSAT_V41 earliest=1539054000 latest=1539212400) AND SERVICE.NICKNAME IN (*) GROUPBY SERVICE.Machine_Name SERVICE.NICKNAME SERVICE.NPID | rename SERVICE.NPID AS NPID | rename SERVICE.NICKNAME AS NICKNAME | rename SERVICE.Machine_Name as Machine_Name | table NICKNAME NPID source_SERVICES Machine_Name ] | lookup MXTIMING_BASE.csv Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | appendpipe [| where isnull(Threshold) | rename TYPE AS BACKUP_TYPE | eval TYPE="*" | lookup MXTIMING_BASE.csv Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | rename BACKUP_TYPE AS TYPE] | sort Threshold | dedup Time, NPID,Context+Command | where Elapsed > Threshold OR isnull('Threshold') | fillnull Tags | eval Tags=if(Tags=0,"PLEASE_ADD_TAG",Tags) | makemv Tags delim="," | eval Tags=split(Tags,",") | search Tags IN (*) | eval source_SERVICES_count=mvcount(split(source_SERVICES, " ")) | eval NICKNAME=if(source_SERVICES_count > 1, "MULTIPLE_OPTIONS_FOUND",NICKNAME) | search | timechart bins=1000 max(Elapsed) by Tags limit=20 This is the job inspect Screen reader users, click here to skip the navigation bar Search job inspector This search is still running and is approximately 100% complete. (SID: admin__admin_bXVyZXhfbWxj__baseSearch_1539267931.395925) search.log Execution costs Duration (seconds) Component Invocations Input count Output count 5.16 .execute_input.flush_prestats 5 895,975 895,975 52.08 command.tstats 89 956,472 1,446,184 41.01 command.tstats.query_tsidx 10 - - 11.04 command.tstats.execute_input 44 956,472 - 0.00 dispatch.check_disk_usage 4 - - 0.00 dispatch.createdSearchResultInfrastructure 1 - - 0.00 dispatch.evaluate 1 - - 0.01 dispatch.evaluate.rename 8 - - 0.00 dispatch.evaluate.eval 1 - - 0.00 dispatch.evaluate.tstats 1 - - 0.00 dispatch.evaluate.noop 1 - - 27.54 dispatch.fetch 45 - - 0.00 dispatch.optimize.FinalEval 1 - - 0.03 dispatch.optimize.matchReportAcceleration 1 - - 0.04 dispatch.optimize.optimization 1 - - 0.00 dispatch.optimize.reparse 1 - - 0.00 dispatch.optimize.toJson 1 - - 0.00 dispatch.optimize.toSpl 1 - - 10.31 dispatch.preview 1 - - 5.20 dispatch.preview.tstats.execute_output 1 - - 3.91 dispatch.preview.command.rename 8 483,976 483,976 0.67 dispatch.preview.command.eval 1 60,497 60,497 0.52 dispatch.preview.write_results_to_disk 1 - - 41.04 dispatch.stream.local 45 - - 0.02 dispatch.writeStatus 14 - - 0.14 startup.configuration 1 - - 0.18 startup.handoff 1 - - Search job properties Server info: Splunk 7.0.3, splunk:8000, Thu Oct 11 15:27:05 2018 User: admin ... View more

Hi Thanks for your help. I am looking for Indexers (Not indexes) , I have a 6TB SSD so i have a really good box but i am just not using it. I understand the VM option, but before i try that i would like to try 2 or 3 Indexers on the same machine to see if it help for performance. I also have a beta site that i can test this on. I cant remember the name of the person that was helping me in the .conf. - sorry 😞 If you can tell me what files to change so i can test that would be great as i cant find any doc on this Cheers Robert ... View more

The issue is i dont know how to do it ... View more

Thanks. But how do i create them and how do i manage ports. As they will be all on the same machine. ... View more

I have a 60 core box but only use 25% of it. So after discussions in .conf we think it best to try and add more indexers onto the same machine. ... View more

Hi. After been to the .conf and talking to a meet the expert we went over my issue. I have large tstat commands taking a long time. He said this can be reduced with multiple indexers. At the moment i have a 60 core Intel chip machine and we only use 25% of this machine. So i am looking to add more indexers onto it. ... View more

Thanks for this as well - i used this also 🙂 ... View more

One last thing if possible. Do you know how to make them smaller. I posted a question but have gotten no hits yet. Cheers Robert ... View more

Hi Thanks for the answer. So i am trying to understand this. Normally the JMX would need a lot of information to pull JMX data down, this is stored in files in the JMX app. for example /dell425srv2/apps/splunk_old_versions/splunk_7_0_3/splunk/etc/apps/Splunk_TA_jmx/local jmx_tasks.conf [MUREX] description = MUREX disabled = 0 index = jmx interval = 20 servers = Splunk_TA_jmx:TEST2 | Splunk_TA_jmx:TRADE_INSERTION sourcetype = jmx templates = Splunk_TA_jmx:MUREX This would also need jmx_servers.conf [TEST2] description = TEST2 has_account = 1 interval = 60 jmx_url = service:jmx:rmi://mx12673vm:9092/jndi/rmi://mx12673vm:9092/jmxrmi [TRADE_INSERTION] description = TRADE_INSERTION has_account = 1 interval = 5 jmx_url = service:jmx:rmi://hp656srv:26827/jndi/rmi://hp656srv:26827/jmxrmi What i need is to add multiple jmx_url onto the jmx_server.conf and create an increased line in servers = Splunk_TA_jmx:TEST2 | Splunk_TA_jmx:TRADE_INSERTION + X + Y etc... What i am not really getting is how this custom search command would work? What would the workflow be in this case? Would it change these files or would it be something different pure command line that i could run and it would do what above is doing but on the fly? Thanks in advance Robert Lynch ... View more

Brill - 100% brill and thanks 🙂 ... View more

HI I have the same issue, could you post up your code please i am finding it hard to read to screen shot. Cheers Rob ... View more

HI I have the same issue, could you post up your code please i am finding it hard to read to screen shot. Cheers Rob ... View more

Hi We found the issues, Splunk was monitoring over 20,000 files most of them where old. When we deleted 19,000 of them the issues was resolved. However i think there is a bug in the forwarder here. Thanks for your help on this Robert ... View more

Hi - Thanks for replay. I don use useAck , i have checks all logs and i cant see any major issues....hmmm This is a large environment with a lot of activity, however the max is 40G - it work go over that. I wounder if i can reduce the 40GB Cheers Rob ... View more

Thanks for your answer 1st - The CPU is fine, it is the memory that is the issue. It is difficult for me to reduce the ... in some case i need them 2nd THP in off on the main Splunk install, however it is on at some of the forwarer servers - do you think this could be the issue? 3rd This is set to 0 on server and forwarder. Cheers Rob ... View more