About mwk1000

mwk1000 · ‎05-12-2025

So if your indexers have separate storage filesystem for indexes consider pre upgrade creating the links ln -s /mypath-to/mongo /opt/splunk/var/lib/splunk/kvstore/mongo for a headach free kvstore update

mwk1000 · ‎05-12-2025

I also hit an upgrade bug with 9.4.1 on a clients indexers , the upgrade migration mongo 4-7 failed to run due to the scripts not using SPLUNK_DB but hardcoding /opt/splunk/var/lib.... The indexers had a separate filesystem /data01/,,,, I was able to create a link from the mongo under /opt/splunk/var/lib/splunk/kvstore... to the "real" one in /data01 and restart triggering the upgrade process to complete properly ....

mwk1000 · ‎03-17-2022

One pass rule - NO it will apply the source:: if it matches ...

mwk1000 · ‎03-17-2022

Yes, remember the ONE PASS rule an even goes thru ONCE based on highest precedence , you can CLONE_SOURCETYPE to create another event and have a second go ....

mwk1000 · ‎03-01-2022

Solved- after some trial and error the user was being fed in to the call since it was a required attribute BUT that triggers a user level save (private ) using "nobody" keeps it in the app...

mwk1000 · ‎02-25-2022

I also have been having "issues" , i have created a utility in python to extract all KO's for a list ( lookup table ) of users. It will clone all KO's to a designated target app. It works well for all objects until we decided to add a "disable source" option. This revealed that "global" objects were not truly being cloned but "acting" as if they are.... A part of the process calls the target to see if it already exists... This is always true for a "global" object returning no 404 error but instead a full result but examining past the atom links the ['entry'][0]['links'] all point back to the original app links.... I can even disable the source and enable the target .... a disaster waiting to happen. If I move the target app ( to the cloud for instance ) it has NONE of the relevant attribute to function ... like the base search ... Is there any FULL documentation to the /move links since adding 'sharing'= app or global does not stop a new private object being created.... (as is suggested in Splunk rest doc's but the examples are VERY simplistic

mwk1000 · ‎05-23-2018

What about the fields generated by the notable action ? event_id urgency owner etc ? I understand the correlation search result rows but the notable is an adaptive response itself , is there a way to get those added fields as part of an adaptive response assigned to a correlation rule ?

mwk1000 · ‎02-07-2018

Upgraded from 2.4.1 all services/entities in Global Team. Team admin role has been inherited by itoa_admin. The "bulk" import options do not appear for any of the existing users but as an admin role they do, adding admin role to an existing itsi admin user does not work. No errors visible in logs, all capabilities are per the docs and this is in a search head clustered environment.

mwk1000 · ‎11-15-2017

except values() also has a limit

mwk1000 · ‎11-07-2017

-- much of the large search load 7-8GB memory / search were in the data model accelerations, performance and another ITSI model I can't recall -- but this is very dependant on YOUR site event volumes. Default of 3 concurrent accelerations / model / sh entity can create a large background load of searches -- BIG searches ....

mwk1000 · ‎11-07-2017

Unfortunately "An important question is which splunkd processes" is always the main splunkd since it is the process and the rest are threads ... so OOM is a nasty problem. I will share one issue we have seen with low memory ( 32GB) hosts and many concurrent search jobs. limits.conf defaults say to check 5 process runners and if none are available start another but keep them around for 7200 seconds each one allocating a variable chunk of memory. Run dmesg on your crashed box and see at the moment of the crash how many threads were running. In our case we has a few hundred so we trimmed up the timelines and expanded the limit from 5 to 20. This worked wonders in eliminating the frequent crashes and the possible bucket corruptions that follow ! The total number of search runners spiking only briefly as needed.

mwk1000 · ‎09-07-2017

-- This was confirmed in a lab by pointing the outputs to lab indexers while using a different set of indexers as search peers. All DM buckets are built fine on search peers and are usable for pivot searches. All DM buckets are segregated by search head group (GUID) so the are isolated from impacting other search clusters you just add up more storage for each search group ( standalone/pool/cluster) and more running searches on your search peers.

mwk1000 · ‎09-07-2017

Summary indexing would require an outputs.conf but persistent data model acceleration actually does not since nothing is indexed. The data model buckets are built via the scheduled search so having them as search peers is enough to get the buckets built. ( Lab setups etc ) In general a complete setup would do both but it's not necessary for simply building the accelerations on the indexers.

mwk1000 · ‎07-07-2017

I feel your pain, the deployer will throw EVERYTHING in default ( ie. not editable ) . I have had some success at fixing config problems with finding the captain, editing the file, reloading ( NOT reboot or you loose the captain - debug refresh ) the captain, and force syncing all members to the captain. UGLY. I think the best bet at this stage of search head cluster maturity is to make the rest API your friend and use the config file calls that allow updates to elements, stanzas, or the whole thing. These would be "legal" synced by raft calls to the cluster. I'm sure it will get better with time, 6.6 just added a fix for ownership that used to kill us when someone left and their ID was no longer available ( yet they owned popular knowledge objects )

mwk1000 · ‎07-07-2017

A bit old I know but for those looking a couple of cautions. TCP is quite a bit heavier than UDP so if you are already dropping packets then TCP may require more hardware capacity immediately. For sheer volume UDP is hard to beat pound for pound and in this case pounds=$$$. How many endpoints are feeding ? A few large ? 10,000 small ? UDP I would prefer in the latter case since the LB will still work on source address ( and today you can set datagram LB ) I have used F5 load balancers with both UDP/TCP to a heavy forwarder running a self contained Splunk app with syslog-ng as the core. This was used to collect, prefilter, proxy to additional subscribers , and log to splunk. This is a "blended" approach that scaled exceptionally well, 1 LB to 4 HF's typical and since the syslog-ng is a Splunk app the logic(configs) and deployment was done via a deployment server. We had 20+ syslog-ng/hf's running as a reliable/flexible/very high performance collection layer.

mwk1000 · ‎06-13-2017

Generally as a safety practice I would replace the use of lazy .*? with a negative character class [^>] and a possessive quantifier ++ or (?>...) when you can. It's quite common syntax in Splunk configs because of the elimination of backtracking and performance boost. Splunk has the potential to apply your regex to huge numbers of events that will NOT match and this will speed up the failures. ( Speed up the matches as well )

mwk1000 · ‎05-11-2016

We are seeing the space char is causing this in the path, GUI bug

mwk1000 · ‎04-14-2015

Multi-threaded so it's a beast and will sustain very high volumes. ( The driving reason it was created )

mwk1000 · ‎04-14-2015

I did just that, I created a Splunk application with syslogcontrol (start/stop/status) compiled the binaries , installed into an app structure with all needed libraries, and updated the syslog-ng configuration files to use environment variables for the "base". It will run from the app deployment structure self contained. Now we just deploy our "app_syslogcollector" and we are up and running as the Splunk account. The log file rotation is also self controlled from within the app. It works very well all under our control , self contained deployed to over 20 hosts... The syslog-ng configs create directories based on the index/sourcetypes we want /apps/syslogs///syslog.log A separate simple inputs app monitors the paths and inputs the files. ( put the asterisk's back in where they belong ) [ monitor:///apps/syslogs/cisco-rtr/hostname/syslog.log] host_segment=4 index=network sourcetype=os:cisco:syslog Note we listen above 1024 to avoid root requirements and conflicts with existing syslog, we are a RECEIVER not a local host syslog. The disk acts as a buffer for "incidents/outages" Splunk will pick up where it left off up to your log rotation settings. Accepts TCP/UDP will forward again externally if you need to ...

mwk1000 · ‎02-17-2015

We ran into severe performance degradation and deduced it was memory page handling. An alternative that worked well on our hardware was setting vm.zone_reclaim_mode = 1 This was set to 0 causing memory allocation to occur across nodes in a NUMA architecture. Setting this value forced reclamation of memory assigned to the local node first. With the transient nature of the Splunk search workload processes the impact was stunning. Server load averages were regularly spiraling out of control many times s over 100 on 12 core / 24 hyperthreaded indexers and searchheads. The symptoms are the same , servers became completely unresponsive for minutes at a time. The root cause may be the same -- poor cross node memory management. We set the parameter and in a matter of a few minutes the load was 2-3 and very hard to drive over 5-6. End user search performance was orders of magnitude faster. Red Hat Enterprise Linux Server release 6.5 my investigation shows this setting to be CPU specific and some users reported worse performance in our case it was dramatically improved. Another option to consider when facing similar performance issues.

mwk1000 · ‎12-19-2014

If it is modified from the original you can look in python.log (in the default logs dir )

mwk1000 · ‎12-11-2013

I have the same problem, I have good results being returned but the truncate statement does not seem to work. One note the { is a special character and should be escaped (\{) if you are actually looking for it. I expected to get the whole JSON structure but it is still being chopped yet sourcetyped as the stanza i have defined. --- I see the web for stripped the \ so ignore if it happened to you as well - Cheers

Posts	22
Solutions	1
Karma Given	0
Karma Received	5
Member Since	‎01-12-2012

Online Status	Offline
Date Last Visited	‎05-12-2025 11:26 AM

Why do the ITSI 3.01 Upgrade itoa_admin users no l...

Re: KVStore does not start when running Splunk 9.4

Re: KVStore does not start when running Splunk 9.4

Re: Do TRANSFORMS in a source stanza and a sourcet...

Re: Do TRANSFORMS in a source stanza and a sourcet...

Re: Python - ReSave Splunk (existing) Knowledge Ob...

Re: Python - ReSave Splunk (existing) Knowledge Ob...

Re: How can I Pass/access Notable Event fields to ...

Why do the ITSI 3.01 Upgrade itoa_admin users no l...

Re: How to overcome sub search limitation (only 10...

Re: SplunkD Causing Linux OOM Condition

Re: SplunkD Causing Linux OOM Condition

Re: Indexer Cluster and Search Head Cluster with D...

Re: Indexer Cluster and Search Head Cluster with D...

Re: What is the correct way to edit conf files and...

Re: At what point do you need to consider scaling ...

Re: How to write a regular expression to extract t...

Re: Why am I getting "Error occurred attempting to...

Re: Is there a good way manage the process of usin...

Re: Is there a good way manage the process of usin...

Re: How do I disable Transparent Huge Pages (THP) ...

Re: Why is my custom sendemail script not working ...

Re: Long json got truncated even though TRUNCATE i...

Are you a member of the Splunk Community?