I need to disagree to use any x.0.0 versions. Especially when dbx’s 4.0.0 contains totally new features (real HA), it’s more than probably that there will be less or more other annoying issues with it! ... View more

The port 8089 is used only for rest api requested a responses, not for sending logs! You need separate port for those like 9997 is in normal situation. It doesn’t mater what it is. Only ensure that it’s allowed in all FWs between SH and indexers. When you are flipping the port to a XXXX or 9998 then indexer discovery tells SH that there is a new receiver port activated and SH should use also it and remove previous 9997. If there is e.g. FW blocking traffic from SH to indexers for those new ports then SH can’t work as expected and , I expect, later when it lost access to its current LM logs there will start those other issues which you have mentioned? You should find some hints from your instances internal logs if this is really what has happened. ... View more

If you could figure out working version, then you could try to download it with this https://github.com/ryanadler/downloadSplunk   ... View more

Yes, you could define several ports if needed, by adding a new receiver into those indexers by app via CM. I’m not sure if I understood correctly that you flip your receiver port to some invalid value or something like that? Basically you could have separate port reserved for internal nodes which is blocked by firewall from normal traffic from UFs etc. This is allowed only from SH etc. Another receiver port is for all other UFs and IHFs (intermediate heavy forwarders). Then when you need to block real incoming indexing traffic, just disable that port. Then SH stop using this, as it gets information that this is closed by indexer discovery! And continue to use that SH only port. But still I said that you should update your license to cover your real ingestion needs or remove unnecessary ingestion. ... View more

Usually it's better to create a new question a did needed then link to old thread. In that way you will get more answers your issue! When you have issue with DS then you should add this configuration into outputs.conf not distributed search.conf. If this is not helping you, then create a new question with more retailed information about your current situation and your configurations! ... View more

Exactly that way. And Splunk has changed requirements to have higher version in Indexers with version 9.x. (not sure which minor x was). Now you can officially have newer UF version than receiving HF/IDX version. ... View more

Yes I'm aware that MC has these panels where you could see some statistics about license usage.  Unfortunately those are not aligned with current license policy 😞 If you have official enterprise then the license policy is 45/60 not 3/30. And if your license size is 100GB+ then it's nonblocking for searches. And independent of your license there is no blocking for ingesting side. So even you have done hard license breach the ingesting is still working, but you cannot make any searches except for internal indexes (figure out why you have done breach)! I'm expecting, that as you have LM in your SH, and you are sending your internal logs (including those which LM is needed) to your idx cluster, except when you have blocked your indexer discovery by wrong port information, you have hit by "missing connection to LM" instead of indexing internal logs. Anyhow you shouldn't flip any internal logs as those are not counted towards your license usage. Only real data which are sent to an other than _* indexes are counted as indexed data. Usually all that date is coming from UF/HF not your SH etc.  So my proposal is that you just switch receiving port of indexers to some valid port which are allowed only from SH side by FW. Then your SH (and LM in SH) can continue send their internals into indexers and everything should works. And same time all UFs with static indexer information cannot send events as the receiving port has changed. If you have any real dat inputs on SH then you should set up HF and move those inputs there. Of course the real fix is buy enough big splunk license.... ... View more

Here is old answer for upgrade order of nodes in distributed environment. https://community.splunk.com/t5/All-Apps-and-Add-ons/Upgrading-Apps-and-Add-ons-in-distributed-environment/m-p/554548/highlight/true#M65820 Quite probably you can this with different order, but then you will gotten some warnings when you are running it before those are in correct versions. ... View more

I don’t believe that you have any issues with those 8.x.x UFs with splunk 9.3.x or even. 9.4.x. Those will work together, maybe some modifications are needed, but probably none. Here is one old post which points to some other post based on your environment.   https://community.splunk.com/t5/Deployment-Architecture/Splunk-Migration-from-existing-server-to-a-new-server/m-p/681655/highlight/true#M28001 If/when you could do a new host which you can use for some testing this shouldn’t be an issue. Just test it with test systems with instructions from those above posts. When you have check and approved those test then just do real migration. I’m not 100% sure that there is not any issues with amz2023 version. I have some feelings that there could be something which need to configure separately e.g. cgroups or something else? You probably find more details from https://splunkcommunity.slack.com/archives/C03M9ENE6AD ... View more

Can you open more how and when you are doing this “flip”? You probably know that when you are over used your license it’s not mater how much you do it? When you are “flip” you indexer receiving port and node ind discovery update this information to its list. Then when someone ask it then it told it to those. Based on that UFs just update their targets based on that. If/when you have FW between your source and IDX then those will block connections and their cannot send events anymore to IDXes.  But if you have UFs configured to use static host+port combination then those try to send continuously to those. If your SHs and other infra nodes are using indexer discovery then those are starting to use those new ports. Of course if there is not FW openings between those nodes and IDXes then traffic stops and when queues get full then probably other issues arise. You should check that those “flipped” ports are open between SHCs and IDXes and then your environment should works as expected.  Then is this the best way to avoid license overuse is another story! ... View more

Time to add a new entry in ideas.splunk.com and ask this feature! Of course you should check if there is already this kind of idea. Then write up that idea here, so we could vote it too! ... View more

No need to use html tags. You could add those into title etc. But when you have lot of those and you will set and unset those based on buttons, clicks etc. then this approach doesn’t work anymore. ... View more

You should also check if CM see those peers as member of indexer cluster. Then also check what errors and maybe warnings which told what has happened.  ... View more

When you are playing with tokens in SXML, you should install this app https://classic.splunkbase.splunk.com/app/1603/ Then add this into your forms. <form version="1.1" theme="light" script="simple_xml_examples:showtokens.js"> After this it shows all tokens what you have and what are their values like When I add ip, but didn't press submit After submit is pressed. https://data-findings.com/wp-content/uploads/2024/09/HSUG-20240903-Tiia-Ojares.pdf   ... View more

Better to use RPM as then there are those pre and post scripts which are doing some cleaning etc. tasks which are not done if you are just unzipping that into /opt/splunk directory! And  with tgz you must always do as root "chown -R splunk:splunk /opt/splunk" or whatever your splunk user is  before you start it after update! ... View more

I supposing that when you are running this  <search id="operational_hours"> <query>| savedsearch set_operational_hours</query> <finalized> <set token="operational_start_time">$result.operational_start_time$</set> It used some indexes or other KO which are not allowed for regular users. You should check it users can use that command on GUI and get results. If I remember correctly there was some restrictions to use at least loadjob in SHC, but I'm not sure if there are same kind of restrictions with savedsearch command? I couldn't find any mentions in docs. Then you must remember this: When the savedsearch command runs a saved search, the command always applies the permissions associated with the role of the person running the savedsearch command to the search. The savedsearch command never applies the permissions associated with the role of the person who created and owns the search to the search. This happens even when a saved search has been set up to run as the report owner. If you need to get this run as an owner instead of running user you must use ref on dashboard for those queries. But even then you cannot add/modify parameters which those searches accept and used. If you try to use those then splunk run those also as user not as owner.  ... View more

Have you started your instance(s) every time after you have applied a new version? This is needed to make a needed conversions e.g. from 8.2.8 -> 9.1.0 etc.! Without those starts it’s almost same to do it directly 8.2.8 -> 9.3.4 especially if you are using tar.gz package. With rpm and deb installing a new, removing some old unneeded files too. But all conversion tasks have done only when you are starting the instance. ... View more

You could use dedup with sortby parameter, as I previously show. ... View more

Hi have you look this https://docs.splunk.com/Documentation/Splunk/9.4.2/Updating/Upgradepre-9.2deploymentservers ? There was some changes on 9.2 how DS has stored client information. This leads also in situation where you see those deployment clients on your SH as it get that information from your indexer's indexes (I suppose that you have forwarded all logs to indexer). r. Ismo ... View more

Normally spunk runs on your nodes all IPs unless you define it differently. So in splunk point of view you don't need to do anything. But usually nodes and network put firewalls between network segments and also in hosts. I suppose that you are running this in linux, so you need to check if there is local firewalld or iptables or some other host based fw in use. Just add those ports 8000 (for GUI) and 9997 (for input data from UFs) or what port you are using for ingesting data.  Based on your OS windows/linux+distro this has done different way. Also you should check if your sources (UFs) are in different network segments and if your users (laptops/workstation) are different segment then open access to those ports. Also there could be some web proxies in use which didn't allow traffic into your Splunk sever GUI (port 8000). ... View more

I modify @uagraw01 your answer to moving your SPL inside </> block in editor. In that way it's much more readable. Can you check that it is still correct and there haven't missed anything! Could you In future use that </> block always when you are adding some SPL, dashboards etc. With it we can ensure that what we see is what you have written, not something what editor has changed! ... View more

As you have used depends attribute on your input, I'm quite sure that this is not a capability issue, instead of it is undefined token. The easiest way to debug tokens is use simple_xml_examples app and add this to your dashboard <form version="1.1" theme="light" script="simple_xml_examples:showtokens.js"> After this addition, you see all used tokens and their values on bottom of page. https://splunkbase.splunk.com/app/1603  Helsinki UG presentation how to create and debug SXML dashboards https://data-findings.com/wp-content/uploads/2024/09/HSUG-20240903-Tiia-Ojares.pdf ... View more

The Splunk’s TZ has set to UTC on browser and workstation has correct PT TZ? What SSO/idp you are using? ... View more

If they haven’t set any time zone then it’s their workstation’s time zone. Is this tz wrong or what is the issue? I’m afraid that you are trying to solve an issue which doesn’t exists! ... View more

What you are meaning with “change tz for all users”? If they are sitting in PT time zone and they are using “use system TZ settings” then it is already in PT time zone if their workstations/ laptops are correctly configured. If you want to change that also for people which are not sitting in PT zone, then I ask why? In internally splunk is storing all times as UTC. Then it shows times by users time zone or what they have set in their account preferences. ... View more