About isoutamo

isoutamo · ‎08-30-2025

What you are meaning with “optional upgrade”? I think that most new versions are somehow optional. If those are not fixing any big problems which are preventing the use of current version then it’s somehow optional. Even it presents a new features it still optional if one doesn’t need those? Of course currently one must update their apps regularly or otherwise those are archived in splunkbase. But even these cases one needs to update e.g. splunk SDK on those. Is that upgrade mandatory or optional it’s up to your opinion.

isoutamo · ‎08-30-2025

Also the best practice is not to use timestamp field as a checkpoint column. There are many reasons for this. With high volume db this field can contains several rows with same value. Also converting checkpoint values between different data types is not a good practice. The best option is use enough big serial as a checkpoint field. You also must have index for that field.

isoutamo · ‎08-30-2025

Here https://help.splunk.com/en/splunk-enterprise/administer/monitor/9.4/configure-the-monitoring-console/multi-instance-monitoring-console-setup-steps is instructions what you need to do. Remember that you cannot put MC in any instance you want, instead of that you must select correct one. Then as earlier said add all needed nodes as search peers except clusters where you should add only cm. As @PickleRick said, DS is just one node. Actually don’t configure MC as an additional role for DS! In almost any environment you should have dedicated DS without any other roles!

isoutamo · ‎08-29-2025

As already said, You cannot change that via DS when your UF's configuration is in etc/system/local. How to fix it? You must create own app for this kind of settings for UF. Probably you need more than one? Then put those configurations and needed serverclass.conf into DS and deploys as usually. After that you must remove those settings manually from UF's conf files at etc/system/local. Restart UF's SplunkForwarder service and now you have those configuration managed by DS. IMHO: you should put also serverclass.conf into own app or even better to put in individual apps for each integration. Don't use GUI as it put those all under etc/system/local and then you have the same issue with DS than you have now in UF. I use this kind of structure: xxxx-app1-uf xxxx-app1-ds-sc xxxx-app1-ihf (if I need to use intermediate hf to manipulate and forward those events) Then I have separate xxxx-ds-base-app where is DS's base configurtions. With this concept I can manage sever integrations at same time independently each other. No need to do manual editing for one serverclass.conf file on DS. Also this can easily automated.

isoutamo · ‎08-28-2025

Exactly that way. You must do data onboarding in test/dev environment and ensure that your splunk shows only what you want! Normally this means that you must have 1st your own dev node etc. where you could create inputs, props and transforms. Then install those in some kind of systest or integration test environment where you could validate your configurations. If you don't have this kind of environments then you have some other issues in audit and governance cases than only PII data in production splunk.

isoutamo · ‎08-28-2025

Click > before event, it opens to you more information where you could see and click button "Event Actions", then select "Show Source". I your event is not too long it open it as it is in disk. There you can see e.g. those escape marks etc.

isoutamo · ‎08-28-2025

I think that acceptable_keys didn't work as your json event is in escaped text mode in disc. If you want to use event like json you must use INGEST_EVAL and json-functions. But I expecting that in that case you hit again a same limit to read that event in, convert it from escaped text to json and save again back to stream. The best option is do this masking before ingestion with some other tool than Splunk's props and transforms. Is it possible that you ask that source already mask it or can you use e.g. Ingest Action or Edge or Ingest Processor? Also one option is Cribl outside of Splunk world.

isoutamo · ‎08-28-2025

As already said those "deleted" events are still in disk and actually if you have access to server cli, you could make those readable again. Basically this means that your delete solutions is not 100% governance compliant. The only way to get rid of those is delete all data from those indexes and then reindex other events. Normal way in data onboarding is that you make everything 1st in test/dev environment and ensure that there is no PII or something other unwanted events. After that deploy those configurations into production. But if you still want to use delete, my suggestion is that use enough small time spans and run it several times instead of running all in one.

isoutamo · ‎08-28-2025

At least earlier DS version "lost" old entries after you reboot it. Then those will be back when they have 1st connected into it. I'm not sure if this is still valid for current DS which are using index to store client information. I have never looked it with those versions.

isoutamo · ‎08-28-2025

I just realized that you are trying to send e.g. SH logs to indexers via HF, if I understand correctly? Why there is a this kind on order to add unnecessary complexity into your environment? Normal way is send all Splunk infra instances (like SH, LM, MC, CM, DS, Deployer etc.) log directly into indexers. This is actually 1st time when I even heard this kind of additional step/requirement! With this setup you will be probably get more issues than what this try to solve (I couldn't even guess what the problem is what you are trying to solve). But as said the issue is that indexers only listen to TLS enabled ports. This means that also those other Splunk Servers are trying to send TLS enabled streams. When you try to sent TLS enabled stream to HF (which just listen plain TCP), it didn't work. My proposal is that forget (at least SH, MC, LM, CM) to use HF between those and Indexers. If you have e.g. DS, HFs etc. outside of your main servers then those could be configured to use HF as IHF. Then just ensure that those are using plain tcp instead of tcp with ssl in outputs. conf.

isoutamo · ‎08-23-2025

This is not the whole truth. Even you don’t need DS for log collection and you can run your environment some time without DS, you could also create “HA” version of DS. With current DS versions 9.2+ it’s architecture has changed to support it. You should read more https://help.splunk.com/en/splunk-enterprise/administer/update-your-deployment/9.4/configure-the-deployment-system/implement-a-deployment-server-cluster

isoutamo · ‎08-22-2025

This work differently in single indexer vs cluster. With single indexer you should use cli command to delete indexed data and remove index definition. This works with that one command. In cluster this is two step task with manual remove files from each node.

isoutamo · ‎08-22-2025

Why you are transforming KV event into JSON? The KV events are much easier and faster handle in Splunk.

isoutamo · ‎08-22-2025

I doubt that it's faster if you indexed those fields in index time. When you create (or splunk create) indexed fields for all JSON event it means that it will create huge .tsidx file. Usually this means that your searches are slower as peers need to do a lot more IO operations than with not indexed fields. Can you share your use case with some sample data and we probably can help you with it? IMHO using KV data instead of JSON you can get much more performer Splunk environment.

isoutamo · ‎08-22-2025

At least from MC you could remove those old "missed" nodes by rebuild forwarder database. MC -> Settings -> Forwarder Monitoring Setup then Rebuild forwarder assets button. You should remember that this clears all old missed nodes over your selected time span. Usually there is no need to reinstall UF. It's better to update it on place. Another issue which you find after remove + install again is, that UF will reindexing all files what it still have on disk. The reason for that is, when you remove old installation you also remove local fishbucket index where UF has bookkeeping what it has already indexed.

isoutamo · ‎08-22-2025

You should configure also your HF to listen with TLS/SSL. Without that UF's cannot send events to HF which then could forward those into Indexers. Have you some real reasons why you want to use HF between source systems and indexers? The best practices is sent events directly into indexers!

isoutamo · ‎08-22-2025

Have you try to run this SPL with macros expanded? You can do it in SPL box with key combinations Ctrl+Shift+E (windows) or Cmd+Shift+E (macOS). With that way you could modify / comment out content of those macros and debug what will happen and where is the real issue.

isoutamo · ‎08-22-2025

If this is bug, it's not 1st one with REST and indexes/volumes 😞 Anyhow when you are pushing a new bundle w/o those "deleted" indexes you should remember that those are not removed from peer's disks! Those are just removed from indexes.conf. If you really want to remove and free that disk space you must remove those from OS level after you have pushed indexes.conf into those and use MC or CM to check that splunk didn't see those anymore.

isoutamo · ‎08-22-2025

Are you sure that no-one hasn't extend the lifetime of this job?

isoutamo · ‎08-22-2025

You should use some more recent version than 9.1. It's already out of support phase. See https://www.splunk.com/en_us/legal/splunk-software-support-policy.html#core Please try e.g. 9.4.4 which should be quite ok for testing and production use. Those bots version are also old and especially TA's etc. which it's used are also outdated and even archived. There is also bots.splunk.com where you can training some use cases with guides.

isoutamo · ‎08-22-2025

Please create a support ticket. Anyhow it's best practice to wait something like X.0.3 or even X.1.2 or similar before go into production. There have been almost every time when a new version has launched more or less nasty and critical bugs.

isoutamo · ‎08-22-2025

At least some ARM / Apple silicon beta has some app management issues. Maybe there is something similar in some cases. So if those change browser / disable extensions didn't help you, you should create a support ticket to Splunk.

isoutamo · ‎08-15-2025

As already said, you don't need to have a HA SHC Deployer as it's not needed continuously on operations. Don' t do overcomplicated environment which creates additional issues sooner or later. BUT I have had situation when Deployer have been down several days/weeks and then it rises issues. So the only thing what you need to do is ensure that you can redeploy Deployer if need within couple of hours. Even you can create some weird configuration and get those to work, don't do it if it leads to unsupported configuration! You or someone after you will be find it when it hits somehow!

isoutamo · ‎08-15-2025

I have reported this to the PM several months ago and he/she promised to look it, but doesn't promise any quick fix 😞

isoutamo · ‎08-15-2025

If I recall correctly 8.2 works ok with tls1.2? Both versions 8 and 9 are using same major version of openssl, so there shouldn't be any (real/unsolvable) issues when you are doing that update. Splunk 10 is different case as it has update it's openssl to version 3. I haven't done that update/migration yet, so I cannot said that for sure. But at least those beta instructions give some additional warnings what you must do before that update.

Posts	5318
Solutions	525
Karma Given	653
Karma Received	1618
Member Since	‎14-02-2017

Online Status	Offline
Date Last Visited	Sunday

Wrong parameters on macOS and logd input?

Find enabled windows inputs by SPL query?

Why can't I change alert with REST- It change perm...

Why does Splunk_TA_aws hang on configure / inputs ...

Don't execute rest of commands if there is no even...

Indexing Ubisecure Ubilogin logs?

Fieldformat didn't work with foreach

Cannot migrate DBConnect 2.4.0 -> 3.2.0 on Splunk ...

How to check splunk multisite cluster status after...

Re: Splunk add-on : Optional upgrade

Re: Issue integrating Oracle DB with DB Connect Sp...

Re: remote instances are not showing up under moni...

Re: Custom app not overriding UF inputs.conf (WinE...

Re: Deleting historical logs

Re: Transforms Truncating - Setting SOURCE_KEY

Re: Transforms Truncating - Setting SOURCE_KEY

Re: Deleting historical logs

Re: Deployment Server: Duplicate clients after rei...

Re: Splunk Forward Internal Logs

Re: Can universal forwarders have 2 different depl...

Re: Rest query showing deleted indexes

Re: Need to exclude or discard specific field valu...

Re: Indexed Extractions not working properly

Re: Deployment Server: Duplicate clients after rei...

Re: Splunk Forward Internal Logs

Re: query help in splunk cloud

Re: Rest query showing deleted indexes

Re: Why Isn't My Search Job Expiring? The "Zombie"...

Re: Splunk UI is very slow

Re: Universal forwarder version 10 memory leak

Re: Splunk Enterprise 10.0.0 unable to "browse mor...

Re: Splunk Search Head Deployer HA

Re: upgrade to kvstore v7 impossible on MacOS?!

Re: Splunk Enterprise version upgrade from 8.2 to ...

Join the Conversation