About isoutamo

isoutamo

If you have configured SHC correctly (send all logs to indexers and no local indexes) then you already have it in place. If not then fix your SHC configuration.

isoutamo

This sounds like @richgalloway said. You have defined some other KOs like field extraction inline or transforms. Maybe eventtypes and/or macros which are private or if those are public, other user’s role hasn’t permissions to see and use those.

isoutamo

As @PickleRick said, it’s just data. Still there could be issues as (if I remember right) it was from version 7.x? After that there have been some changes in splunk internals, but I suppose that it will still works? Or you find out quite soon if not. Then those used add ons are different story. You could find newer versions for quite many, but probably not for all. Without correctly “onboarded” data it will be harder to find those answers, but not impossible. You have also option to run bots in bots.splunk.com for training purpose.

isoutamo

That's true. Then you must remember that rebalancing just count number of buckets when it does its work. Because buckets can have different sizes the disk space usage is not rebalanced just count of those. In rebalancing there are two options: rebalance primaries rebalance buckets 1st one is done automatically in quite many situations e.g. rolling restart etc. 2nd one is always manual work which target it set to 90% level. What I have done by myself is modify that %-level. Depending on environment I have used e.g. 95-99% to get better distribution of buckets over nodes. After you have gotten suitable level, you should adjust that % level back to 90%.

isoutamo

If you do a restart for your test license and check are those internal logs before restart usable and there is no error in internals then it should work. Also you could use that testing like in https://community.splunk.com/t5/Knowledge-Management/SmartStore-How-to-verify-splunk-indexer-connectivty-to-remote/m-p/399606 and it works then it works. There are more examples on help.splunk.com for test it.

isoutamo

I'm not sure it this is valid response or not? But when I look license file I couldn't found any feature for S2 in enterprise license. Unfortunately I haven't any demo/test node which I can switch to free license to test it and check how that license file changes. But I think that It could be possible that it's working with free version too. Best option is if you could test it with trial license and minio as S3 backend. If it works then it should work also with GCP environment. Of course it's better if you could test this also in GCP too. There is no mention of S2 as restricted/removed features in https://help.splunk.com/en/data-management/splunk-enterprise-admin-manual/9.4/configure-splunk-licenses/about-splunk-free If/when you have tested this, please inform us, is it working or not!

isoutamo

If I recall right there could be only one defaultGroup at time. It's not mater if it is under tcpout or syslog stanza, splunk use same for both. So if you want all goes to Splunk then use defaultGroup only in tcpout stanza and when you need to route some events into syslog use transforms with _SYSLOG_ROUTING. You should also remember that when you have configured more than one output targets which are used for same event, then if any of those stopped to receive events then all those stopped to work sooner or later. See more e.g. blockOnCloning etc.

isoutamo

If possible before you start you should backup whole splunk installation including SPLUNK_DB as there is no any supported rollback method! You should remember first stop splunk then full backup of it installation and then continue. And remember that you should have all those splunk versions including oldest to do a full restore if needed. You probably have some/many additional apps in your installations? Probably some of those have also interactions how to upgrade those version by version. You must check this and also try to get those needed versions. Unfortunately that could be quite hard task to do as many apps have only latest or couple of latest versions in splunkbase and also instructions are probably missing! So most important thing is make great plan how to proceed. One option (especially if you don't need old data) is start from scratch. Just install needed apps etc. into a new and start to get data in. Of course if you need old data regularly then this is not an option for you.

isoutamo

When you found answer, please add it here. So other can get it too when they have a same issue. Here is link to time modifier section on help.splunk.com https://help.splunk.com/en/splunk-enterprise/search/search-manual/10.0/specify-time-ranges/specify-time-modifiers-in-your-search

isoutamo

Or your last change index has named differently. You should check it from e.g. with btool on command line or use Admin little helper from splunkbase if you are in Splunk Cloud.

isoutamo

Here is excellent conf presentation how to find why queue is full. https://conf.splunk.com/files/2019/slides/FN1570.pdf

isoutamo

I doubt it. As far as I know, downloading is restricted to person who has generated the NFR license in portal.

isoutamo

Normally download right is a person who have generate and download NFR license in partner portal. If it was you and it didn’t work, you should contact partnerverse email or your contact at splunk.

isoutamo

As said the reason why those aren’t logged anymore into internals, is how GET and POST methods works. You can read more e.g. from https://www.w3schools.com/tags/ref_httpmethods.asp

isoutamo

On left side “interesting fields” with verbose search mode, did you have field name ResponsePayload or @ResponsePayload on it? If there are then you could try to use something like |spath input=ResponsePayload to get it as a json. But this needs that content of this field is real json format without errors! Basically you could take content of this field and test it with jq or some other tool to ensure it. If you haven’t this field extracted then again check that this is real json and also that it’s content doesn’t have any additional characters in source which breaks json format. I have seen e.g. that “ mark has replaced \u0022 which means that those are strings not jsons.

isoutamo

Have you registered this app in Entra side? Without that it cannot access EntraId. There are instructions in this manual how you should troubleshoot it. Just follow those instructions and tell to us if there are anything in internal logs where you need our help!

isoutamo

What you are meaning with “optional upgrade”? I think that most new versions are somehow optional. If those are not fixing any big problems which are preventing the use of current version then it’s somehow optional. Even it presents a new features it still optional if one doesn’t need those? Of course currently one must update their apps regularly or otherwise those are archived in splunkbase. But even these cases one needs to update e.g. splunk SDK on those. Is that upgrade mandatory or optional it’s up to your opinion.

isoutamo

Also the best practice is not to use timestamp field as a checkpoint column. There are many reasons for this. With high volume db this field can contains several rows with same value. Also converting checkpoint values between different data types is not a good practice. The best option is use enough big serial as a checkpoint field. You also must have index for that field.

isoutamo

Here https://help.splunk.com/en/splunk-enterprise/administer/monitor/9.4/configure-the-monitoring-console/multi-instance-monitoring-console-setup-steps is instructions what you need to do. Remember that you cannot put MC in any instance you want, instead of that you must select correct one. Then as earlier said add all needed nodes as search peers except clusters where you should add only cm. As @PickleRick said, DS is just one node. Actually don’t configure MC as an additional role for DS! In almost any environment you should have dedicated DS without any other roles!

isoutamo

As already said, You cannot change that via DS when your UF's configuration is in etc/system/local. How to fix it? You must create own app for this kind of settings for UF. Probably you need more than one? Then put those configurations and needed serverclass.conf into DS and deploys as usually. After that you must remove those settings manually from UF's conf files at etc/system/local. Restart UF's SplunkForwarder service and now you have those configuration managed by DS. IMHO: you should put also serverclass.conf into own app or even better to put in individual apps for each integration. Don't use GUI as it put those all under etc/system/local and then you have the same issue with DS than you have now in UF. I use this kind of structure: xxxx-app1-uf xxxx-app1-ds-sc xxxx-app1-ihf (if I need to use intermediate hf to manipulate and forward those events) Then I have separate xxxx-ds-base-app where is DS's base configurtions. With this concept I can manage sever integrations at same time independently each other. No need to do manual editing for one serverclass.conf file on DS. Also this can easily automated.

isoutamo

Exactly that way. You must do data onboarding in test/dev environment and ensure that your splunk shows only what you want! Normally this means that you must have 1st your own dev node etc. where you could create inputs, props and transforms. Then install those in some kind of systest or integration test environment where you could validate your configurations. If you don't have this kind of environments then you have some other issues in audit and governance cases than only PII data in production splunk.

isoutamo

Click > before event, it opens to you more information where you could see and click button "Event Actions", then select "Show Source". I your event is not too long it open it as it is in disk. There you can see e.g. those escape marks etc.

isoutamo

I think that acceptable_keys didn't work as your json event is in escaped text mode in disc. If you want to use event like json you must use INGEST_EVAL and json-functions. But I expecting that in that case you hit again a same limit to read that event in, convert it from escaped text to json and save again back to stream. The best option is do this masking before ingestion with some other tool than Splunk's props and transforms. Is it possible that you ask that source already mask it or can you use e.g. Ingest Action or Edge or Ingest Processor? Also one option is Cribl outside of Splunk world.

isoutamo

As already said those "deleted" events are still in disk and actually if you have access to server cli, you could make those readable again. Basically this means that your delete solutions is not 100% governance compliant. The only way to get rid of those is delete all data from those indexes and then reindex other events. Normal way in data onboarding is that you make everything 1st in test/dev environment and ensure that there is no PII or something other unwanted events. After that deploy those configurations into production. But if you still want to use delete, my suggestion is that use enough small time spans and run it several times instead of running all in one.

isoutamo

At least earlier DS version "lost" old entries after you reboot it. Then those will be back when they have 1st connected into it. I'm not sure if this is still valid for current DS which are using index to store client information. I have never looked it with those versions.

Posts	5159
Solutions	520
Karma Given	625
Karma Received	1571
Member Since	‎14-02-2017

Online Status	Offline
Date Last Visited	Monday

Wrong parameters on macOS and logd input?

Find enabled windows inputs by SPL query?

Why can't I change alert with REST- It change perm...

Why does Splunk_TA_aws hang on configure / inputs ...

Don't execute rest of commands if there is no even...

Indexing Ubisecure Ubilogin logs?

Fieldformat didn't work with foreach

Cannot migrate DBConnect 2.4.0 -> 3.2.0 on Splunk ...

How to check splunk multisite cluster status after...

Re: Can we use the concept of Splunk Shared Data M...

Re: Report shows missing fields after owner reassi...

Re: Can we install Botsv3 on splunk 10.0.0 in win...

Re: Indexer Rebalance Limits

Re: Does SmartStore work on a standalone Splunk in...

Re: Does SmartStore work on a standalone Splunk in...

Re: Splunk forwarding selected logs to third party...

Re: Splunk 7.1.2 Migration from Windows 2016 to Sp...

Re: Generating a report for previous week

Re: lastChanceIndex license usage?

Re: Queue Constantly being full

Re: Splunk Enterpise Security download Issue

Re: Splunk Enterpise Security download Issue

Re: Changes (regression?) to export activity audit...

Re: JSON Extraction of Values and Fields Logs Usin...

Re: Not getting Entra ID logs - Splunk Add-on for ...

Re: Splunk add-on : Optional upgrade

Re: Issue integrating Oracle DB with DB Connect Sp...

Re: remote instances are not showing up under moni...

Re: Custom app not overriding UF inputs.conf (WinE...

Re: Deleting historical logs

Re: Transforms Truncating - Setting SOURCE_KEY

Re: Transforms Truncating - Setting SOURCE_KEY

Re: Deleting historical logs

Re: Deployment Server: Duplicate clients after rei...

Are you a member of the Splunk Community?