We are having issus with Data models from Splunk_SA_CIM running for a very long time (hitting the limit) and causing out of memory (OOM) issues on our indexers. We have got brand new physical servers with 128 GB RAM and 48 Cores. The Enterprise security search head cluster has data models enabled which are both running on old and new hardware. Though we are getting OOM on new hardware and every run hits our 30+ min limit. Example on configuration for auth DMA: allow_old_summaries = true allow_skew = 5% backfill_time = -1d cron_schedule = */5 * * * * earliest_time = -6mon hunk.compression_codec = - hunk.dfs_block_size = 0 hunk.file_format = - manual_rebuilds = true max_concurrent = 1 max_time = 1800 Any tips on troubleshooting data models running for a very long time and causing out of memory (OOM)? Thanks! ... View more

Hi, Today we have a standalone enterprise security (ES) search head which we would like to migrate to a new ES search head cluster (and keep all the knowledge objects). Is there instructions on how to turn a standalone ES search head into a ES search head cluster? Thanks, Sebastian ... View more

Hi, Every time I apply shcluster bundle the deployer pushes all apps in /opt/splunk/etc/shcluster/apps to the SHC members, even if there hasen't been any modification the the app. I can see that the checksum is the same on the deployer and the SHC members but some how the deployer still pushes the app. We are using push mode merge_to_default and are on Splunk version 9.0.1. Every apply shcluster bundle takes several hours. Any ideas? ... View more

Hi, We have upgraded from version 8.1.6 to version 9.0.1 recently and have discovered a new problem not seen before. Each time I push the apps from the deployer to the SHC all apps get pushed regardless if they are old, modified or new. This results in each apply shcluster bundle takes 5-6 hours to complete. Before the upgrade a push took a couple of minutes. Each push also generates a default.old.<DATE> for all apps. Is there a way to remove this behaviour? We prefer not to have a lot of default.old.<DATA> files and that the pushes become much faster. Thanks! ... View more

Hello, What is the procedure for migrating a deployer to a new server? We are running on Linux and are on version 8.1.6 on deployer and search head cluster and we cannot reuse IP-addresses or hostnames. I have read https://docs.splunk.com/Documentation/Splunk/8.1.6/DistSearch/BackuprestoreSHC but it's not crystal clear since we are only migrating the deployer and the search head cluster will remain the same. Thanks! ... View more

Hello, I recently upgraded our deployer/deployment server from 8.1.6 to version 9.0 and when I try to push configuration to our search head cluster i get an error that I have not seen before: [splunk@aa130XXXXX bin]$ ./splunk apply shcluster-bundle -target https://aa130XXXXX:8089  Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.  Please refer to the documentation for the details. Do you wish to continue? [y/n]: y WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. Your session is invalid.  Please login. Splunk username: XXXXX Password:  Error in pre-deploy check, uri=https://aa130XXXXX:8089/services/shcluster/captain/kvstore-upgrade/status, status=401, error=No error Our search head cluster is still on version 8.1.6 Thanks! ... View more

Hello, When I run the splunk apply cluster-bundle command it seems to create a bundle for all apps in /$SPLUNK_HOME/etc/shcluster/apps to /$SPLUNK_HOME/var/run/splunk/deploy/apps. The last modified date on all app catalogs in /$SPLUNK_HOME/var/run/splunk/deploy/apps is always the date of when I last ran the apply shcluster bundle command. We have merge_to_default set, could that have something to do with all apps being pushed to the search head cluster regardless if there has been any change in the apps? How can I troubleshoot what the problem could be? If the apply command finishes it can take up to hours but usually it's something that times out. ... View more

Hello, In which direction must firewall openings be configured for indexers and search heads to be able to communicate with the license manager and fetch license etc.? Is it only  <search_head/indexer> --> License manager (on port 8089) or both ways, so <search_head/indexer> --> <License_manager> AND  <License_manager> --> <search_head/indexer> (on port 8089). I have seen network topologies saying different things regarding this. Is one way enough or do we need both ways on port 8089? Thank you! ... View more

Hello, I would like to know if it is safe to delete below on all of our Splunk hosts: /opt/splunk/var/run/searchpeers/<hostname>-1633305600/apps/splunk_archiver/java-bin/jars/vendors/spark/3.0.1/lib/ Similar files exist on a lot of our Splunk hosts and we get notifications daily about them because of log4j. So is it safe to delete the above path and similar? It is just replications right? Thanks in advance! ... View more

Hello, We have a clustered environment which collects 2000gb+/day with indexes.conf settings below and the rest of the settings is default. When does the frozenTimePeriodInSecs starts its count? Is it when the data is in the hot, warm or cold buckets? When will the buckets roll from hot to warm, and from warm to frozen in my case? Is it after 90 days since MaxHotSpanSecs default is 90 days? What is the approximate retention time for data with this config? And the maxWarmDBCount = 4294967295 seems really high in this case. See config below: [index_name] homePath = volume:hot_warm/index_name/main/db coldPath = volume:cold/index_name/main/colddb thawedPath = /opt/splunk/indexes/index_name/main/thaweddb maxWarmDBCount = 4294967295 frozenTimePeriodInSecs = 31104000 maxDataSize = auto_high_volume maxTotalDataSizeMB = 4294967295 repFactor = auto   Thanks in advance! ... View more