Hi, I have a distributed on Prem Splunk Enterprise Deployment at 8.1.x. Splunk is running under Systemd. I recently noticed that the previous admin did not tune up the ulimits. I was wondering if anyone knew how to tune these settings based on individual host role/hw. For instance if I override with "systemctl edit Splunkd.service" command [Service] LimitNOFILE=65535 < tech support suggestion LimitNPROC=20480 < tech support suggestion LimitDATA=(80% total RAM?)< tech support suggestion LimitFSIZE=infinity < tech support suggestion TasksMax=20480 <mirrored LimitNPROC per tech support I have read the docs and seen the default suggestions for NOFILE and NPROC but how do you determine the other limits? Specifically NPROC and DATA? ref: >>> https://docs.splunk.com/Documentation/Splunk/9.0.2/Installation/Systemrequirements#Considerations_regarding_system-wide_resource_limits_on_.2Anix_systems Thank you! ... View more

Hi  I am not having much luck. I want to find all schedule reports and alerts that use a specific index (e.g. index=foo) or the name contains a keyword (e.g. fooBar). I tried the _audit index     index=_audit search = *"index=foo"* OR savedsearch_name=*fooBAR* provenance=scheduler | stats values(savedsearch_name)      I get some of the alerts (hopefully "provenance = scheduler" means it is scheduled) but I was looking for a better way, maybe with >>  |rest      | rest /servicesNS/-/search/saved/searches | table title,triggered_alert_count,search, cron_schedule,alert_type,alert_condition | rex field=search "index=(?<indexName>.[^\s]+)" | search indexName=foo     However I am not having much luck getting alerts that contain "index=foo" in the search field. Any advice appreciated. Thank you ... View more

I am having no luck listing users' memberships with in a group, using ldapsearch. I am not an AD LDAP expert, either. Lets say I have a domain called Foo, and an OU (group) called Bar, with 10 users.  Each user has additional memberships to other groups. I am looking to list the membership attr for each user. I am starting with  | ldapsearch domain=default search="(&(objectClass=user))"... but I don't know what to add. Thank you  ... View more

Hi, I am b/t a rock and a wall, looking for any suggestion to solved this.   I am using the URL ToolBox to dissect URI "ut_path" into fields separated by "/" characters. For instance >>>      index=foo sourcetype="bar" Requested_URI=* | lookup ut_parse_simple_lookup url AS Requested_URI | fields ut_* Requested_URI User_ID | table User_ID RequestUri ut_scheme, ut_netloc, ut_path, ut_query, ut_fragment, ut_params ut_path = /a1/f1/f2/f3/4/5 ut_path = /a1/f1/f2 ut_path = /a1/f1/f2/f3/f4 ut_path = /a1/f1/f2/f3       The "ut_path" field has different value paths of varying length, each section (like f1) needs to get extracted into a new field so that I can run stats on it.    Is there a way to auto-extract dynamically, or conditionally? Thank you! ... View more

After following, well verified steps as noted in > https://community.splunk.com/t5/Deployment-Architecture/How-to-move-the-SHC-deployer-to-another-host-Part-2/m-p/604671#M25839 I was not able to successfully connect and test a push from the new deployer to the shcluster members.  I received an error >>> Error while deploying apps to first member, aborting apps deployment to all members: Error while fetching apps baseline on target=https://host:8089: Non-200/201 status_code=401; {"messages":[{"type":"ERROR","text":"Unauthorized"}]} Here are my steps: 1. copied the contents of /opt/splunk/etc/shcluster from the old deployer to the new deployer /opt/splunk/etc/shcluster 2) configured the new deployer [shclustering] stanza with the info from the old deployer [shclustering] stanza in /opt/splunk/etc/system/local server.conf 3) Updated conf_deploy_fetch_url in server.conf on each of the shc members 4) restarted the new deployer and a rolling restart on the shc members 5) did a test apply bundle and then received an error unauthorized. I believe the issue could be with the pass4SymmKey (on the new deployer) not being the same as the pass4SymmKey on the SHC members. I did a ./splunk show-decrypt --value <key> from the old deployer [shclustering] pass4SymmKey = <key> shcluster_label = Company_shcluster1 I used the decrypted key as the key for the new deployer pass3SymmKey but ultimately I am not able to run a successful push. Is there a way to recover these keys? The previous admin did not save the original secrets used to setup the deployer. Any advice greatly appreciated. Thank you ... View more

Hi -  I am trying to get the Splunk App for AWS Security Dashboards working. Apparently the default index the app is using is "main".   I need to change this. I know I could change the index name by editing the xml but that would require a lot of changes. I am hoping someone knows where the central change location is located.   Thank you. ... View more

So I need to move a deployer to a dedicated host. I have a 3 member shc on version 8.1.3, all healthy. I have read a number of posts that give similar answers (to the answer I received for my original post), such as > "copy over the /opt/splunk/etc/shcluster to the new deployer" "configure the new deployer (to use the cluster's secret key and to set the SHC label), move the configuration bundle from the old deployer to the new deployer, and then point the cluster members to the new deployer" "migrate the shcluster folder structure and any shclustering stanza configurations you have on the deployer to the new deployer" " also break the SHC and rebuild with new deployer info"   While all these answers make sense, IDK exactly know what to reconfigure/change so I read > https://docs.splunk.com/Documentation/Splunk/8.1.3/DistSearch/BackuprestoreSHC And that takes you down the rabbit hole of backups and restores that don't entirely seem necessary, so I am wondering if anyone can verify the minimum changes that need to be made? OR if I should follow the above link's instructions.   As I understand it, I just need to do the following>>> 1) Build new Deployer ( new IP, new FQDN), install Splunk... 2) configure with the [shclustering] stanza in /opt/splunk/etc/system/local  server.conf [shclustering] pass4SymmKey =<secret> shcluster_label = <name> 3) on each SHC member,  edit the [shclustering] stanza [shclustering] conf_deploy_fetch_url = https://<newIP>:8089 make sure "pass4SymmKey" and "shcluster_label" is same as on new Deployer 4) copy over the /opt/splunk/etc/shcluster to the new deployer 5) Restart everything  Does that seem right?  I don't have the luxury of a dev environment to test...  Do I need to put shc members in detention or stop splunk on everything before I make the changes? Any advice is appreciated. Thank you   ... View more

Hi  I have been thru the docs but don't see any specific detail regarding system hardware needed for Specific Roles.... Let's say you have a 3 member SHC and a 10 member IDXC. IF you were to put the deployer role on a separate box, what would you need at a minimum? (For instance > maybe 8 Cores, 12G Ram, 1OOG Disk) And what would you need for the following roles if each and every one was on a dedicated VM? License Manager Monitoring Console Index Cluster Master Deployment Server I find the documentation is more specific for SHs and Indexers, but per documentation >>> “For detailed sizing and resource allocation recommendations, contact your Splunk account team”. I am up against a HW resource wall, and need to shuffle some things around.  So any advice or lessons learned on the "minimum hardware requirements topic" are greatly appreciated.   ... View more

Hi,  Unfortunately I inherited a Splunk deployment where the previous admin co-located multiple roles to one Splunk host.   The admin put Deployment Server, SHC Deployer, and Monitoring Console roles all on a single box (on prem).  In order to update the Deployment Server, Support told me I need to first remove the MC and Deployer roles.   I can move the MC no problem, but moving the SHC Deployer is causing some concern.   The deployer is set to the standard "merge_to_default" but I not sure how to copy over the files to the new deployer.  I know I need to make sure all the apps on the current get moved over, but what about the local settings created by the user?    Support says having DS and Deployer roles on same box is not supported, which I agree, but I am not getting any guidance from them. Any advice is greatly appreciated.    Thank you     ... View more

I am trying to monitor license warnings but I am not sure when the Splunk License Day ends and rolls over to the next day. Are all Splunk Licenses set to UTC by default? Please advise. Thank you ... View more