Hi According to the Splunk Docs from version 9.1: "the installer creates a virtual account as a "least privileged" user called splunkfwd" After an upgrade to version 9.1.2 I am having trouble with the UF autostarting. Looking at Windows Event Logs I can see the following error: Which suggests the account is actually "SplunkForwarder" not "splunkfwd" When I check the Windows Service Log On user I also see the user "SplunkForwarder":   And "SplunkForwarder" is also the only Splunk related user I can see when I run the following command to list all users: get-service | foreach {Write-Host NT Service\$($_.Name)}   Can someone confirm that the Doc is incorrect and the virtual account created is in fact SplunkForwarder? Or is "splunkfwd" created somewhere else?   Thanks   ... View more

Hi, I have noticed over the last 4 days I had an increased number of Search Bundle replication errors: 12-21-2023 09:50:12.604 +0000 WARN ConfReplicationThread [9209 ConfReplicationThread] - Error pushing configurations to captain=https://searchHeadCaptain:8089, consecutiveErrors=1 msg="Error in acceptPush: Non-200 status_code=400: ConfReplicationException: Cannot accept push with outdated_baseline_op_id=16ed9160640170315673324237791a4cfe256d59; current_baseline_op_id=cd93950208af34df00957e721b87128d3629d2d1" These occur in groups every 4 hours. I have also seen CPU spikes on the Search Heads that started occuring at the same time and also every 4 hours. Further investigation has shown that the following events from conf.log have also been occuring at the same time every 4 hours { [-]    component: ConfOp    data: { [-]      applied_at: 1703264397      asset_id: 220d8bbce6d790850cda3980c5784c62b1a9f9ff      asset_uri: [ [+]      ]      from_repo: https://searchHeadCaptain:8089      op_id: 102aa206f930da5eef0d47163b354c61254566c5      optype: 2      optype_desc: WRITE_STANZA      payload: { [-]        alias: Risk        metadata: { [-]          permissions: { [-]          }        }        value: ***TRANSIENT***://6613      }      payload_extra: ***ALLOW_SKIP_ON_WRITE***      status: applied      task: pullFrom      to_repo: https://searchHeadPeer.com:8089      to_repo_change_count: 20214    }    datetime: 12-22-2023 16:59:57.097 +0000    log_level: INFO } Does anyone know what these events mean and how I can find out what is causing them? Bundle replication errors:   conf.log events:   CPU spikes:   ... View more

Hi, I am runing Splunk Stream to collect DNS data from Domain Controllers. On some of the busy DCs the Splunk_TA_stream is generating lots of the following errors:     ERROR [9412] (SplunkSenderModularInput.cpp:435) stream.SplunkSenderModularInput - Event queue overflow; dropping 10001 events     Looking at the Splunk Stream Admin-Network Metrics dashboard these seem to occur at the same the Active Network Flows seem to be hitting a limit: I would like to increase the number of network flows allowed in an attempt to stop the event queue overflows. Looking at the documentation I can see 2 configurations that seem relevant: maxTcpSessionCount = <integer> * Defines maximum number of concurrent TCP/UDP flows per processing thread. processingThreads = <integer> * Defines number of threads to use for processing network traffic. Questions: 1) What is the default for maxTcpSessionCount and processingThreads? 2) Would parameter would it be better to increase? Also are these the correct parameters to be looking to tune with the errors I am getting. If not what should I look at? ... View more

Hi, I am running a Search Head Cluster with 7 search heads on Splunk 8.2.9. 2 of the search heads are generating the following error messages at ~5 second intervals for a period of time before stopping:     ERROR DigestProcessor [38271 TcpChannelThread] - Failed signature match ERROR HTTPAuthManager [38271 TcpChannelThread] - Failed to verify HMAC signature, uri: /services/shcluster/member/consensus/pseudoid/raft_request_vote?output_mode=json     The search head cluster is otherwise running as expected as far as I can tell. The search heads that are producing these errors are the only 2 that have been elected as captain in the last 30 days from examining the logs. There are no preferred captain or similar configurations set. I have checked the [shclustering] pass4SymmKey values on each search head. They are all configured to the same value although use different Splunk Secrets to encrypt. I am not sure when the errors first started appearing so can't link this to a specific upgrade on configuration change unfortunately. The thread_id values seem to stay around for between 10-30 minutes. Sometimes 2 thread_ids will be active at once, sometimes none are active for a period. When looking at other logs for a particular thread_id around the same time period (at info logging level) I can't find see anything that adds any more cluses to what is causing the errors.   ... View more

Following the best practices for removing an LDAP user I am at the stage where I want to remove  the $HOME/splunk/etc/users/$userid folder. I have tried deleting from the cluster captain but this does not seem to be distributed across the cluster. Do I need to delete from each member individually?   Thanks. ... View more

Hi, I'm trying to get the Splunk App for Web Analytics to work and am having trouble with the Web data model acceleration. Currently when viewing the Analytics Center and Audience tabs I get 'No results found'. After some investigating this is because the Web.eventtype=pageview part of the SPL query that runs is not returning any results (when I examine the search and remove this part results are returned). When I look at the events in the datamodel via Pivot there are no events where this is true, in fact the Web.eventtype field is empty for all events in the data model. Note: this is when the data model is accelerated. When I turn off acceleration I can see that the field 'eventtype' in the data model is created by an auto-extracted field. I can then go to the edit page of the eventtype field and preview the output of the extraction, then the eventtype field is populated as expected. I can check this through a Pivot again and the values for the eventtype field values are still present. I can't work out why the field values disappear when enabling data model acceleration. I'm only working on a small data set currently for testing (<10Mb) and have given 2+ hours for the data model acceleration to be built. I have also generated the lookup tables required by the App a number of time, so that is not the issue. I am using v2.2.2 of the App and Splunk v8.0.2. I believe I have the App set up correctly as the Real Time tab is showing data. I also breifly was able to view data on the Analytics Center and Audience tab yesterday but can't work out what changes caused this to occur! Edit: When I runthe Data Model Audit I get the following errors on the Top Accelerations visualizations: [map]: Failed to fetch REST endpoint uri=https://127.0.0.1:8089/servicesNS/nobody/SplunkAppForWebAnalytics/admin/summarization/tstats:DM_SplunkAppForWebAnalytics_myWeb?count=0 from server https://127.0.0.1:8089. Check that the URI path provided exists in the REST API. ... View more