I'm having trouble getting a new deployment client to connect to the DS. I can see connectivity is established, but the client keeps logging an error:     DC:DeploymentClient ... channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected       Looking at the splunkd_access log on the DS I can see the handshake message being recieved with a 401 by the DS     10.X.X.2 - - ... "POST /services/broker/connect/GUID/CLIENTNAME/guff/linux-x86_64/8089/9.0.2/GUID/universale_forwarder/CLIENTNAME HTTP/1.1" 401     I have plenty of Windows machines in the environment connecting successfully to this DS (also running on Windows). But this server and a few other Linux machines are not connecting. Any advice?   ... View more

Just putting this here for others who come across this problem since I got no results when I searched here. After upgrading to Splunk 9.0.1 and configuring an SSL cert for the kvstore I got this error on one of my two instances. On Windows, the kvstore relies on the server cert being in the Windows local machine certificate store. At startup it converts the supplied PEM (with embedded cert and password protected key) into a PFX which it then imports into the store. The error referenced in Subject is preceded by:   ERROR MongodRunner [9060 KVStoreConfigurationThread] - Command cmd="{CMD.EXE /C ( "C:\Program Files\Splunk\bin\openssl.exe" pkcs12 -inkey "C:\Program Files\Splunk\etc\auth\mycerts\splunkd.pem" -in "C:\Program Files\Splunk\etc\auth\mycerts\splunkd.pem" -passin pass:xxxx -export -out "C:\Program Files\Splunk\etc\auth\mycerts\splunkd.pem.pfx" -passout pass:xxxx )}" failed: exited with code 1. unable to load private key\r\n10460:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:.\crypto\evp\evp_enc.c:590:\r\n10460:error:0906A065:PEM routines:PEM_do_header:bad decrypt:.\crypto\pem\pem_lib.c:476:\r\n   I tried this command and it did not work. I tried the openssl command by itself in PowerShell and it DID work. The problem turned out to be one of the special character in the password I had set on my private key. I used open SSL to write out a new copy of the key with a different password with no special characters and lo and behold it worked. Just be careful to replace the new password in all locations it might be used (EG: Twice in server.conf and in inputs.conf on an indexer). The command for changing the password on your key is as follows:   .\openssl.exe rsa -aes265 -in mykey.key -out mynewkey.key -passin pass:oldpassword -passout pass:newpassword   Just make sure you are aware of what the special character in your old password might be and use PowerShell not the command prompt ... View more

I have been using tstats to get event counts by day per sourcetype, but when I search for events in some of the identified sourcetypes search returns no results. I am a Splunk admin and have access to All Indexes. Here are the searches I have run:   | tstats count where index=myindex groupby sourcetype,_time   One of the sourcetype returned was novell_groupwise (which was quite a surprise to me), but when I search   index=myindex sourcetype=novell_groupwise   on a day that tstats indicated there was events on, nothing is returned. Can anyone explain this discrepancy? ... View more

I've got Splunk 8.1.2 running on Windows Server 2019. I've installed the latest version of DB Connect after after configuring the the General page it always says "Cannot communicate with task server, please check your settings" There are no errors in the dbconnect logs (index=_internal sourcetype=dbx*) But the output of one line might be interesting it says:     [main] INFO org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.3.2) 'QuartzScheduler' with instanceId 'NON_CLUSTERED' Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally. NOT STARTED. Currently in standby mode. Number of jobs executed: 0 Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 32 threads. Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.     I am not sure if this is normal or not. The query server has no problem running on port 9999 and there is nothing else listening on port 9998 I have tried all of the things I could find in posts on this forum and in the documentation.  I can't find anywhere to go with this. ... View more

Hi, I'm trying to use Splunk to provide a report on servers where a service is absent. So I have one event per service per host. So if there are 10 services running on 1 host, that is 10 different events. My idea was to do a search which combines all of the services on a host into a single field and then search where that field doesn't contain the value I am looking for, but I have no idea how to achieve this. Here are a couple of sample raw events from the same host   20200702162757.583428 Caption=Remote Desktop Configuration Description=Remote Desktop Configuration service (RDCS) is responsible for all Remote Desktop Services and Remote Desktop related configuration and session maintenance activities that require SYSTEM context. These include per-session temporary folders, RD themes, and RD certificates. Name=SessionEnv PathName=C:\WINDOWS\System32\svchost.exe -k netsvcs StartMode=Manual StartName=localSystem State=Running Status=OK wmi_type=Service 20200702162757.583428 Caption=Symantec Endpoint Protection WSC Service Description=Allows Symantec Endpoint Protection to report status to the Windows Security Center. Name=sepWscSvc PathName="C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\14.3.558.0000.105\Bin64\sepWscSvc64.exe" StartMode=Auto StartName=LocalSystem State=Running Status=OK wmi_type=Service    Assume I want to return hosts where the second service entry is absent. ... View more