You are getting the error message as Digital Shadows app has dependency defined in digital_shadows.json. cat -n digital_shadows.json | more 1 { 2 "appid": "8cebdfd7-afe9-48eb-b840-2d98f860ed79", 3 "name": "Digital Shadows", 4 "description": "This app integrates with Digital Shadows SearchLight to ingest and investigate credentials found in data breaches", 5 "type": "information", 6 "main_module": "digital_shadows_connector.pyc", 7 "app_version": "1.0.1", 8 "min_phantom_version": "3.0.251", 9 "product_vendor": "Digital Shadows", 10 "product_name": "Digital Shadows SearchLight", 11 "product_version_regex": ".*", 12 "publisher": "Digital Shadows Ltd.", 13 "package_name": "phantom_digital_shadows", 14 "license": "Copyright (c) 2017 Digital Shadows Ltd.", 15 "pip_dependencies": { 16 "pypi": [ 17 {"module": "httplib2"} 18 ] 19 }, Problem: Your Phantom server is unable to connect to pypi.org and connection times out while installing the dependent python package. How to fix? 1. You can manually download and install the package and remove the below dependencies code from digital_shadows.json. Remove : line numbers starting from 15-19 15 "pip_dependencies": { 16 "pypi": [ 17 {"module": "httplib2"} 18 ] 19 }, And then try installing app via phenv python2.7 /opt/phantom/bin/compile_app.pyc -i. Or fix the connection issues ie see why pypi.org is not accessible. ... View more

As per documentation :https://docs.splunk.com/Documentation/Splunk/7.2.5/Data/MonitorWindowseventlogdata#Index_exported_event_log_.28.evt_or_.evtx.29_files : Files that have been exported from another machine do not work with the Splunk Web Upload feature. This is because those files contain information that is specific to the machine that generated them. Other machines won't be able to process the files in their unaltered form. So the upload feature is not working. 2.Constraints for monitoring Windows Event log files directly As a result of API and log channel processing constraints on Windows XP and Server 2003 systems, imported .evt files from those systems do not contain the "Message" field. This means that the contents of the "Message" field do not appear in your Splunk index. Splunk Enterprise on Windows XP and Windows Server 2003/2003 R2 cannot index .evtx files exported from systems running Windows Vista and later or Windows Server 2008/2008 R2 and later. Splunk Enterprise on Windows Vista and later and Server 2008/2008 R2 and later can index both .evt and .evtx files. If your .evt or .evtx file is not from a standard event log channel, you must make sure that any dynamic link library (DLL) files required by that channel are present on the computer on which you are indexing. Splunk Enterprise indexes an .evt or .evtx file in the primary locale/language of the computer that collects the file. You can check/find the constraints at https://docs.splunk.com/Documentation/Splunk/7.2.5/Data/MonitorWindowseventlogdata#Index_exported_event_log_.28.evt_or_.evtx.29_files. Detailed Steps on how to get monitor the exported evtx/windows event logs on other windows server : I tested the below steps in my lab environment and they work perfectly fine. I am able to export windows events logs and copy to another server and monitor it. I was able see the events getting indexed and I was able see parsing was working as expected. I installed two windows servers with same version ( OS Windows server2012) Source host on which I exported the events :ip-OACA0965 I installed Splunk software on second host : ip-0ACA0DBF and setup inputs.conf to monitor the exported files. On ip-0ACA0DBF : under updated $SPLUNK_HOME/etc/system/local/inputs.conf as below Stanza : Copied exported sourceevents.evtx from ip-OACA0965 to ip-0ACA0DBF [monitor://C:/sourceevents.evtx] queue = winparsing crcSalt = whitelist=.evtx$ I copied sourceevents.evtx to C:/ on ip-0ACA0DBF server. Restart Splunk service and I am able to see the exported windows event logs indexed and was able to search the data successfully. ... View more

As per documentation :https://docs.splunk.com/Documentation/Splunk/7.2.5/Data/MonitorWindowseventlogdata#Index_exported_event_log_.28.evt_or_.evtx.29_files : Files that have been exported from another machine do not work with the Splunk Web Upload feature. This is because those files contain information that is specific to the machine that generated them. Other machines won't be able to process the files in their unaltered form. So the upload feature is not working. 2.Constraints for monitoring Windows Event log files directly As a result of API and log channel processing constraints on Windows XP and Server 2003 systems, imported .evt files from those systems do not contain the "Message" field. This means that the contents of the "Message" field do not appear in your Splunk index. Splunk Enterprise on Windows XP and Windows Server 2003/2003 R2 cannot index .evtx files exported from systems running Windows Vista and later or Windows Server 2008/2008 R2 and later. Splunk Enterprise on Windows Vista and later and Server 2008/2008 R2 and later can index both .evt and .evtx files. If your .evt or .evtx file is not from a standard event log channel, you must make sure that any dynamic link library (DLL) files required by that channel are present on the computer on which you are indexing. Splunk Enterprise indexes an .evt or .evtx file in the primary locale/language of the computer that collects the file. You can check/find the constraints at https://docs.splunk.com/Documentation/Splunk/7.2.5/Data/MonitorWindowseventlogdata#Index_exported_event_log_.28.evt_or_.evtx.29_files. Detailed Steps on how to get monitor the exported evtx/windows event logs on other windows server : I tested the below steps in my lab environment and they work perfectly fine. I am able to export windows events logs and copy to another server and monitor it. I was able see the events getting indexed and I was able see parsing was working as expected. I installed two windows servers with same version ( OS Windows server2012) Source host on which I exported the events :ip-OACA0965 I installed Splunk software on second host : ip-0ACA0DBF and setup inputs.conf to monitor the exported files. On ip-0ACA0DBF : under updated $SPLUNK_HOME/etc/system/local/inputs.conf as below Stanza : Copied exported sourceevents.evtx from ip-OACA0965 to ip-0ACA0DBF [monitor://C:/sourceevents.evtx] queue = winparsing crcSalt = whitelist=.evtx$ I copied sourceevents.evtx to C:/ on ip-0ACA0DBF server. Restart Splunk service and I am able to see the exported windows event logs indexed and was able to search the data successfully. ... View more

Steps to Fix: We have to rebuild Performance Counters with LODCTR from an elevated command prompt. Launch Command Prompt as Administrator (right click Runs As Administrator). Drop into the C:\WINDOWS\System32 directory by typing CD\ then CD Windows\System32 To rebuild your resource counters type the following command: lodctr /r This will rebuild your counter values and may take a few moments so please be patient. If you wish to query the counters to make sure they were correctly set, use: lodctr /q This will give you something like the following: Performance Counter ID Queries [PERFLIB]: Base Index: 0x00000737 (1847) Last Counter Text ID: 0x000031D2 (12754) Last Help Text ID: 0x000031D3 (12755) [.NET CLR Data] Performance Counters (Enabled) DLL Name: %systemroot%\system32\netfxperf.dll Open Procedure: OpenPerformanceData Collect Procedure: CollectPerformanceData Close Procedure: ClosePerformanceData [.NET CLR Networking] Performance Counters (Enabled) DLL Name: %systemroot%\system32\netfxperf.dll Open Procedure: OpenPerformanceData Collect Procedure: CollectPerformanceData Close Procedure: ClosePerformanceData and etc 5.Now reopen the Performance Monitor and check it. You will see that the error was fixed. Note 1: In the query if you have provider name that is Disabled, Use this Cmdlet to enable it. lodctr /e: for example (think in above query Performance Counters is disable ) : lodctr /e:Performance Counters ... View more

In our case the Splunk instance on which Splunk was restarting frequently was Search head cluster member. We removed the Search head cluster member from node and we noticed that Splunk was starting with no issues and was stable As soon as we added the node back to cluster we started seeing that search head cluster member restarting Splunk instance. After removing and adding Search head member we noticed the below error in logs : splunkd.log:03-21-2019 12:52:21.488 +0800 ERROR DistributedBundleReplicationManager - HTTP response code 409 (HTTP/1.1 409 Conflict). Checksum mismatch: received copy of bundle="/opt/splunk/var/run/searchpeers/z.z.z.z.com-1553143774.bundle" has transferred_checksum=13796498114204145188 instead of checksum=12534329482337580946  As Splunk service was starting to frequently the above memory got rolled over earlier and only after removing and adding search head member back to cluster we noticed the above error and found that the root cause. To fix the issue temporarily we updated server.conf : 1. We updated server.conf : server.conf [shclustering] conf_deploy_fetch_mode = none  2. Restarted Splunk : Server came up fine. 3. We were able to add the server to cluster back successfully. Long term fixed the bundle that was causing this issue. ... View more

I tested in my lab and found that you get "Error connecting: SSL not configured on client" if you sslPassword is incorrect in server.conf. Lab testing details 1. One server as setup License master 2. One server as license slave configuration was working fine and license was able to connect License master node. 3. On license slave : I took a copy of server.conf and changed sslPassword to incorrect one. 4. After changing the sslPassword I started seeing below errors in splunkd.log ( similar errors noticed on your server) ../../../var/log/splunk/splunkd.log:02-27-2019 06:14:10.261 +0000 ERROR LMTracker - failed to ping master=https://y.y.y.y:8089, check server.conf for master_uri setting, this may lead to search being disabled for this machine err='Unable to connect to license master=https://y.y.y.y:8089 Error connecting: SSL not configured on client' ../../../var/log/splunk/splunkd.log:02-27-2019 06:14:11.240 +0000 ERROR LMTracker - failed to send rows, reason='Unable to connect to license master=https://y.y.y.y:8089 Error connecting: SSL not configured on client' ../../../var/log/splunk/splunkd.log:02-27-2019 06:15:11.241 +0000 ERROR LMTracker - failed to send rows, reason='Unable to connect to license master=https://y.y.y.y:8089 Error connecting: SSL not configured on client' 5. And similar errors noticed while checking the licensing details( Settings-> license details). Solution : . Make sure you have the correct sslPassword and update in $SPLU K_HOME/etc/system/local/server.conf ... View more

As per https://support.google.com/accounts/answer/6010255?hl=en :If an app or device doesn’t meet our security standards, Google will block anyone who tries to sign in from that app or device. Because these apps and devices are easier to break into, blocking them helps keep your account safe. Change your settings to allow less secure apps into your account. We don't recommend this option because it can make it easier for someone to break into your account. If you want to allow access anyway, follow these steps: To Fix: 1) I logged into gmail account. 2) Go to https://www.google.com/settings/security/lesssecureapps and Turn On this feature. 3) Go to https://accounts.google.com/DisplayUnlockCaptcha and click Continue. 4) I tried to send email from Splunk: index=_internal | head 1 | sendemail to="myemailid@gmail.com" format="html" server=smtp.gmail.com:587 use_tls=1 5)And I was able to see the emails send from Splunk to gmail successfully. Summary: Issue is external to Splunk, Google was blocking emails from Splunk. ... View more