The issue is due to the frozen bucket is place in the /opt/xyz_frozen/ directory outside the /opt/splunk directory. The splunk user does not have the permission to write to the /opt/xyz_frozen directory. Thus, not able to start the search head. Creating the frozen directories for the frozen bucket and giving the permission to the user have resolved the issue. mkdir -p /opt/xyz_frozen/XYZ_network chown splunk_user:splunk_group /opt/xyz_frozen/XYZ_network ... View more

On deleting multiple container, Please refer to the knowledge base for container deletion script: https://my.phantom.us/kb/80/ To check if the container exists or it has removed: Each of the Container has Container ID, so by referencing the container ID you should find whether it is still in database. 1) By URL https://Phantom_IP_Address/rest/container/Container_ID 2) By Rest API curl -H "Content-Type: application/json" -k -s -u username:password -X GET https://Phantom_IP_Address/rest/container/Container_ID ... View more

The system health here is basically the reflection of the phantom instance OS Memory, Load average and Disk Usage. Also the utilisation of the memory and CPU of the processes. There is no threshold where it is impacting phantom performance as long as the resource still available. The utilisation can be more than 90% and Phantom still working fine because the resources are still there. Having said that there is Linux kernel Out of Memory (OOM) on keeping the score of each process. This is more from the Operating System Linux. Thus if a certain process reach the score threshold, this process will be killed. Please refer to the following url for information on OOM: https://dev.to/rrampage/surviving-the-linux-oom-killer-2ki9 Phantom is an automated response platform. If it is operating at 70-80% of system capacity, it is to be expected that an event surge will buckle the VM. This is not best practice. Ideally, headroom for surge events should be planned for, and VM specs adjusted accordingly. Additionally, There is script that can be use to monitor the container status threshold using REST API: https://my.phantom.us/kb/47/ This would require user login. You would need to register to phantom on getting the login credentials. ... View more

1) Open a CMD via run as admin 2) Rebuild performance counters, run commands: C:\Windows\system32\lodctr.exe /R C:\Windows\SysWOW64\lodctr.exe /R 3) Now re-register the computer´s performance libraries with the following: C:\Windows\System32\wbem\winmgmt.exe /RESYNCPERF C:\Windows\SysWOW64\wbem\winmgmt.exe /RESYNCPERF After perform this, the repair the perfmon metrics are now coming in. Reference: https://www.admin-enclave.com/en/articles/windows/306-resolved-the-open-procedure-for-service-bits-in-dll-c-windows-system32-bitsperf-dll-failed-performance-data-for-this-service-will-not-be-available-the-first-four-bytes-dword-of-the-data-section-contains-the-error-code.html ... View more

The ES need to import custom apps (phantom) to Enterprise Security. It is believe this has not been setup. In order to set this up, here are the steps: 1. Go to Enterprise Security > General > App Imports Update > update_es 2. On the Application Regular Expression enter the following at the end: | (phantom) The Application Regular Expression will be: (appsbrowser)|(search)|([ST]A-.)|(Splunk_[ST]A_.)|(DA-ESS-.)|(Splunk_DA-ESS_.)|(phantom) 3. Save this 4. Once you save it, you need to enable the update_es. On the right hand side of the update_es , you will see the status, click on Enable 5. Now try to again with you Correlation Search, you will find phantom will be in the list of your Adaptive Response Action. 6. Once this is done you can create the Saved search, click on Save and Preview. For further information on ES App Imports Update: https://docs.splunk.com/Documentation/ES/4.7.1/Install/ImportCustomApps ... View more

1.) To list the non-internal indexes and non-internal indexes a. To list of all non-internal indexes: | eventcount summarize=false index=* | dedup index | fields index b. To list of all sourcetype within the non-internal indexes: | tstats count where index=* by index, sourcetype 2.) inputs.conf a. The configuration for scripts needs to be in formatted with script rather than monitor b. The script need to be located at the $SPLUNK_HOME/bin/scripts Example: [script:///opt/splunkforwarder/bin/scripts/client-stats.sh] 3.) The scripts need the environment setup and the jar file to run. Copy the setup env file and the jar file to the $SPLUNK_HOME/bin/scripts. Change the permission and ownership of the file accordingly. ... View more

There is Index db path collision check for summaryHomePath starting from version 7.2.X Possible affect to customers: Any customer that has not configured with summaryHomePath or just configured with volume referencing for any index, I.e homePath = volume:hot at downward 7.2.x version Symptom: Indexer won’t be able to start after upgrade Or bundle push for indexes.conf change without the required configuration. Error: Problem parsing indexes.conf: Cannot load IndexConfig: index= Path=$SPLUNK_HOME/INDEX_NAME/summary given as value of param=homePath collides with value of param2=summaryHomePath of index2= Reason: Every index is going to use same summary index path(location). To fix the issue: Adding summaryHomePath = volume:hot/$_index_name/summary to every index configuration. Please see our doc: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf summaryHomePath = It is recommended that you specify the path with the following syntax: summaryHomePath = $SPLUNK_DB/$_index_name/summary At runtime, Splunk expands "$_index_name" to the name of the index. For example, if the index name is "newindex", summaryHomePath becomes "$SPLUNK_DB/newindex/summary". May contain a volume reference (see volume section below) in place of $SPLUNK_DB. Volume reference must be used if data retention based on data size is desired. CAUTION: Path must be writable. If not specified, Splunk creates a directory 'summary' in the same location as homePath ... View more

Resolution is to reduce the size of maxVolumeDataSizeMB to lower value 90GB. maxVolumeDataSizeMB is not to be exactly the same measure of df/file system The Reason is that when calculating the size of bucket, splunk only add up the file size under the buckets. Directory is not counted. Suppose customer has a big number of buckets/directory, the margin between splunk and df measurement can be big. A normal directory size is 4K, if they have one million directory under the bucket directory, there would be 4GB gap. ... View more

The _raw field is not being saved after submitting the save button. This is due to the _raw field is not part of the Phantom Common Event Format (CEF). _raw field is intentionally not saved by design. There is workaround by performing the following: After extracting the _raw field, enter the CEF field as "message". This will put the _raw data into message field in Phantom. This will allow you to save the _raw field. The _raw data will be in the message field in Phantom. ... View more

The issue is due to there was an upgrade of Qualys from version 8.15 to version 8.16 Cause: The cause was a Qualys TA API that was used to pull the qualys_kb data, there was a new field included by Qualys in the later version 8.16. The parsing script cannot handle it, then the system was unable to download the data. Recommendation: To add the new field in the parsing script (kbpopulator.py). Then, the data can be downloaded and the lookup can start working again. Within the kbpopulator.py file, change the following line to include the new field CVSS_VECTOR_STRING: File location: /opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/qualysModule/splunkpopulator/kbpopulator.py Modify the following line: CVS_HEADER_COLUMNS = ["QID", "SEVERITY"] + QID_EXTRA_FIELDS_TO_LOG + ["CVSS_BASE", "CVSS_TEMPORAL", "CVSS_VECTOR_STRING", "CVE", "VENDOR_REFERENCE"] def init(self, logger=None): Detail Log After the upgrade of Qualys Web version 8.16, there was message "Exception while parsing. dict contains fields not in fieldnames: 'CVSS_VECTOR_STRING'" causing the API could not be loaded. Before the upgrade of Qualys Web version 8.15 2019-01-05T07:30:11Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform - Start qualys TA ... 2019-01-05T07:30:13Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform - Found QWEB_VERSION=8.15 2019-01-05T07:30:13Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform - Running for qualys://knowledge_base 2019-01-05T07:30:13Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform - knowledge_base cron/duration: 30 * * * * 2019-01-05T07:30:13Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform - Current time (2019-01-05 07:30:00) matches cron format (30 * * * *) defined for knowledge_base. Running now. ... 2019-01-05T07:36:38Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Update lookup file: /opt/splunk/etc/apps/TA-QualysCloudPlatform/lookups/qualys_kb.csv with 41069 QIDs 2019-01-05T07:36:39Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Updated lookup file: /opt/splunk/etc/apps/TA-QualysCloudPlatform/lookups/qualys_kb.csv with 41069 QIDs 2019-01-05T07:36:39Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Parsed 41069 knowledgebase entry. Logged=0 2019-01-05T07:36:39Z PID=27895 [MainThread] DEBUG: TA-QualysCloudPlatform [knowledge_base] - Done with parsing, returning. 2019-01-05T07:36:39Z PID=27895 [MainThread] DEBUG: TA-QualysCloudPlatform [knowledge_base] - Removing tmp file /opt/splunk/etc/apps/TA-QualysCloudPlatform/bin/../tmp/kb_2019-01-05-30-11_MainThread_27895_batch_1.xml 2019-01-05T07:36:39Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Done logging knowledgebase 2019-01-05T07:36:39Z PID=27895 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - End qualys TA After the upgrade to Qualys Web version 8.16 2019-01-05T14:30:08Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform - Start qualys TA ... 2019-01-05T14:30:10Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform - Found QWEB_VERSION=8.16 2019-01-05T14:30:10Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform - Running for qualys://knowledge_base 2019-01-05T14:30:10Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform - knowledge_base cron/duration: 30 * * * * 2019-01-05T14:30:10Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform - Current time (2019-01-05 14:30:00) matches cron format (30 * * * *) defined for knowledge_base. Running now. ... 2019-01-05T14:35:59Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Update lookup file: /opt/splunk/etc/apps/TA-QualysCloudPlatform/lookups/qualys_kb.csv with 41069 QIDs 2019-01-05T14:35:59Z PID=61781 [MainThread] DEBUG: TA-QualysCloudPlatform [knowledge_base] - Exception while parsing. dict contains fields not in fieldnames: 'CVSS_VECTOR_STRING' :: Traceback (most recent call last): 2019-01-05T14:35:59Z PID=61781 [MainThread] ERROR: TA-QualysCloudPlatform [knowledge_base] - could not load API response 2019-01-05T14:35:59Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - Done logging knowledgebase 2019-01-05T14:35:59Z PID=61781 [MainThread] INFO: TA-QualysCloudPlatform [knowledge_base] - End qualys TA ... View more