The issue is due to the frozen bucket is place in the /opt/xyz_frozen/ directory outside the /opt/splunk directory. The splunk user does not have the permission to write to the /opt/xyz_frozen directory. Thus, not able to start the search head. Creating the frozen directories for the frozen bucket and giving the permission to the user have resolved the issue. mkdir -p /opt/xyz_frozen/XYZ_network chown splunk_user:splunk_group /opt/xyz_frozen/XYZ_network ... View more

On deleting multiple container, Please refer to the knowledge base for container deletion script: https://my.phantom.us/kb/80/ To check if the container exists or it has removed: Each of the Container has Container ID, so by referencing the container ID you should find whether it is still in database. 1) By URL https://Phantom_IP_Address/rest/container/Container_ID 2) By Rest API curl -H "Content-Type: application/json" -k -s -u username:password -X GET https://Phantom_IP_Address/rest/container/Container_ID ... View more

The system health here is basically the reflection of the phantom instance OS Memory, Load average and Disk Usage. Also the utilisation of the memory and CPU of the processes. There is no threshold where it is impacting phantom performance as long as the resource still available. The utilisation can be more than 90% and Phantom still working fine because the resources are still there. Having said that there is Linux kernel Out of Memory (OOM) on keeping the score of each process. This is more from the Operating System Linux. Thus if a certain process reach the score threshold, this process will be killed. Please refer to the following url for information on OOM: https://dev.to/rrampage/surviving-the-linux-oom-killer-2ki9 Phantom is an automated response platform. If it is operating at 70-80% of system capacity, it is to be expected that an event surge will buckle the VM. This is not best practice. Ideally, headroom for surge events should be planned for, and VM specs adjusted accordingly. Additionally, There is script that can be use to monitor the container status threshold using REST API: https://my.phantom.us/kb/47/ This would require user login. You would need to register to phantom on getting the login credentials. ... View more

1) Open a CMD via run as admin 2) Rebuild performance counters, run commands: C:\Windows\system32\lodctr.exe /R C:\Windows\SysWOW64\lodctr.exe /R 3) Now re-register the computer´s performance libraries with the following: C:\Windows\System32\wbem\winmgmt.exe /RESYNCPERF C:\Windows\SysWOW64\wbem\winmgmt.exe /RESYNCPERF After perform this, the repair the perfmon metrics are now coming in. Reference: https://www.admin-enclave.com/en/articles/windows/306-resolved-the-open-procedure-for-service-bits-in-dll-c-windows-system32-bitsperf-dll-failed-performance-data-for-this-service-will-not-be-available-the-first-four-bytes-dword-of-the-data-section-contains-the-error-code.html ... View more

The ES need to import custom apps (phantom) to Enterprise Security. It is believe this has not been setup. In order to set this up, here are the steps: 1. Go to Enterprise Security > General > App Imports Update > update_es 2. On the Application Regular Expression enter the following at the end: | (phantom) The Application Regular Expression will be: (appsbrowser)|(search)|([ST]A-.)|(Splunk_[ST]A_.)|(DA-ESS-.)|(Splunk_DA-ESS_.)|(phantom) 3. Save this 4. Once you save it, you need to enable the update_es. On the right hand side of the update_es , you will see the status, click on Enable 5. Now try to again with you Correlation Search, you will find phantom will be in the list of your Adaptive Response Action. 6. Once this is done you can create the Saved search, click on Save and Preview. For further information on ES App Imports Update: https://docs.splunk.com/Documentation/ES/4.7.1/Install/ImportCustomApps ... View more

1.) To list the non-internal indexes and non-internal indexes a. To list of all non-internal indexes: | eventcount summarize=false index=* | dedup index | fields index b. To list of all sourcetype within the non-internal indexes: | tstats count where index=* by index, sourcetype 2.) inputs.conf a. The configuration for scripts needs to be in formatted with script rather than monitor b. The script need to be located at the $SPLUNK_HOME/bin/scripts Example: [script:///opt/splunkforwarder/bin/scripts/client-stats.sh] 3.) The scripts need the environment setup and the jar file to run. Copy the setup env file and the jar file to the $SPLUNK_HOME/bin/scripts. Change the permission and ownership of the file accordingly. ... View more

There is Index db path collision check for summaryHomePath starting from version 7.2.X Possible affect to customers: Any customer that has not configured with summaryHomePath or just configured with volume referencing for any index, I.e homePath = volume:hot at downward 7.2.x version Symptom: Indexer won’t be able to start after upgrade Or bundle push for indexes.conf change without the required configuration. Error: Problem parsing indexes.conf: Cannot load IndexConfig: index= Path=$SPLUNK_HOME/INDEX_NAME/summary given as value of param=homePath collides with value of param2=summaryHomePath of index2= Reason: Every index is going to use same summary index path(location). To fix the issue: Adding summaryHomePath = volume:hot/$_index_name/summary to every index configuration. Please see our doc: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf summaryHomePath = It is recommended that you specify the path with the following syntax: summaryHomePath = $SPLUNK_DB/$_index_name/summary At runtime, Splunk expands "$_index_name" to the name of the index. For example, if the index name is "newindex", summaryHomePath becomes "$SPLUNK_DB/newindex/summary". May contain a volume reference (see volume section below) in place of $SPLUNK_DB. Volume reference must be used if data retention based on data size is desired. CAUTION: Path must be writable. If not specified, Splunk creates a directory 'summary' in the same location as homePath ... View more

Resolution is to reduce the size of maxVolumeDataSizeMB to lower value 90GB. maxVolumeDataSizeMB is not to be exactly the same measure of df/file system The Reason is that when calculating the size of bucket, splunk only add up the file size under the buckets. Directory is not counted. Suppose customer has a big number of buckets/directory, the margin between splunk and df measurement can be big. A normal directory size is 4K, if they have one million directory under the bucket directory, there would be 4GB gap. ... View more