Hi @earl-b , To achieve redundant cluster manager synchronization in a Splunk environment hosted on a Linux server, you can utilize the rsync utility. This tool will assist in syncing the primary cluster master's configuration to the standby cluster. To manually run the synchronization, execute the following rsync command on the primary cluster master: "rsync -avz --delete /path/to/source user@target_node_ip:/path/to/destination" For automated configuration synchronization, you can employ the crontab utility. This allows you to schedule the rsync command to run at regular intervals, ensuring continuous synchronization between the clusters.   ... View more

@FQzy   You can check the _internal logs to find the specific error related to "failed to create input" in the app by using the following search query: index=_internal source=*cisco*. You can also filter the logs by setting the log level to "error." For troubleshooting any add-ons, refer to the "Troubleshoot Add-ons" document available in the Splunk Documentation. Troubleshoot add-ons - Splunk Documentation. You can provide internal error to developer team for future investigation. Also, @PickleRick response was not generated using any AI.  ... View more

The issue you're facing with index clustering, where one indexer peer shows an `addingbatch` status and fluctuates between `up` and `batchadding`, while the other peer shows `up` and then goes to `pending` status, suggests potential problems with data replication, network connectivity, or resource allocation. ### Possible Causes and Solutions: 1. **Data Replication Lag or Bottlenecks**: - **Cause**: The `addingbatch` status indicates that the peer is adding data from the replication queue to the index, but it is either delayed or encountering issues. This can occur if there is a backlog in the replication queue or if the peer is unable to process the data quickly enough. - **Solution**: - Check for any network latency or packet loss between the indexer peers. Ensure there is sufficient bandwidth for replication traffic. - Verify if the indexers have adequate disk I/O performance. If the disks are slow or under heavy load, consider upgrading the storage or optimizing disk usage. 2. **Connectivity Issues Between Peers**: - **Cause**: The fluctuating statuses (`up` to `batchadding` or `pending`) could indicate intermittent network connectivity issues between the indexer peers or between the indexers and the cluster master. - **Solution**: - Review the network configuration and ensure that all indexer peers and the cluster master are correctly configured to communicate with each other. - Check the Splunk internal logs (`index=_internal`) for any network-related errors or warnings (`source=*splunkd.log`). 3. **Cluster Master Configuration or Load Issues**: - **Cause**: The cluster master may be overwhelmed or improperly configured, leading to inconsistent status updates for the peers. - **Solution**: - Verify the cluster master’s health and ensure it is not overloaded. - Review the cluster master's logs for any errors or configuration issues that might be causing delays in managing the peer status. 4. **Resource Constraints on Indexer Peers**: - **Cause**: The indexers might be under-resourced (CPU, memory, or disk space), causing them to be slow in processing incoming data or managing replication. - **Solution**: - Check the hardware resources (CPU, RAM, disk space) on each indexer. Ensure they meet the requirements for the volume of data being handled. - Increase the allocated resources or optimize the current configuration for better performance. 5. **Splunk Version Compatibility or Bugs**: - **Cause**: There may be bugs or version compatibility issues if different versions of Splunk are running on the cluster master and indexer peers. - **Solution**: - Make sure that all instances (cluster master and indexer peers) are running compatible versions of Splunk. - Review the [Splunk Release Notes](https://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes) for any known issues or bugs that might match your problem. 6. **Configuration Issues**: - **Cause**: Misconfiguration in the `indexes.conf` or other related files may cause replication or status reporting issues. - **Solution**: - Review your `indexes.conf`, `server.conf`, and `inputs.conf` files for any configuration errors. Ensure that all settings are aligned with best practices for index clustering. ### Next Steps: 1. **Log Analysis**: - Review the `_internal` logs (`splunkd.log`) on all affected peers and the cluster master. Look for errors, warnings, or messages related to clustering or replication. 2. **Network Diagnostics**: - Run network diagnostics to ensure there are no connectivity issues between indexer peers or between the peers and the cluster master. 4. contacting Splunk Support for further assistance. By systematically checking these areas, you can identify the root cause and apply the appropriate solution to stabilize the indexer peers in your Splunk cluster. ... View more

Hi @FQzy  To set up the Cisco Security Cloud app in Splunk, you can find detailed guidance and documentation on the (https://splunkbase.splunk.com/app/7404). ### Key Steps for Setup: 1. Download and Install: You can download the Cisco Security Cloud app from Splunkbase. Make sure to follow the specific instructions for installation, which include compatibility checks and required add-ons. 2. Configure Data Inputs: The app requires the configuration of several data inputs based on your Cisco security products (e.g., Firewalls, Intrusion Protection, Web Security, etc.). The documentation provides step-by-step guidance for each type. 3. Troubleshooting Common Errors: For issues like "failed to create an input," ensure that all prerequisites (like appropriate permissions and network settings) are met. You may need to consult the app's [Splunkbase page](https://splunkbase.splunk.com/app/5558) for specific troubleshooting tips. If you run into errors or specific issues during setup, it might also be helpful to check the community discussions and resources available on the Splunk website. For your reference you can refer this document: https://developer.cisco.com/docs/cloud-security/cloud-security-app-for-splunk/#cloud-security-app-for-splunk ... View more

  ### Updated Solution Using Internal Logs Here's an alternative method to monitor your daily license usage without requiring the "dispatch_rest_to_indexers" capability: 1. Search Query for License Usage: Use the following search query to fetch the license usage information from the internal logs: index=_internal source=*license_usage.log* type="RolloverSummary" | eval GB_used=round(b/1024/1024/1024, 2) | eval license_usage_percentage = (GB_used / 250) * 100 | stats latest(GB_used) as GB_used latest(license_usage_percentage) as license_usage_percentage | where license_usage_percentage > 70 | table GB_used, license_usage_percentage This query: - Fetches the license usage data from the internal license usage logs. - Converts the bytes used (`b`) to gigabytes (GB). - Calculates the percentage of license usage based on your 250 GB daily limit. 2. Create Alerts Based on Usage Percentage: You need to create three separate alerts based on different license usage thresholds. - Alert for 70% Usage: - Go to Search & Reporting in Splunk Cloud. - Enter the updated query in the search bar. - Click on Save As and select Alert. - Name the alert (e.g., "License Usage 70% Alert"). - Set the trigger condition to: license_usage_percentage > 70 - Choose the alert type as "Scheduled" and configure it to run once per day or at your preferred schedule. - Set the alert actions (e.g., email notification, webhook). - Alert for 80% Usage: - Repeat the steps above to create another alert. - Set the trigger condition to: license_usage_percentage > 80 - Alert for 90% Usage: - Follow the same process to create the final alert. - Set the trigger condition to: license_usage_percentage > 90 3. Configure Alert Notifications: - For each alert, set up the notification actions under Alert Actions (e.g., send an email, trigger a webhook). 4. Test Your Alerts: - Manually run the search and ensure it produces the expected results. - Adjust thresholds or conditions to test if the alerts trigger correctly. ... View more

you can run below query in your CMC. | rest splunk_server_group=* /services/licenser/pools | eval total_quota_gb = round(your_quota_field / (1024 * 1024 * 1024), 2) | eval used_gb = round(your_used_field / (1024 * 1024 * 1024), 2) | eval usage_percentage = round((used_gb / total_quota_gb) * 100, 2) | table splunk_server, total_quota_gb, used_gb, usage_percentage | eval alert_level = case( usage_percentage > 90, "Critical", usage_percentage >= 80, "High", usage_percentage >= 70, "Medium", true(), "Normal" ) | eval alert_message = case( usage_percentage > 90, "License usage has crossed critical threshold at " . usage_percentage . "%. Immediate attention required!", usage_percentage >= 80, "License usage has reached " . usage_percentage . "%. Please take immediate action.", usage_percentage >= 70, "License usage has reached " . usage_percentage . "%. Please take action.", true(), "License usage is within normal range." ) | where usage_percentage > 70 | table splunk_server, total_quota_gb, used_gb, usage_percentage, alert_level, alert_message   Make sure to replace your_quota_field & your_used_field with the correct field name representing the license quota in your Splunk Cloud environment. ... View more

Hi @anandhalagaras1, you can try this query.  rest /services/licenser/pools | eval total_quota_gb = round(usage_quota / (1024 * 1024 * 1024), 2) | eval used_gb = round(usage_used / (1024 * 1024 * 1024), 2) | eval usage_percentage = round((used_gb / total_quota_gb) * 100, 2) | table total_quota_gb, used_gb, usage_percentage | where usage_percentage >= 70 AND usage_percentage < 80 | eval alert_level = "70%-79%" | eval alert_message = "License usage has reached " . usage_percentage . "%. Please take action." | eval alert_level = if(usage_percentage >= 80 AND usage_percentage < 90, "80%-89%", alert_level) | eval alert_message = if(usage_percentage >= 80 AND usage_percentage < 90, "License usage has reached " . usage_percentage . "%. Please take immediate action.", alert_message) | eval alert_level = if(usage_percentage >= 90, "90% and above", alert_level) | eval alert_message = if(usage_percentage >= 90, "License usage has crossed critical threshold at " . usage_percentage . "%. Immediate attention required!", alert_message) | table alert_level, alert_message ... View more

Hi @anandhalagaras1  You can try this query: | rest /services/licenser/pools | eval total_quota_gb = toint(usage_quota / 1024 / 1024 / 1024) | eval used_gb = toint(usage_used / 1024 / 1024 / 1024) | eval usage_percentage = round((used_gb / total_quota_gb) * 100, 2) | table total_quota_gb, used_gb, usage_percentage | where usage_percentage >= 70 AND usage_percentage < 80 | eval alert_level = "70%-79%" | eval alert_message = "License usage has reached " . usage_percentage . "%. Please take action." | if(usage_percentage >= 80 AND usage_percentage < 90, appendpipe [| eval alert_level = "80%-89%"; eval alert_message = "License usage has reached " . usage_percentage . "%. Please take immediate action."], "") | if(usage_percentage >= 90, appendpipe [| eval alert_level = "90% and above"; eval alert_message = "License usage has crossed critical threshold at " . usage_percentage . "%. Immediate attention required!"], "") | table alert_level, alert_message ... View more