Hi Team, While setting up our new remote Heavy Forwarder, we configured it to collect data from 20 universal Forwarders and Syslog devices, averaging about 30GB daily. To control network bandwidth usage, we applied a maximum throughput limit of 1MBps (1024KBps) using the maxKBps setting in limits.conf on the new remote Heavy Forwarder. This setting is intended to cap the rate at which data is forwarded to our Indexers, aiming to prevent exceeding the specified bandwidth limit. However, according to Splunk documentation, this configuration doesn't guarantee that data transmission will always stay below the set maxKBps. It depends on factors such as the status of processing queues and doesn't directly restrict the volume of data being sent over the network. How can we ensure the remote HF is not exceeding the value set in maxKBps in any case. Regards VK ... View more

Hi All, I currently have a primary standalone Enterprise Security (ES) search head located in the main data center. Every day, a cronjob is executed to copy the entire /opt/splunk/etc/apps directory to the secondary standalone Enterprise Security search head, which is located in the DR site. Now, the question arises: should I also copy the primary KVStore data, located in the var/lib directory, to the secondary ES search head? Currently, I'm only syncing the apps folder and not the var/lib directory. In the event of an issue with the primary search head in the future, I plan to bring up the secondary search head. Will there be any issues with the KVStore data if I'm not syncing the var/lib directory between the primary and secondary search heads? Note :Since we're not using any custom-made KVStore lookups and only depend on the default ones generated by different Enterprise Security apps, it makes us wonder if syncing the var/lib directory between the primary and secondary search heads is essential. Regards VK ... View more

Hi Team, We're encountering a problem with the Incident Review History tab in Splunk ES. Clicking on Incident Review, then a specific notable (like 'Tunnelling Via DNS'), followed by History and clicking 'View all review activity for this Notable Event', results in an empty history being displayed for all the notables. Any leads on this would be highly appreciated. Note : Recently, we have upgraded to Splunk ES to 7.1.2 from 7.0.0 Regards VK18  ... View more

Hi Team, At present, SSL encryption is enabled between the Universal Forwarder (UF) and the Heavy Forwarder (HF), while communication from HF to Indexers occurs without SSL encryption. However, there are plans to establish an SSL channel between the HF and Indexers in the future. Additionally, communication between Indexers and the License Master, as well as between HF and the License Master, currently operates through non-SSL channels. There is a requirement to transition these communications to SSL-enabled connections. Could you provide guidance or documentation outlining the necessary implementation steps for securing the communication from Indexers & HF to License Master to facilitate these changes? ... View more

Hi All, We have approximately 100 Splunk Universal Forwarders (UFs) installed at a remote site, and we're interested in setting up a Heavy Forwarder (HF) at that location to forward the data to the indexers from the UFs. Additionally, we plan to deploy the deployment server on the same virtual machine (VM). Based on the documentation, it appears that a deployment server can be co-located with another Splunk Enterprise instance as long as the deployment client count remains at or below 50. We would like to better understand the rationale behind this limitation of 50 clients and why it is not possible to manage more than 50 clients by adding another component of Splunk Enterprise ?   Regards VK ... View more

Dear Team, We are planning to upgrade our existing underlying OS/VM infrastructure. As part of this process, we need to ensure the backup and restoration of our Splunk environment in case any issues arise. Below, you can find the details of our environment: Search Head Cluster (SHC) A standalone Splunk Security SH. Indexer cluster All other management servers(DS/CM/deployer/LM) Heavy Forwarders/Universal Forwarders (UFs) In addition to backing up $SPLUNK_HOME/etc and $SPLUNK_HOME/var, as well as the kvstore, are there any other components or data that we need to back up to ensure a successful restoration process? ... View more

We are currently using a regex pattern to match events against our raw data, and it works perfectly when we use the search app. The pattern we are using is: C:\\Windows\\system32\\cmd\.exe*C:\\ProgramData\\Symantec\\Symantec Endpoint Protection\\14\.3\.8289\.5000\.105\\Data\\Definitions\\WebExtDefs\\20230830\.063\\webextbridge\.exe* However, when we try to use this regex pattern in a lookup table, the events are not being matched. This seems to be because of the wildcard in the pattern. Despite defining the field name in the lookup definition (e.g., WILDCARD(process)), it still doesn't match the events. I'm wondering if Splunk lookup supports wildcards within strings, or does it only support them at the beginning and end of strings? Any insights or guidance on this matter would be greatly appreciated. Regards VK ... View more

We have activated several data models for use with Splunk Enterprise security scenarios and are interested in clarifying the retention period for the summaries generated by these data models. According to the Splunk documentation, the retention period is determined by the accelerated summary range. For instance, if our network traffic accelerated summary range is set to 15 days, does this imply that the retention period is also 15 days, and that it stores 15 days' worth of summaries? ... View more

Why SSL status show as "false" despite configuring SSL. In Our environment we have enabled TLS configuration between forwarders and receivers. The connection is established and we could see data is coming through secure TLS channel into splunk. I have validated manually as well using openssl client module and verification was successful with status ok. We could see for the hosts, SSL as false and it keeps changing at random times to True or False. And connection type is cookedSSL for the False host.   I have checked all the tcpoutputproc and tcpinputproc in splunkd logs, cannot find any errors related to SSL.   But found below WARN messages on one of the forwarders. Is this causing the problem ? Any leads on this. ... View more

Hi Team, I would like to establish an SSL/TLS-connection with third party CA certificates between the UFs -> HFs -> indexers. The order which i'm following to configure the TLS connection is below. -----BEGIN CERTIFICATE----- ... (certificate for your server)... -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- ...<Server Private Key – Passphrase protected> -----END RSA PRIVATE KEY----- ------BEGIN CERTIFICATE----- ... (the certificate authority certificate)... -----END CERTIFICATE----- Now, the question here is, can we remove RSA private key from the certficate. Do we need private key in order to establish the secure connection to the HF from UF? ... View more

Hi Team, We have installed an app "Microsoft Teams Alert Cards" to create an alert in MS Teams through webhook. Everything is working well but it displays the hostname of the search head and username of the person who has scheduled the alert, when we go the Teams group and hover the mouse on Go to Search on MS Teams. It displays hostname and username, Below screenshot for reference where I have masked the hostname and user id. How can we ignore or remove the hostname and userid on the alert ? Tried checking in advance edit of Alert, couldn't find any option related to this settings. Regards VK ... View more

Hi All, Duo connector installation docs for splunk isn't clear for multi site cluster environment. Can anyone suggest where to install this Duo connector app ? I assume this needs to be installed on any HF and configure it to one custom index to receive the logs? I do have the admin api keys and other integration keys to configure it. Please suggest if anyone installed Duo connector in multi site.   ... View more

Hi Team, We have the current infrastructure : UF -> HF -> Indexers Now, the question here is can we set up external load balancer between UF and HF ? The reason is we got 6 HF's and 20UF's spread across different zones. Opening Firewall ports will be tedious task from 20 UF's to 6 HF's  as this OT environment and we do not want to expose multiple IP's due to security reasons. Regards VK ... View more

Hi All, We have installed Splunk Enterprise Security 7.0.1 and OT for security add-on on it, and we would like to upgrade Splunk Enterprise security  to the latest version ES 7.1.1. would like to understand on, We made a lot of changes on search queries for OT for security add-on to bring-up the data. Will that changes would go away after ES upgrade? Can you please confirm VK. ... View more

Hi All, How can we stop duplicate notables which are getting generated in the Incident Review page for same event id in the Enterprise security . Do we need to adjust any settings ? Time range  Earliest = -70M Latest = -10M Cron schedule : every 35 Minutes It's happening for all correlation searches . Regards VK   ... View more

Hi Team. I'm looking for a way to rename a correlation search that has been created with the wrong format. The CS is currently disabled but I don't see a way to actually rename it. If i delete it from Saved searches, Will it remove all notables which got created from this custom CS created? Regards Varun   ... View more