Hi Community! i have a (kind of ) special problem with my data routing. Topology: We have 2 different Clusters, one for ES and one for Splunk Enterprise. Each clusters consist of minimum 1 Search head 4 Indexer peers (Multisite Cluster). All hosted on RedHat Virtual Machines. Usecase: On all Linux systems (including Splunk itself) are some sources defined for ES and some sources for normal Splunk Enterprise indexes. E.g.: /var/log/secure - ES (Index: linux_security) /var/log/audit/audit.log - ES (Index: linux_security) /var/log/dnf.log - Splunk Enterprise (Index: linux_server) /var/log/bali/rebootreq.log - Splunk Enterprise (Index: linux_server) Problem: The Routing of those logs from the collecting tier (Universal Forwarder, Heavy Forwarder) is fine, because those components have both clusters as output groups defined including props / transforms config.  On Search heads there are only the search peers defined as output group (ES Search head --> ES Indexer Cluster, Splunk Enterprise Search head --> Splunk Enterprise Cluster). This is due to several summary searches and inputs from the Search head, im not able to adjust the routing like we do on the Heavy Forwarder because of the frequent changes made my powerusers. That is working fine so far except for the sources that require to be sent to the opposite cluster. Same for the logs directly on the Indexer Tier, the defined logs requires to get sent to the other cluster. So simplified: The log /var/log/secure on Splunk Enterprise Cluster Search head / Indexer needs to be sent to ES Cluster Indexer. The log /var/log/dnf.log on the ES Cluster Search head / Indexer needs to be sent to the Splunk Enterprise Indexer. What i have done already: Configured both Indexer Clusters to sent data to each other based on the specific index in outputs.conf. With this the events are now available in the correct cluster, but are also available as duplicates in their source cluster. I try to get rid of the source events! Splunk Enterprise Indexer outputs.conf: [indexAndForward] index = true [tcpout] ... forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent) forwardedindex.3.blacklist = .* forwardedindex.4.whitelist = linux_secure forwardedindex.5.blacklist = _.* forwardedindex.filter.disable = false useACK = false useClientSSLCompression = true useSSL = true [tcpout:es_cluster] server = LINUXSPLIXPRD50.roseninspection.net:9993, LINUXSPLIXPRD51.roseninspection.net:9993, LINUXSPLIXPRD52.roseninspection.net:9993,LINUXSPLIXPRD53.roseninspection.net:9993 ES Indexer outputs.conf: [indexAndForward] index = true [tcpout] ... forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker|_dsclient|_dsphonehome|_dsappevent) forwardedindex.3.blacklist = .* forwardedindex.4.whitelist = linux_server forwardedindex.5.blacklist = _.* forwardedindex.filter.disable = false useACK = false useClientSSLCompression = true useSSL = true [tcpout:rosen_cluster] server = LINUXSPLIXPRD01.roseninspection.net:9993, LINUXSPLIXPRD02.roseninspection.net:9993, LINUXSPLIXPRD03.roseninspection.net:9993,LINUXSPLIXPRD04.roseninspection.net:9993 Additionally i tried to setup props.conf / transforms.conf like we do on HF to catch at least events from Search head and send them to the correct _TCP_ROUTING queue but without any success. I guess because they got parsed already on the Search head. Splunk Enterprise props.conf: [linux_secure] ... SHOULD_LINEMERGE = False TIME_FORMAT = %b %d %H:%M:%S TRANSFORMS = TRANSFORMS-routingLinuxSecure = default_es_cluster Splunk Enterprise transforms.conf: [default_es_cluster] ... DEST_KEY = _TCP_ROUTING FORMAT = es_cluster REGEX = . SOURCE_KEY = _raw ES props.conf: [rhel_dnf_log] ... SHOULD_LINEMERGE = True TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%Q TRANSFORMS-routingLinuxDNF = default_rosen_cluster ES transforms.conf: [default_rosen_cluster] ... DEST_KEY = _TCP_ROUTING FORMAT = rosen_cluster REGEX = . SOURCE_KEY = _raw Example: Source: /var/log/dnf.log _time _raw host source index splunk_server count 2024-09-10 12:07:21 2024-09-10T12:07:21+0000 DDEBUG timer: config: 3 ms linuxsplixprd51.roseninspection.net (Indexer ES) /var/log/dnf.log last_chance linux_server linuxsplixprd01.roseninspection.net linuxsplixprd51.roseninspection.net 2 2024-09-11 12:24:31 2024-09-11T10:24:31+0000 DDEBUG timer: config: 4 ms linuxsplixprd01.roseninspection.net (Indexer Splunk Enterprise) /var/log/dnf.log linux_server linuxsplixprd01.roseninspection.net 1 2024-09-10 13:15:04 2024-09-10T11:15:04+0000 DDEBUG timer: config: 3 ms linuxsplshprd50.roseninspection.net (Search head ES) /var/log/dnf.log last_chance linux_server linuxsplixprd01.roseninspection.net linuxsplixprd50.roseninspection.net 2 2024-09-10 13:22:53 2024-09-10T11:22:53+0000 DDEBUG Base command: makecache linuxsplshprd01.roseninspection.net  (Search head Splunk Enterprise) /var/log/dnf.log linux_server linuxsplixprd01.roseninspection.net 1 2024-09-11 11:55:51 2024-09-11T09:55:51+0000 DEBUG cachedir: /var/cache/dnf kuluxsplhfprd01.roseninspection.net (Heavy Forwarder) /var/log/dnf.log linux_server linuxsplixprd01.roseninspection.net 1 Any idea how i can achieve to get rid of those duplicate events at the source cluster (last_chance)? ... View more

Hi,  we implemented persistent queues to catch high latencies between the locations.  E.g. Connection is down for some minutes / hours and the firewall is not buffering logs itself so it gets lost if queues are filled up..  Is there a way to monitor the persistent queues? Fill ration or other metrics?    I dont want to build custom scripts collecting bash information.   ... View more