Hi,     I'm trying to isolate why I'm not able to drop data from a HEC Collector endpoint. I have some docker logs I don't need to ingest. The Splunk HF is still on 7.3.8 for backwards compatibility, so I don't know if that's in play here. I checked with btool, and the files did load correctly. inputs.conf:  - Sidenote here: When I set "source" value, it remained as "httpevent". But when I changed Sourcetype, the event changed correctly, which is odd.     [http://tpas_token] disabled = 0 index = elm-tpas-spc token = DD0D58D8-9F38-4A96-956C-XXXXXXXXXXXXXX source = tpas-event sourcetype = tpas-event     props.conf  - Sidenote: I tried also [ tpas-event ], and that also did not work     [ source::tpas-event ] TRANSFORMS-drop-handlers = drop-handlers      transforms.conf     [ drop-handlers ] REGEX = handlers.py|connection.py DEST_KEY = queue FORMAT = nullQueue       ... View more

Hi! I'm having a struggle trying to get Splunk to recognize a file that's in Asterisk Delimited Format. I have the props.conf set like this below, running on a Splunk 7.3.8 HF, sending the cooked data to a 8.1.72 Search Peer. Nothing I've tried will get the data to parse correctly. Everything I'm reading, this should work. I've opened a support case, but I'm going around in circles with them, so if anyone has any thought here, I would appreciate it!       [ sourcetype ] SHOULD_LINEMERGE=true LINE_BREAKER=([\r\n]+) NO_BINARY_CHECK=true CHARSET=UTF-8 disabled=false FIELD_DELIMITER=* FIELD_NAMES=timestamp,..... TRUNCATE=50000       Thanks, Stephen  ... View more

I am trying to use the Splunk Addon for New Relic v2.2.0, and I keep getting errors. I think the New Relic API has changed slightly, and I tried to edit the modalert python file, yet I'm still getting errors such as: 02-22-2022 04:52:01.962 -0500 ERROR sendmodalert - action=alerts_to_newrelic STDERR - NameError: name 'basestring' is not defined 02-22-2022 04:52:01.962 -0500 ERROR sendmodalert - action=alerts_to_newrelic STDERR - if isinstance(self.sid, basestring) and 'scheduler' in self.sid: 02-22-2022 04:52:01.962 -0500 ERROR sendmodalert - action=alerts_to_newrelic STDERR - File "/opt/splunk/etc/apps/Splunk_TA_New_Relic/bin/splunk_ta_new_relic/cim_actions.py", line 124, in __init__   We are on Splunk 8.1.7. Thanks! Stephen ... View more

Hi folks,    I'm trying to figure out if and how I could drop data, or modify data based on a metadata tag. In looking at the props.conf spec, it seems you can only reference sourcetype, source or host. My inbound data is setting up a metadata tag of "namespace" (It's Splunk Connect for Kubernetes, if it helps any). I did try [namespace::<value>], but it doesn't seem to work. I can't get specific enough with the existing choices of host, source or sourcetype.  My 2 use cases are: 1) Run a props.conf SEDCMD on some "cluster_name" only 2) Drop some "namespace" tags I don't want to ingest And neither one of these appear to work, and may be expected not to, based on documentation? Are there other options? Thanks! Stephen ... View more

Hi! I'm trying to continue to tune our Splunk Search Heads (We currently have 15!). I'm noticing a few odd behaviors and wanted to see what I could do. 1) Over time, the number of PendingDiscard messages keeps piling up 2) Our Search Head CPU and Memory usage is relatively low to medium 3) Over time, our concurrency counts keep growing (I need to lower my limits, which may also help here) 3) On our search heads, the executor_workers is set to the default of 10. executor_workers = <positive integer> * Only valid if 'mode=master' or 'mode=slave'. * Number of threads that can be used by the clustering thread pool. * A value of 0 defaults to 1. * Default: 10  Which leads to my question. I've seen a couple of answers posts around setting this on the Search Head, but the Docs for .7.2.10 look to me like it's more designed for the Indexers.  But would adding more executor workers allow for more searches to run? What I'm seeing is no more than 15-25 searches run on my 16 Core (32 vCPU) box, even though my limits are much higher than that. And the box isn't even breathing hard. Can someone offer some advice here? Thanks! Stephen ... View more

Hi! I am looking to try to standardize my configuration across my Search Head Cluster. I have 15 Search Heads, and what I am looking to to is move my etc/system/local configs to a searchhead app (let's call it etc/apps/searchhead).    Looking at my files, most of them should be fine, but I was wondering about the syntax for the distsearch.conf lookups. What I have now is like: lkp1 = apps/idm_search/lookups/lkpInterceptAttempt.csv Would that same path find the file when the file is in /opt/splunk/etc/apps/searchhead/distsearch.conf? Or do I have to be more explicit about it's location? Thanks! Stephen ... View more

Hi! I am trying to deal with some technical debt, and I thought I had an understanding of what I needed. 😞  Objective: I have some search artifacts inside a custom app on my Search Head Deployer that once deployed, users are unable to delete. On the Deployer, those searches were in the Default folder. I did give this what I thought was a thorough reading https://docs.splunk.com/Documentation/Splunk/7.2.10/DistSearch/PropagateSHCconfigurationchanges, but I may have missed something. What I tried: 1) Backup the app 2) Moved the contents on the Deployer Server of shcluster/apps/app_name/default/savedsearches.conf to local/savedsearches.conf 3) I updated the local/app.conf to a "deployer_push_mode = full". What appeared to happen: It seems that it may have merged the deployer local folder into the default on the cluster, which may make sense after another spin through the document. What's the best approach for cleaning up this app? Should I just deploy it to  my Search Head Deployer as an app, then remove the searches I want to clean up, then redeploy to the cluster? Thanks! Stephen ... View more