Have a look at this solution https://community.splunk.com/t5/Splunk-Search/Can-you-create-modify-a-lookup-file-via-REST-API/m-p/193699 @mthcht wrote a script that works. I modified it a little for my use but it is basically the same solution and works on a single head or on a SHC. The gist is that it loops through and reads the contents in python and then uploads a big string. ... View more

Hi. What version of Splunk are you running? I ran into a bad bug with both Splunk Enterprise 9.3.7 and 9.4.5. The heavy forwarders sending to DNS load balanced indexers get TCPOUT blocked.  This bug does not appear to be on the known issues despite many attempts by me trying to get it added there. It does not happen with 9.2.4. The Splunk JIRA that was opened is SPL-288904  The bug is said to be fixed in the upcoming releases 9.3.9 , 9.4.7, 10.0.3, 10.1 Hopefully soon. A workaround is the setting of dnsResolutionInterval in outputs.conf dnsResolutionInterval = <integer> * The base time interval, in seconds, at which indexer Domain Name Server (DNS) names are resolved to IP addresses. * This is used to compute runtime dnsResolutionInterval as follows: Runtime interval = 'dnsResolutionInterval' + (number of indexers in server settings - 1) * 30. * The DNS resolution interval is extended by 30 seconds for each additional indexer in the server setting. * Default: 300 seconds (5 minutes)   Splunk had recommended we set dnsResolutionInterval =480 (tcpout blocked). I tried 1000 (also blocked).  I have set it to 10000 (ie 10,000) and after ~ 3 days this seems to be working.   ... View more

Looks like you can find the quarantined lookup files https://splunk.my.site.com/customer/s/article/Monitor-quarantined-lookup-files     ... View more

Hi @yazeedallabadi2 did you see this https://splunk.my.site.com/customer/s/article/HF-paused-data-flow-due-to-connection-timeout-but-didn-t-resume-data-transfer-after-the-connection-is-established I see that it says it affects UF as well as HF ... View more

Hi @jcorcorans I haven't discovered any great way to parse the chef-client.log A few things that can help 1) look for the log_level when it isn't INFO/WARN [2025-02-24T19:06:07+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report   2) for log rotate, I see we have directives in /etc/logrotate.d/chef-cilent "/var/log/chef/client.log" { weekly rotate 12 compress postrotate systemctl reload chef-client.service >/dev/null || : endscript } 3) and if you have a number of servers and you are running chef a lot and want to know when to truly spend time debugging since we find a chef operation can fail due to timeout or load, you check over a time period and see if in the end things are running okay.  So we have something like this: if after 3 times chef run is still not good then investigate idx=your_index sourcetype=chef:client ("FATAL: Chef::Exceptions::ChildConvergeError:" OR "FATAL: Chef::Exceptions::ValidationFailed" OR "Chef run process exited unsuccessfully" OR "INFO: Chef Run complete" OR "INFO: Report handlers complete") | eval chef_status=if(searchmatch("ERROR") OR searchmatch("FATAL"), "failed", "succeeded") | stats count(eval(chef_status="failed")) AS num_failed, count(eval(chef_status="succeeded")) AS num_succeeded,latest(chef_status) as latest_chef_status by host | search num_failed > 3 AND latest_chef_status!="succeede To monitor the logs, a simple monitoring stanza in your inputs [monitor:///var/log/chef/client.log] sourcetype=yourchefsourcetype index=your_index ... View more

Hi @secure as @gcusello stated you can have only one base search. What would it mean to have 2 in a panel?  The base search just returns the results so how could you use 2 together? I am not sure if this helps but you can have a base search use another base search. ... View more

Here's some good information about the dispatch directory: https://docs.splunk.com/Documentation/Splunk/9.3.1/Search/Dispatchdirectoryandsearchartifacts Splunk normally does age things out but read the doc above. Perhaps the disk is full for other reasons? https://community.splunk.com/t5/Splunk-Search/Splunk-says-dispatch-directory-is-full-but-when-I-go-to-the/m-p/370243 One thing that can cause your dispatch directory to grow is if you adjust the time to live (TTL) of jobs. ... View more

This answer https://community.splunk.com/t5/Dashboards-Visualizations/How-to-use-token-in-a-multi-select-form-input/m-p/480570 is close to what you want.  You would end up with a set of sourcetype=data1 OR sourcetype=data2 etc. And you can initialize the default value with comma separated values as shown in https://community.splunk.com/t5/Dashboards-Visualizations/choose-all-Multiselect-values-by-default-without-using/m-p/357860 ... View more

Sounds like you might want to use two bin commands. First bin by time: | bin _time span=1h   Then bin the netPerf.netOriginLatency into 5 (?) bins e.g. | bin netPerf.netOriginLatency bins=5 See the bin command https://docs.splunk.com/Documentation/Splunk/9.2.2/SearchReference/Bin Finally you could do a timechart with your bins (you will have to do your percentage etc calculation) ... View more

What I tend to do to get all the results in email or Slack is to use stats as described here https://community.splunk.com/t5/Reporting/Using-result-fieldname-in-email-text-body-splunk-email-alert/m-p/399711   ... View more

Splunk has finally added the issue to their known issues page https://docs.splunk.com/Documentation/Splunk/9.2.0/ReleaseNotes/KnownIssues  https://docs.splunk.com/Documentation/Splunk/9.2.1/ReleaseNotes/KnownIssues ... View more

If you always want to ignore the same hosts each time, you could create a lookup file with names of the hosts and use a search as described in this post: https://community.splunk.com/t5/Splunk-Search/How-to-search-for-all-IP-s-not-in-a-lookup-table/m-p/371170 Something like index=myindex NOT [|inputlookup mylookup.csv | fields host]   ... View more

Hi can you say a little more about what the intended field values are that you are trying to achieve? ... View more

Hi. We have Splunk case 3421789 opened for this bug. For us it is installing from rpm that fails. It is not an option to install from tar.     ... View more

So there's a bug with installing Splunk Enterprise 9.2.x and the universal forwarder on the same server, something that should work. I have opened a case with Splunk and requested them to document the issue in the known issues. They have not done that yet.  ... View more

Thanks @hrawat  The logs are as expected then 05-03-2024 17:46:52.999 +0000 WARN AutoLoadBalancedConnectionStrategy [24761 TcpOutEloop] - Current dest host connection 1.2.3.4:5678, oneTimeClient=0, _events.size()=993, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0, _lastHBRecvTime=Fri May 3 17:46:48 2024 is using 475826 bytes. Total tcpout queue size is 512000. Warningcount=2001 ... View more

Hi. We just upgraded from 9.0.6 to 9.1.4 and are seeing these same warnings. Do we know that this was fixed in 9.1.4? ... View more

Hi. Have a look at  https://community.splunk.com/t5/Splunk-Search/Why-is-Lookup-definition-in-transforms-conf-not-returning/td-p/589334 Sounds like there's a lookup definition in a transforms.conf and the corresponding lookup file does not exist. You can use btool on the Splunk head to locate the setting. For example /opt/splunk/bin/splunk btool transforms list --debug | grep file   You can see all the lookup file definitions. ... View more

Thanks @isoutamo  ... View more

How about | eval MyNewField=Host+Domain ... View more

Hi.  So you tried | makeresults ns=project* | eval _raw="\"totalTimeTaken\":4" | rex field=_raw "\"totalTimeTaken\":+(?<Response_Time>\d+)" | stats avg(response_time)   And there are two problems. 1) the first makeresults .. I don't know what the ns=project* is. Here's the reference https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Makeresults 2) your rex is extracting the value into the field Response_Time but then use do stats avg on response_time in lowercase.  Case matters in Splunk field names. Here's what seems to work | makeresults=1 | eval _raw="\"totalTimeTaken\":4" | rex field=_raw "\"totalTimeTaken\":+(?<Response_Time>\d+)" | stats avg(Response_Time) ... View more