Hi @jcorcorans I haven't discovered any great way to parse the chef-client.log A few things that can help 1) look for the log_level when it isn't INFO/WARN [2025-02-24T19:06:07+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report   2) for log rotate, I see we have directives in /etc/logrotate.d/chef-cilent "/var/log/chef/client.log" { weekly rotate 12 compress postrotate systemctl reload chef-client.service >/dev/null || : endscript } 3) and if you have a number of servers and you are running chef a lot and want to know when to truly spend time debugging since we find a chef operation can fail due to timeout or load, you check over a time period and see if in the end things are running okay.  So we have something like this: if after 3 times chef run is still not good then investigate idx=your_index sourcetype=chef:client ("FATAL: Chef::Exceptions::ChildConvergeError:" OR "FATAL: Chef::Exceptions::ValidationFailed" OR "Chef run process exited unsuccessfully" OR "INFO: Chef Run complete" OR "INFO: Report handlers complete") | eval chef_status=if(searchmatch("ERROR") OR searchmatch("FATAL"), "failed", "succeeded") | stats count(eval(chef_status="failed")) AS num_failed, count(eval(chef_status="succeeded")) AS num_succeeded,latest(chef_status) as latest_chef_status by host | search num_failed > 3 AND latest_chef_status!="succeede To monitor the logs, a simple monitoring stanza in your inputs [monitor:///var/log/chef/client.log] sourcetype=yourchefsourcetype index=your_index ... View more

Hi @secure as @gcusello stated you can have only one base search. What would it mean to have 2 in a panel?  The base search just returns the results so how could you use 2 together? I am not sure if this helps but you can have a base search use another base search. ... View more

Here's some good information about the dispatch directory: https://docs.splunk.com/Documentation/Splunk/9.3.1/Search/Dispatchdirectoryandsearchartifacts Splunk normally does age things out but read the doc above. Perhaps the disk is full for other reasons? https://community.splunk.com/t5/Splunk-Search/Splunk-says-dispatch-directory-is-full-but-when-I-go-to-the/m-p/370243 One thing that can cause your dispatch directory to grow is if you adjust the time to live (TTL) of jobs. ... View more

This answer https://community.splunk.com/t5/Dashboards-Visualizations/How-to-use-token-in-a-multi-select-form-input/m-p/480570 is close to what you want.  You would end up with a set of sourcetype=data1 OR sourcetype=data2 etc. And you can initialize the default value with comma separated values as shown in https://community.splunk.com/t5/Dashboards-Visualizations/choose-all-Multiselect-values-by-default-without-using/m-p/357860 ... View more

Sounds like you might want to use two bin commands. First bin by time: | bin _time span=1h   Then bin the netPerf.netOriginLatency into 5 (?) bins e.g. | bin netPerf.netOriginLatency bins=5 See the bin command https://docs.splunk.com/Documentation/Splunk/9.2.2/SearchReference/Bin Finally you could do a timechart with your bins (you will have to do your percentage etc calculation) ... View more

What I tend to do to get all the results in email or Slack is to use stats as described here https://community.splunk.com/t5/Reporting/Using-result-fieldname-in-email-text-body-splunk-email-alert/m-p/399711   ... View more

Splunk has finally added the issue to their known issues page https://docs.splunk.com/Documentation/Splunk/9.2.0/ReleaseNotes/KnownIssues  https://docs.splunk.com/Documentation/Splunk/9.2.1/ReleaseNotes/KnownIssues ... View more

If you always want to ignore the same hosts each time, you could create a lookup file with names of the hosts and use a search as described in this post: https://community.splunk.com/t5/Splunk-Search/How-to-search-for-all-IP-s-not-in-a-lookup-table/m-p/371170 Something like index=myindex NOT [|inputlookup mylookup.csv | fields host]   ... View more

Hi can you say a little more about what the intended field values are that you are trying to achieve? ... View more

Hi. We have Splunk case 3421789 opened for this bug. For us it is installing from rpm that fails. It is not an option to install from tar.     ... View more

So there's a bug with installing Splunk Enterprise 9.2.x and the universal forwarder on the same server, something that should work. I have opened a case with Splunk and requested them to document the issue in the known issues. They have not done that yet.  ... View more

Thanks @hrawat  The logs are as expected then 05-03-2024 17:46:52.999 +0000 WARN AutoLoadBalancedConnectionStrategy [24761 TcpOutEloop] - Current dest host connection 1.2.3.4:5678, oneTimeClient=0, _events.size()=993, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0, _lastHBRecvTime=Fri May 3 17:46:48 2024 is using 475826 bytes. Total tcpout queue size is 512000. Warningcount=2001 ... View more

Hi. We just upgraded from 9.0.6 to 9.1.4 and are seeing these same warnings. Do we know that this was fixed in 9.1.4? ... View more

Hi. Have a look at  https://community.splunk.com/t5/Splunk-Search/Why-is-Lookup-definition-in-transforms-conf-not-returning/td-p/589334 Sounds like there's a lookup definition in a transforms.conf and the corresponding lookup file does not exist. You can use btool on the Splunk head to locate the setting. For example /opt/splunk/bin/splunk btool transforms list --debug | grep file   You can see all the lookup file definitions. ... View more

Thanks @isoutamo  ... View more

How about | eval MyNewField=Host+Domain ... View more

Hi.  So you tried | makeresults ns=project* | eval _raw="\"totalTimeTaken\":4" | rex field=_raw "\"totalTimeTaken\":+(?<Response_Time>\d+)" | stats avg(response_time)   And there are two problems. 1) the first makeresults .. I don't know what the ns=project* is. Here's the reference https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Makeresults 2) your rex is extracting the value into the field Response_Time but then use do stats avg on response_time in lowercase.  Case matters in Splunk field names. Here's what seems to work | makeresults=1 | eval _raw="\"totalTimeTaken\":4" | rex field=_raw "\"totalTimeTaken\":+(?<Response_Time>\d+)" | stats avg(Response_Time) ... View more

Hi. Your search is so close to what I do.. change search -> where   | tstats count where index=aws by host | table host | where NOT [| tstats count where index=windows by host | table host] ... View more

Glad to hear this is what you needed.    You can accept this solution to indicate the question was answered to your liking.  thanks! ... View more

Hi so how is are the events being split by Splunk? And do you have any props to tell splunk how to split the events? ... View more