Can you please show your props.conf? Depending on that, it might be possible that test_key is extracted AFTER your new extractions - therefore it doesn't work. ... View more

I think he means one search head and another search head... ... View more

I'd guess that the indexer is under such heavy load that the connection times out. Please show us your actual "better values" for the settings above, and also take a close look at your monitoring console about Ressource usage etc. ... View more

Hey, did you try this? https://answers.splunk.com/answers/457956/how-to-drill-down-from-a-splunk-dashboard-to-an-ex.html Also note the second answer about not encoding special characters 😉 ... View more

What is your alert condition? ... View more

This shouldn't happen. However, your question is rather generic. Check | index=_internal ERROR to see if you get any errors in the logs, also take a look into var/log/splunk/splunkd.log on those forwarders. ... View more

Mh, using the host field not for the host, but for a grouping by day isn't very good practice. However, it should still work. Did you try this: | tstats prestats=t count where host=01_04_2018 by _time sourcetype | timechart count by sourcetype This should give you a timechart diagram of the data, and that shouldn't change on every query. ... View more

Exactly that one. ... View more

Yes, exactly. You could put this in a custom props.conf : [yoursourcetype] LINE_BREAKER = ((?:Body\s*)|(?:HoggingThreadCount \d+))([\r\n]+) You can see it work here: https://regex101.com/r/iygoQ2/3 I've inserted empty lines and ##### where the events would get broken. You must however be aware of the following: - The part before your first data (the mail headers) would be a single event on it's own, you would have to either live with that or build another props entry to drop it - This will be done to all your data from the same sourcetype - You will loose the data from the header, especially the timestamp Depending on the requirements, you might be better off with one or the other solution. ... View more

It should be either moved or you should check if you have another setting like this in the right file, somewhere. 🙂 ... View more

Is it only me, or is the [target-broker:deploymentServer] part in the wrong .conf file? Shouldn't that be in deploymentclient.conf ? ... View more

You could break your mail into multiple events, as @PowerPacked suggested. Alternatively, you could do do something like this: | makeresults | eval test="Date = '03-May-2018 07:04:31 -0400' From = '' To = ', ' Subject = 'Daily Server Health Report' CC = '' mailbox = 'INBOX' size = 51929 Message Body coreServer1 HealthState HEALTH_OK HoggingThreadCount 0 coreServer2 HealthState HEALTH_OK HoggingThreadCount 0 coreServer3 HealthState HEALTH_WARN,[ThreadPool has stuck threads] HoggingThreadCount 1 fsServer1 HealthState HEALTH_OK HoggingThreadCount 0 fsServer2 HealthState HEALTH_OK HoggingThreadCount 0 fsServer3 HealthState HEALTH_OK HoggingThreadCount 0 genServer1 HealthState HEALTH_OK HoggingThreadCount 0 genServer2 HealthState HEALTH_OK HoggingThreadCount 0 ociServer1 HealthState HEALTH_OK HoggingThreadCount 0 ociServer2 HealthState HEALTH_OK HoggingThreadCount 0 ociServer3 HealthState HEALTH_OK HoggingThreadCount 0 sourceServer1 HealthState HEALTH_OK HoggingThreadCount 0 sourceServer2 HealthState HEALTH_OK HoggingThreadCount 0" | rex max_match=0 field=test "(?<allevents>\S+\nHealthState\s+\S+\nHoggingThreadCount\s+\d+)" | fields _time allevents | mvexpand allevents | rex field=allevents "^(?<Server>.*?)\s+HealthState\s+(?<HealthState>.*?)\s+HoggingThreadCount\s+(?<ThreadCount>.*?)$" | table _time Server HealthState ThreadCount Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂 ... View more

| eval "AVG TPS"=round("AVG TPS", 2) ... View more

What role does the user have you are logged in with? ... View more

PS: You can contact Splunk support and put your vote to enhancement request SPL-142222, that I field a few days ago. It's asking to allow having attachments to alert mails compressed, which would most likely reduce their size a lot. 🙂 ... View more

Hey, the solution is in the error message: reason=Unauthorized It might mean your password expired, your account was locked or deleted, anything like this. Your ServiceNow admins can most likely tell you more about this. Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂 ... View more

Sorry, please try again, I updated my answer. ... View more

Can you show your actual query? ... View more

Did you try to access those logs as the user Splunk runs at, to make sure it's not a permission issue? If that is fine, try /opt/splunkforwarder/bin/splunk list inputstatus to see the status of all of your inputs - you should see your monitor there and also it's status. ... View more

I can't say from experience, but I'd guess(!) if there's any, it's minimal.. just what is not used to have your GUI updated, but I'd say the actual search load is the same. Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂 ... View more

Yes, it's correct. 🙂 When for example your initial query contains results, but you're doing a | stats by somefield , and that somefield doesn't exist in any those events, you get 0 results. An alert that aims for more than 0 results would therefore not fire. Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂 ... View more