Did you restart that instance yet ? I don't see screenshot here ... View more

execprocessor queue is %819 percent, is hesplfwd002 an indexer ? execprocessor would not typically stop indexing. you can restart that instance anyways. for file operation errors in the indexer cluster( typically from antivirus or account permissions) , I can suggest you this search also: index=_internal (log_level=ERROR OR log_level=WARN) (host=myindexers) (component=PeriodicReapingTimeout OR component=BucketMover OR component=BucketReplicator OR component=CMRepJob OR component=DatabaseDirectoryManager OR component=DispatchCommandProcessor OR component=HttpClientRequest OR component=HttpListener OR component=IniFile OR component=ProcessRunner OR component=TPool OR component=TcpInputProc OR component=TcpOutputFd OR component=IndexerService OR component=S2SFileReceiver) | stats values(host) dc(host) last(_raw) count by punct,component | fields - punct This will get you last error of each type , and components to check for issues. if this points to anything, you could adjust logging levels on that component to see what is failing. if it is an indexer in cluster, do in CLI to restart : splunk offline this will take some time, then: splunk start Finally, check bucket status on the Cluster Master. Check which server has copies failing. ... View more

if indexer or indexer cluster is not down, this message generally indicates high queue utilization in Splunk. if you have DMC set up: Settings > Monitoring Console > Indexing > Performance > < 2 Options> first, if you have a cluster, check deployment first to see what is busy. then check the instance that is indexing a lot, or high in Splunk queues. Panel > Median Fill Ratio of Data Processing Queues Change dropdowns to all queuest and maximum. if you don't have DMC set up, this search would show you all of your queues. Find your indexer from host name: index=_internal group=queue | eval percfull=round(((current_size_kb/max_size_kb)*100),2) | search percfull>80 | dedup host, name | eventstats dc(host) as hostCount | table _time host name hostCount current_size_kb max_size_kb percfull | rename _time as InfoTime host as Host name as QueueName current_size_kb as CurrentSize max_size_kb as MaxSize percfull as Perc% | sort - Perc% | convert ctime(InfoTime) ... View more

it doesn't give you exactly what you need. however, representation wise, I believe this is not too far from what you need: index=_internal | bucket _time span=1h | where date_hour>=8 and date_hour<16 | sort date_mday | eval mhour = date_wday.":".date_hour | chart count over mhour by sourcetype You should also note that, timechart command would have its own span and I am not sure how 1h would work in weekly chart. ... View more

Can you give more information about your OS and version of the Splunk ? ... View more

I think you do some unnecessary conversion on time fields. you could simply connect all in epoch with strptime() and make your comparisons | eval endepochDate=relative_time(strptime(table1_time, "%m/%d/%Y %H:%M:%S"), "+72H@H") | lookup account_admin User as email OUTPUTNEW "Server Connection" AS servcon "Service Account" AS servacct Time AS ark_time | where strptime(ark_time,"%m/%d/%Y %H:%M:%S") > strptime(table1_time,"%m/%d/%Y %H:%M:%S") and strptime(ark_time,"%m/%d/%Y %H:%M:%S") <= endepochDate ... View more

If you can merge your queries into one SELECT, I think it is definitely better than UNION. UNION is a shortcut to me if I have the query from some DB admin and not have a time to make it better 🙂 ... View more

As far as I know, DB connect doesn't support SP (Stored Procedure) style queries. Once upon a time, I remember using UNION SQL command to have my query work. if you must drop table after transaction, perhaps a second scheduled search that runs after 1st one can do it with: |dbxquery "DROP TABLE <TABLE_NAME>" ... View more

Hey, I will soon work on this task and grade your answer 🙂 ... View more

Can you pick my solution as answer then 😄 ? No problem, I learned a lot here from other people ... View more

Assuming this is your regex for the field: ...|rex "\nBody:\s(?<_raw>[^\}]+)" then I would do ....| regex field=_raw "\nBody:\s(?<new_raw>[^\}]+)" | eval _raw=new_raw please note that pre-extracted fields such as host, _time, sourcetype will still be there. you can always see time of the event even if it doesn't exist in new_raw. If you want to index from HF on this way, this will require you props.conf and transforms.conf. But that's not the question after all 🙂 ... View more

Technically, same settings from Search head deployer is sent to all search head instances. The way you sounded in your question first reminds me search modes of Splunk: fast, smart and verbose Are you sure search is made on smart or verbose on both searches ? is it possible second search is made on fast mode ? if it is not the case: one thing would be comparing props.conf and transforms.conf on both search heads as well as SH deployer as mentioned. if this is not the case also, on splunk CLI ./splunk cmd btool props list --debug | grep <Field_that_you_are_looking for> you should look if SH gets setting from same file on each search head. Especially a search head is added to cluster later than initial members, I had couple cases where previous users added some stuff to etc/system/local that conflicts with my changes from search head deployer. Thanks. ... View more

I was about to write exactly the same. Too late 🙂 ... View more

Selected answer correct for if you have one field name, for multiple similarly I use: |sort - duration |dedup field_name ... View more

what If i dont wan't to update my db connect for other reasons, and just disappear this message. Should i delete entry from restmap.conf at ...\splunk_app_db_connect\default ? [admin_external:health] handlertype=python handlerfile=rh_healthlog.py handleractions=create,edit,list,remove,enable,disable,_reload what is the correct way of deleting this entry? workflowactions.conf ? ... View more

if you know dropdown format, you can achieve this in search like: (Let's say there is a space between) index=foo.. | eval mval=$token$| rex field=mval "^(?<firstname>\w+)\s(?<lastname>\w+)$" | search (user=mval OR user=firstname.".".lastname Having 'user=' in first pipe next to index would be more optimized. However, it would require some tweaking in Dashboard XML or Macro. Macro: you need to create a macro that takes only one parameter (your token), and returns a string like: (user="fname lname" OR user=fname.lname) Ex: [| makeresults | eval mval=$param$ | rex field=mval "^(?<fname>\w+)\s(?<lname>\w+)$" | eval search = "(user=".fname," ".lname." OR user=".fname.".".lname.")" | return $search] Then you can call it in your search as: index=foo 'mymacro($token$)' XML: I couldn't think of a way now using 'change' or 'set' tags. however, I am sure there are ways. I am sure somesoni will explain this part 🙂 ... View more

I appreciate all the answers here that I have used in my tasks. In addition to above, there is one more thing to check: ACL entries of account that runs forwarder on RH (or SEL). If nothing works above, this is a good thing to check and add necessary config. ... View more

this info should be in WinEvent:Security logs. I don't have that app to check win logins. if you can provide search by clicking that dashboard or application name/dashboard name of the view, I can help further. ... View more

My answer would be same as somesoni eval jobname=replace(jobname,"\\\n","") ... View more

Hi Heathramos, I had similar need recently and made it there with following: index=_audit "action=login attempt" sourcetype=audittrail NOT SEARCH | table _time user src dest info if you are looking for failed only, you can either add |search info=failed to the end of the search OR: index=_audit "action=login attempt" sourcetype=audittrail info=failed NOT SEARCH | table _time user src dest info ... View more

I saw similar in other answers | rest splunk_server=local /servicesNS/-/-/data/ui/views as well as rest /services/saved/searches I am having issues to return results from these and I am an admin. Do you know what could be my issue? Are there anyway to combine audittrail logs with some other internal log to get the same results ? ... View more