Hi becksyboy, @tommoore I had a similar issue where the input stopped unnoticed for more than 2 weeks, and once it was restarted the events were no longer available from the MS API. It took me some time to troubleshoot the script/issue, but once I found who and where the checkpoint is accessed it was easy to manually check and update the checkpoint hidden deep inside this weird REST API / KV store construct. You can use this command to see the checkpoint: curl -k https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/ -u <username> And you can use this command to modify the checkpoint: curl -k --header "Content-Type: application/json" --request POST --data '[ { "state" : "{\"max_date\": \"2018-11-20 18:56:17.772814\"}", "_user" : "nobody", "_key" : "O365_<input name here>_checkpoint"}] ' https://127.0.0.1:8089/servicesNS/nobody/TA-MS_O365_Reporting/storage/collections/data/TA_MS_O365_Reporting_checkpointer/batch_save -u <username> Hope this helps should you have further issues ... cheers, MuS ... View more

Hi, Is this all of the script or is there more? If so can you please update and all of the script? cheers, MuS ... View more

hi dbashyam, It would indeed be possible to automate this using a custom alert action, see the docs for more https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModAlertsIntro First your search would query the REST endpoint /services/search/jobs/ to get what ever you define as a kill reason. Than you need to make sure the sid of the search is available as the result of this search. The reason for this is the alert action can then use $job.resultCount$ and do a post against the REST API using curl like this: curl -k -u admin:pass https://localhost:8089/services/search/jobs/<sid>/control -d action=cancel More details about this can be found in the docs https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTsearch#search.2Fjobs.2F.7Bsearch_id.7D.2Fcontrol You it is possible, but it takes some work to get there. Hope this helps ... cheers, MuS ... View more

Hi pranay04, That's pretty easy and straight forward in Splunk 🙂 Based on your provided examples take this run everywhere command: | makeresults | eval flubber="com.ibm.mq.MQException,ORA Error,OutOfMemory Error" | makemv delim="," flubber | mvexpand flubber | rename comment AS "Everything above this was used to create dummy data!" | eval status=case(match(flubber, "com.ibm.mq.MQException"), "MQError", match(flubber, "ORA Error"), "DB error",match(flubber, "OutOfMemory Error"), "OOM", 1=1, " ¯\_(ツ)_/¯ ") | chart count by status The result will look like this: Hope this helps ... cheers, MuS ... View more

Not quiet sure what you are asking, but the rules are exactly the same ¯\_(ツ)_/¯ Maybe summoning @YamadaNoel can help with the language here ? cheers, MuS ... View more

Can you please share the props.conf setting for the sourcetype paper trail ? cheers, MuS ... View more

Hi Shephali, There are multiple possible reasons why this is not working: the option must be set under the [thruput] stanza in limits.conf btool does NOT actually show the config Splunk is using, it just merges the on disk files and shows the potential config being used by Splunk to see the actual live config Splunk uses run $SPLUNK_HOME/bin/splunk show config limits check / verify file permissions last but not least it could be a bug? cheers, MuS ... View more

Hi kumagaur, based on the provided examples this will work: | makeresults | eval TeamMember="A1 A2 A3 A4 A5 A6 A7" | makemv TeamMember | mvexpand TeamMember | eval Team = if((TeamMember="A1" OR TeamMember="A2") AND TeamMember="A4" AND TeamMember="A7", "foo" , "nope") But, also based on your provided information this eval will never match anything because TeamMember has never at the same time either A1 or B2 and A4 and A7 . I would review either the use case or the events, because this based on the provided information will never work. Hope this helps ... cheers, MuS ... View more

Hi manekar, there are plenty of script examples on Github, like this one https://github.com/jorritfolmer/puppet-splunk Hope this helps ... cheers, MuS ... View more

Default behaviour on Windows Event logs and how they are rolled over will result in event loss! Confirmed by multiple customers and Splunk support (Case# 1214731), answer from Support: Basically when the log size has reached full, the O/S overwrite the file and that does not give us enough time to complete the reading the event. Thus, they recommend to set up "Overwrite events as needed(oldest events first)" instead. I am not sure if it's possible but if the O/S can save the events to any other folder just before archiving, we can monitor the folder instead. However, when I see the EventLog properties, the feature does not exist. Please use "Overwrite events as needed(oldest events first)" to avoid the missing event issue. cheers, MuS ... View more

That is the Splunk Enterprise Security Content Update or ESCU, meow details here https://docs.splunk.com/Documentation/ESSOC/latest/user/About cheers, MuS ... View more

Okay, this sounds like a conceptual/design problem here rather than a real Splunk problem. There are options that could help you to increase the performance of the universal forwarder, like splitting the inputs stanzas to be more specific on the path and add specific sourcetypes, index names to each stanza for example. Because using just one single input stanza for such a mount of data brings in a lot of other concerns like is all data going into the same index, if not who is doing the parsing and re-writes the index name? This can be handled by the UF in a very efficient way. Also, by using a batch stanza you rely on the IOPs performance of the disk, because each file will be deleted after it has been read adding wait-time to the overall performance. The current concept makes it really hard for the universal forwarder to perform the best, and by adding more and more data it will not work better. Try to question if this is the best way for the universal forwarder to read the amount of data you produce and then change the way the data gets ingested. cheers, MuS ... View more

Hi othersider2, check splunkd.log of the universal forwarder for error messages related to this file. Most likely a permission issue, and the UF is not able to delete the file. cheers, MuS ... View more

Look @pjamesburwell a lot of us on answers are Splunk users, been through almost every possible use case, and just try to be helpful by providing information or answers. There is no way for a universal forwarder to trigger an alert action on its own. BUT, and this is me being pedantic here 😉 you could place the alert action script into c:\program files\SplunkUniversalForwarder\bin\scripts and call it directly from the search head using UNC path names (given you can connect from the search head to the server running the universal forwarder, and the script is allowed to restart a service). You see, again the restart is done outside of Splunk like @nickhillscpl perfectly explained. If you need a more solid alternative to this, have a look at Phantom https://www.splunk.com/en_us/software/splunk-security-orchestration-and-automation.html which is a tool for orchestration and automation and could do this. cheers, MuS ... View more

Hi stamstam, some time ago a customer had a similar problem when they used an universal forwarder on a 2 CPU server to read over 10 millions directories - long story short: because it took the UF too long to list all directories the actual log files never got read because they hit the ignoreOlderThan options set. The main reason for this is the universal forwarder uses a so called Breadth-first search to discover any files and directories (Technical details can be found here https://en.wikipedia.org/wiki/Breadth-first_search). An easy understandable image of the process looks like this (it is from the wikipedia page) The cause for this behaviour was the use of a single wildcard monitor stanza on the top level directory. The solution was to use very specific monitor stanzas instead, and the performance increased immediately and the customer never had a similar issue again. Hope this helps ... cheers, MuS ... View more

Happy new year pjamesburwell, As I wrote before The most important question would be how you can restart the service remotely and how it can be scripted. So, If you can restart the service using RCP, use RCP - If you can restart the service using Powershell, use Powershell. No, it does not use the Splunk management port 8089 - this port is only for Splunk internal activities. Another thing I just noticed, you will not be able (as far as I know) to do such a thing on the forwarder. Your alert must run on a full Splunk enterprise instance and that instance will use the remote service restart script to remotely restart the service. Hope this helps ... cheers, MuS ... View more

Two options here: Download the app and try it or contact the app owner 😉 cheers, MuS ... View more

The OP have asked How do I pass the server name to the script? so how can this not be the answer to this question? cheers, MuS ... View more

Hi pjamesburwell, you might want to check the alert action section in the docs https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModAlertsIntro to read what is possible. The most important question would be how you can restart the service remotely and how it can be scripted. Another option would be to check out https://www.splunk.com/en_us/software/splunk-security-orchestration-and-automation.html which is purpose built for such use cases. Hope this helps ... cheers, MuS ... View more

Hi nick405060, I don't want to claim any knowledge here, but this awesome answer https://answers.splunk.com/answers/657104/sccm-queries-for-integration-with-splunk.html of @rich7177 helped me a lot lately to get all the things from SCCM into ES. Hope this helps ... cheers, MuS ... View more

Hi HattrickNZ, a completely different approach would be to use a SPL that will show a text instead of No results found . Here is an example : index=_internal sourcetype=splunkd use anything here | stats count by sourcetype | append [| stats count | eval sourcetype=if(isnull(sourcetype), "Nothing to see here, move along!", sourcetype)] | streamstats count AS line_num | eval head_num=if(line_num > 1, line_num - 1, 1) | where NOT ( count=0 AND head_num < line_num ) | table sourcetype count This will produce something like this if something matches: Hope this helps ... cheers, MuS ... View more

I updated the answer to reflect the working search. Please accept this answer so it is marked as answered, and others can benefit from it as well 🙂 cheers, MuS ... View more