Hi waddellt, please check this article to troubleshoot the kvstore https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/TroubleshootKVstore Per the article it seems that your failed kvstore message is related to: failed - Failed to bootstrap and join the search head cluster. if you are working on a Splunk Enterprise Security search head cluster you can also run a command to resync or if it did not work, clean-up the kvstore for this particular server. try first : - Resync kvstore (https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/ResyncKVstore#Resync_stale_KV_store_members) - splunk resync kvstore [-source sourceId] Note: if you are running on a cluster, please manual run a backup on the kvstore from a note that kvstore is working properly, check this procedure here(https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/BackupKVstore) Or if you are running on a stand alone instance you can clean the kvstore. Please be carefully, because it will reset all the data into the kvstore and you can lose the data that was there. On the previous link I provided, you have the both commands. splunk clean kvstore --local ... View more

check this link with information about how you can setup the proxy and edit the configuration from CLI. you have to edit the service_now.conf at $SPLUNK_HOME/etc/apps/Splunk_TA_snow/local. If the file is not under local folder, you can copy the service_now.conf from default to local and run the proper configuration https://docs.splunk.com/Documentation/AddOns/released/ServiceNow/Setuptheadd-on ... View more

Hi shrinkhalarana, I believe this error can be related with bucket issue, you can try to repair the bucket using this command for single bucket. the splunk service on indexer have to be stopped in order to repair a bucket. run splunk rebuild for further information check this document https://docs.splunk.com/Documentation/Splunk/7.3.2/Indexer/Bucketissues#Rebuild_a_single_bucket ... View more

Per my understanding, it seems your search head is setup as indexer at management console. You have to change the search head server role to only search head. If the server role is not properly defined at MC, the server will be added to other role, in some cases you have to manually changed it. Management Console/Settings/Forwarders/General Setup/Actions/Edit and change to "search head" role After run this configuration, save it and restart the MC and check if the server is setup to "search head" role. It is considered a best practice to forward all search head internal data to the search peer (indexer) layer. Check the document below https://docs.splunk.com/Documentation/Splunk/7.3.2/DistSearch/Forwardsearchheaddata Make sure your search head clusters is f ... View more

I believe on this moment, you should choose to use python2. As a previously mentioned, you can use the old eventgen version on Splunk 8. I believe it will does fit your needs to run your demo. ... View more

10-29-2019 10:49:45.581 +1100 ERROR ModularInputs - Unable to initialize modular input "modinput_eventgen" defined in the app "SA-Eventgen": Introspecting scheme=modinput_eventgen: script running failed (exited with code 1).. ... View more

It is a pleasure share the knowledge. Happy Splunking! ... View more

you have to edit the server roles at MC and setup the search head hosts to search head. Go to Management Console/Settings/Forwarders/General Setup/Actions/Edit and change to "search head" role As you are working on an indexer cluster environment, the data is being replicated to the entire indexer cluster, so if one of indexers cluster peers went down, you are still able to search the data. I recommend for you to read this document to understand how indexer replication works https://docs.splunk.com/Documentation/Splunk/7.3.2/Indexer/Aboutclusters ... View more

On Splunk Cloud there are some constrains, but I did not find it on the Splunk Cloud document about restrictions to output the limits of rows as a result. For further information ->https://docs.splunk.com/Documentation/SplunkCloud/8.0.0/Service/SplunkCloudservice#Splunk_Cloud_service_limits_and_constraints For sure, Splunk Cloud should have setup some configuration to avoid long running searches. It is required to keep a good performance on Splunk Cloud. I suggest to open a case with splunk support to report the issue you are having and also request for them about the parameters configuration for your environment. I also suggest a change to your report in order to get only the required fields for this report using the fields command. Please tweak your report to see if there is any other improvement you can apply. It can be something like this: index=firewall action=allowed | fields _time, srcip, dstip, srcport, dstport, action | table _time, srcip, dstip, srcport, dstport, action | lookup ip_intel ip as srcip OUTPUT description threat_key domain | search description!="" | table _time srcip dstip srcport dstport action description domain threat_key ... View more

I am running on MacBook and I had similiar issue. I also added the python configuration required for the new Eventgen version "Note: Please set "python.version = python3" in "server.conf" in $SPLUNK_HOME/etc/system/local in [general] stanza" and it did not work either. I am still running Splunk 8.0 version, but I redeployed the previous SA-Eventgen version = 6.5.1, and I remove the python3 configuration from server.conf, and it is working using the old python version, but I believe this new eventgen version is running on any issues that I am not able to figure it out. I hope this can help you. ... View more

the space for dispatch files is full. It is possible that splunk is not able to cleanup the files that is being generated, check this article, this will assist you to cleanup the dispatch directory or even adjust the limits.conf for this configuration. -> https://answers.splunk.com/answers/389879/dispatch-directory-is-full-how-do-we-clear-it-up.html ... View more

when I typed the response was missing this part, here is the procedure To thaw an archived bucket: – Copy the bucket directory from the archive to the index's thaweddb directory – Stop Splunk – Run splunk rebuild path to bucket directory Also works to recover a corrupted directory Does not count against license – Start Splunk Data in thaweddb is searchable along with other data, is not frozen, and does not count against index total size – Delete the bucket directory when no longer needed and restart Splunk ... View more

Please go to the manage apps and browser for SA-Eventgen and check if on the app configuration is setup yes to visible, if not, edit properties, select yes to visible. If you are still not able to see the app, restart splunk service. ... View more

per my reseach, this is the process where you setup the limits to the entire splunk environent https://docs.splunk.com/Documentation/Splunk/7.3.2/Search/Limitsearchprocessmemoryusage#Enable_a_search_process_memory_threshold limits.conf definition enable_memory_tracker = * Specifies if the memory tracker is enabled. * When set to "false" (disabled): The search is not terminated even if the search exceeds the memory limit. * When set to "true": Enables the memory tracker. * Must be set to "true" to enable the "search_process_memory_usage_threshold" setting or the "search_process_memory_usage_percentage_threshold" setting. * Default: false search_process_memory_usage_threshold = * To use this setting, the "enable_memory_tracker" setting must be set to "true". * Specifies the maximum memory, in MB, that the search process can consume in RAM. * Search processes that violate the threshold are terminated. * If the value is set to 0, then search processes are allowed to grow unbounded in terms of in memory usage. * Default: 4000 (4GB) search_process_memory_usage_percentage_threshold = * To use this setting, the "enable_memory_tracker" setting must be set to "true". * Specifies the percent of the total memory that the search process is entitled to consume. * Search processes that violate the threshold percentage are terminated. * If the value is set to zero, then splunk search processes are allowed to grow unbounded in terms of percentage memory usage. * Any setting larger than 100 or less than 0 is discarded and the default value is used. * Default: 25% ... View more

Sorry I pasted the wrong regex "event_type=(?\w[a-zA-Z-]+)\D+(?\d*) ... View more

please try this one: "event_type=(?\w[a-zA-Z-]+)\D+(?\d*) https://regex101.com/r/4HkCyb/1 very good site to test regex expressions -> www.regex101.com ... View more

you have an option to group the eventcode when there are having the same message pattern. blacklist1 = EventCode="(566|4662)" Message="Object Type:(?!\s*groupPolicyContainer)" blacklist2 = EventCode="(4634|4907)" Message="Account Name:(\W+\w+$)" Here you have a document with more details about blacklist windows events https://gist.github.com/automine/a3915d5238e2967c8d44b0ebcfb66147 you also have an option to use the add-on because all the configuration is already there, so you can also customise this add-on with those new blacklist ... View more

The issues I had and fixed with Icacls was related with system local account on windows servers. Which is the user you are re-applying those permissions? I believe you have to run this command as administrator. I found this other command from this link -> https://stackoverflow.com/questions/2928738/how-to-grant-permission-to-users-for-a-directory-using-command-line-in-windows icacls "c:\somelocation\of\path" /q /c /t /grant Users:F F gives Full Access. /q /c /t applies the permissions to subfolders. In some cases, I have to move splunk to other folder, re-apply the permissions, moved again to original folder, and re-deploy the permissions again. I am not really sure why this happen, but it seems something is prevent or holding the folder to received the correct permission. Make sure the user you are running the command has the proper access to change the permission, as stated on the document, you have to run the permissions as administrator. ... View more

the age of data is defined as the retention police. Check this additional link where you can have more information about it -> https://answers.splunk.com/answers/553180/how-to-find-the-retention-period-of-an-index.html ... View more