This lead to a solve for me. I had our user change the time format at the beginning of their logs.  ... View more

I've been ignoring this number for 8 or 9 years now as it's never made any sense and doesn't correspond to the results of the queries mentioned above.   How does this counter even work? ... View more

@jamie00171    Upon further testing, it seems that doing EVENT_BREAKER_ENABLE = true EVENT_BREAKER = "([\r\n]+)" Works, but only if accompanied by also doing force_local_processing = true   Are the event breaker flags ignored if there's no force local processing? ... View more

Thanks for the answer, looks like using the token name $application$ also does work, but I assume $value$ is the better way to go. But yes, I just left off data to keep it easier to read what I was actually asking for. My dashboard has a lot more going on than just these elements, but this was ultimately asking the question in the simplest format possible. ... View more

I just wanted to throw out a "me too" on this and also offer up that I believe the issue lies down further into the code than what nickhillscpl had shared in his "answer" (which is also why I think the issue got buried further down as well with the error checking they added.) The section of code it is really stemming from is the line: total_records = sc.get_total_records_for_vuln(scan_id) This function call looks like this: def get_total_records_for_vuln(self, scan_id): args = {'type': 'vuln', 'sourceType': 'individual', 'scanID': scan_id, 'query_type': 'vuln', 'query_tool': 'listvuln', 'query_view': 'all'} self._build_query(None, args) args['query']['startOffset'] = 0 args['query']['endOffset'] = 0 result = self.perform_request('POST', 'analysis', args) return int(result['totalRecords']) Inside this is the "perform_request" function which basically is what makes the actual REST API call to the Security Center server. I had already modified my own code block in here to include a little bit of "error" help on the request call itself, but this only seems to control for HTTP errors themselves and not for the API errors which are giving back "valid" responses, but the server is rejecting it. ... View more

The below powershell script will use the commandline tool "sc.exe" to output the service name and status, and then format the results with a simple echo output so Splunk can read it as an input. $output = sc.exe query state= all $formatted = new-object PSObject for ($i=0; $i -lt $output.Length; $i++) { if ($output[$i] -like "SERVICE_NAME:*") { $service = $output[$i] -replace 'SERVICE_NAME: ','' $status = $output[$i+3] -replace ' STATE : ','' $status = $status -replace ' ',',' $newout = $service + "," + $status echo $newout } } Sample results: smphost,1,STOPPED SmsRouter,1,STOPPED smstsmgr,1,STOPPED SNMPTRAP,1,STOPPED SplunkForwarder,4,RUNNING Spooler,4,RUNNING sppsvc,4,RUNNING ... View more

On a Unix machine, I would just use logrotate. It looks like people have written this functionality in a Power Shell script: http://stackoverflow.com/questions/16795463/using-a-rolling-log-file-within-a-powershell-script-that-gathers-performance-cou ... View more

Took a while to figure out how to restart the UF from a scripted input. I tried many options, in most cases the forwarder stopped but never started again. Not an option. The root cause is that a stop of splunkd on Windows forces the parent process of the powershell script itself to stop executing. This solution works (tested with UF version 6.3.3 an 6.4): I used a .path file, a powershell script (.ps1) and a command file (.cmd). Put all three scripts in the same app/bin folder. The path file enables bypassing the default security Powershell security policy, this (normally) prevents the .ps1 from executing when called directly from inputs.conf. The .ps1 is the do something app, the cmd file is necessary to spawn a separate process (not a child of splunkd) that will restart the forwarder. in YOUR_APP\default\inputs.conf on the forwarder: [script://.\bin\<myscript.path>] interval = -1 disabled = 0 myscript.path: $SystsemRoot\System32\WindowsPowershell\v1.0\powershell.exe -ExecutionPolicy Bypass -command " &'$SPLUNK_HOME\etc\apps\YOUR_APP\bin\<myscript.ps1>' " The file myscript.ps1 code to disable script input in inputs.conf You really need this code to prevent boot loops. Do not put other stuff in this inputs.conf, without modifing this code. It will replace every = 0 to = 1 present in the inputs.conf. You might place other inputs.conf stanza's in a different inputs.conf. $scriptdir = Split_path $script:MyInvocation.MyCommand.Path $inputsconf_file = resolve-path ($script_dir, "..", "default", -join "\") $inputsconf_file = $inputsconf_file, "inputs.conf" -join "\" (Get-Content $inputsconf_file) | Foreach-Object {$_ -replace " 0", " 1"} | Foreach-Object {$_ -replace "=0 ", "=1"} | Foreach-Object {$_ -replace "false ", "true"} | Out-File $inputsconf_file The file myscript.ps1 code to initiate a restart: $reset_script = $script_dir, "ufrestart.cmd" -join "\" $args = "SplunkForwarder" start-process -FilePath $reset_script -Arguments $args the ufrestart.cmd: @echo off sleep 5 net stop %1 net start %1 exit Have fun! ... View more

Hey Javier, don't worry about it. It's a classic case of concurrency. I also, feel that sometime the notifications comes way late. ... View more

Good result fairje. It is strange and I wonder if there is a bug here. I found an old comment by "itinney" here: https://answers.splunk.com/answers/38753/regex-for-multiline-events.html He indicates that uses (?m) seems to behave like using (?sm), i.e. (?s) gets tuned on if (?m) is used. Note, I've not proved this but it would be strange behaviour as it defeats the purpose of using (?m) which is to cause ^ and $ to match the begin/end of each line (not only begin/end of string). Something to watch out for anyway. ... View more

Yeah, looks to be that way, having messed with it a couple times. The alternative name was what was causing the issues. I have accepted my own answer for that reason and will make a comment on the docs page. ... View more

Had read all through it, and I think it was throwing me off since I was afraid that the "tcpout-server://" setting would maybe somehow keep it from load balancing correctly. by instead duplicating all the data equally to the servers. I will give this a go and see if that works. I am just not sure the best way to validate that it is load balancing and not sending the data out to all of my indexers at the same time (e.g. instead of it logging something once, I would guess to find my data copied 4 times over on my 4 indexers.) ... View more

Thanks, Below command fix that issue. chmod 400 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key I was not able to open Splunk Add on builder TA due to this issue. Now its been fixed by changing the permission to the mongo key file. ... View more

All your indexers (really all of your servers) should be showing up in your license master under whatever pool (default pool is the catch all) you have specified. For example I have 14 servers registering under my default license pool, although I only have 4 indexers. As long as you turn off indexing on everything else this should be fine and you will only see hits against your license from your indexers. I would validate the settings on your servers that are not showing up in the pool to see if maybe you typo'd a setting. From the GUI go to Settings - Licensing - Switch to Slave. If you are already set as a slave, change yourself to a local master, and then reset it back as a slave making sure your settings are correct. (alternatively, these settings are stored in your server.conf file). As for the licensing itself, if things are not balanced across your indexers evenly I would check your outputs files across the deployment to ensure that you are appropriately set to auto load balance across all four indexers. It could be that the events are going to one server over another because of this. If these settings are valid and you are still seeing it unevenly balanced, check your splunkd.log and verify that you are not showing errors in TCPOut / TCPIn that says you were unable to connect to X server. Sorry this one is a bit all over the place, since it could be a couple of different issues. Now if you mean you are actually going over your license, you might want to talk with your Splunk Account rep about pulling your events under the license or getting a larger license. Hope this helps! ... View more

I also had same issue while upgrading ES from 3.3 to 4.1. ./splunk search '| testessinstall' worked for me. Awesome and thanks for the workaround. ... View more

looks like in this case, you'll just have to use a wildcard earlier in your statement. Will it still work if you use: [monitor:///opt/log/192.168.*.(37|38|39|40)/Juniper.log] I have consistently run into issues with tailing wildcards when configuring multiple file monitors to act recursively from a higher point in the directory tree. ... View more

I'm right there with you Bryan... To get past this I searched through all the SA's and TA's looking at the inputs.conf.spec files. I found that the stanza Splunk_TA_vmware was expecting lives in the SA-VMNetAppUtils inputs.conf.spec file (SA-VMNetAppUtils/README/inputs.conf.spec). I copied the necessary stanza from there, made a backup copy of the inputs.conf.spec file in Splunk_TA_vmware, and then pasted the missing stanza into the inputs spec file. From there I ran my cluster bundle apply and saw no errors. For reference, here is the missing stanza: `[dm_accel_settings://default] * Configure an input for enforcing data model acceleration settings automatically * when another app is installed. Note that only one app at a time should define * the acceleration settings for a given data model. Usually this is the "primary" * app for the suite of apps on the system, e.g. SplunkEnterpriseSecuritySuite. acceleration = [true|false] * True if the data model should be accelerated. manual_rebuilds = [true|false] * If true, changes to the datamodel will not automatically rebuild the HPAS * data store.` ... View more

Ah, makes sense. probably not the right place for this, but this App hasn't been updated by the owner since like, 2011 (doesn't even state that it supports Splunk version 6) is there anyway that if I get this thing working fully to submit updates to the base app? ... View more

Hi , Can someone please help me. I rain in heavy forwarder the above script but Splunk is not starting if it is down. I stopped the splunk and ran the script manually. chmod +x filename ./filename Then I am getting like below "Starting splunk (via systemctl):". Splunk is not getting started. Please help me on this immediately. It is a unix machine and I tried $ service splunk status but it failed. Starting splunk (via systemctl): [ OK ] Starting splunk (via systemctl): [ OK ] Error occurred connecting on port 8089 for splunk Starting splunk (via systemctl): [ OK ] Starting splunk (via systemctl): [ OK ] Starting splunk (via systemctl): [ OK ] Error occurred connecting on port 8089 for splunk Starting splunk (via systemctl): [ OK ] ... View more