The use case seems simple enough: Lets say we have index sensitive_data that contains... sensitive data. We want to ensure that ONLY role data-team has access to this index. How to do this reliably? There are a few ways to achieve this, but none are reliable or convenient: We could edit our [default] role in authorize.conf to include all the other indexes, but exclude sensitive_data This is not a great approach for a few reasons: It adds extra overhead: Adding a new index for general consumption has an additional step (add the new index to the white list) It is error prone: Everyone who admin user permissions needs to know NOT to use All internal indexes because you could inadvertently expose the sensitive_data index. We could of course set up scripting to deal with the extra step, and alerting to catch the occasional whoops. But those are just hacks. We could use a search filter to to say index!=sensitive_data and add that to our [default] role (This approach is not recommended for some reason, I have not experimented with it personally. I am curious to know why, or what problems this could cause.) We could index this sensitive data on a separate indexer. (Our forwarders would then be tied to that indexer specifically.) Ideally, we could restrict access to an index such that no other settings could override this restriction, either inadvertently or otherwise, unless it was clear to the admin making the change what the effect was. I found several other answers here that deal with this question but none are very current, so I thought it would be worthwhile to bring it up again. http://answers.splunk.com/answers/170362/how-to-restrict-user-access-to-a-new-index.html http://answers.splunk.com/answers/80812/what-is-the-precidence-for-excluding-an-index-from-a-role-using-the-gui.html http://answers.splunk.com/answers/32940/restrict-index-access.html Does anyone know a better way of doing this? Is it Splunk's express intention that Splunk not contain highly sensitive data where its exposure would amount to a critical breach? I'm sure there are Splunk customers that deal with PII, PCI, HIPPA, SOX extra, so how do you guys deal with this issue? Thanks in advance ... View more

I am using a scripted input from ausearch to get logs from audit.d inputs.conf [script://./bin/get_ausearch.sh] sourcetype=linux_audit interval=* * * * * get_ausearch.sh sudo /sbin/ausearch --start recent -k testing I have tested this with splunkd running as both root and as splunk (which is in sudoers ) and I get the same result. The result I get in Splunk is 11-05-2014 10:17:00.074 -0600 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/myapp/bin/get_ausearch.sh" This is actual output from ausearch (note the ERROR and the ``) it is just not the correct output. Simultaneously I can manually run the script (or copy the command verbatim) and get the correct results I expect to see. I am also redirected stdout, stderr to files and got the same results. Any idea what is going on here? NOTE I could, of course, monitor the audit.log file itself but I want to filter on the key, and not index all of the audit events. I also realize that the suggested approach is to use the rlog.sh from Splunk Add-on for Unix and Linux but this is very narrow and specific monitoring use case, so I am trying to come up with the lightest approach possible. ... View more

My forehead is sore from banging it on my desk. Please help. I cannot get scripts to run from an alert. The following is all the relevant info I have been using to trouble shoot this. Implementation To make it easy, I configured the alert to run "always" so the search term is irrelevant. The alert should be triggered every minute regardless. (I have also ran this with a search that I was manually triggering events in) The powershell script #!C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe "hello world" | Out-File -FilePath .\hello_ps.txt The savedsearches.conf stanzas [script_test_ps] action.script = 1 action.script.filename = test.ps1 alert.digest_mode = True alert.suppress = 0 alert.track = 0 auto_summarize.dispatch.earliest_time = -1d@h cron_schedule = * * * * * enableSched = 1 search = sourcetype=nothing Verification We know the search is running, we can see the search run and we can watch the script being called but it never actually executes. From python.log 2014-10-02 18:19:03,171 Central Daylight Time INFO runshellscript:188 - runshellscript: ['C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe', 'D:\\Splunk\\bin\\scripts\\test.ps1', '0', 'index=testing sourcetype=script_test', 'index=testing sourcetype=script_test', 'script_test_ps', 'Saved Search [script_test_ps] always(0)', 'https://splunkweb.domain.tld/app/search/@go?sid=scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935', '', 'D:\\Splunk\\var\\run\\splunk\\dispatch\\scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935\\results.csv.gz'] From scheduler.log 10-02-2014 18:19:03.889 -0500 INFO SavedSplunker - savedsearch_id="neil.peterson;search;script_test_ps", user="neil.peterson", app="search", savedsearch_name="script_test_ps", status=success, digest_mode=1, scheduled_time=1412291940, dispatch_time=1412291942, run_time=0.562, result_count=0, alert_actions="script", sid="scheduler_bmVpbC5wZXRlcnNvbg__search__RMD59d824a49b6b738b2_at_1412291940_27935", suppressed=0, thread_id="AlertNotifierWorker-1" Troubleshooting Powershell execution policy is unrestricted PS D:\Splunk\bin\scripts> Get-ExecutionPolicy Unrestricted Splunkd is running as a service account PS D:\Splunk\bin\scripts> Get-WmiObject win32_service | Where-Object {$_.name -like "splunk*"} | Select-Object name, startname name startname ---- --------- Splunkd domain\svc.splunk splunkweb domain\svc.splunk The service account is in the Administrators groups The troubleshooting steps from this wiki article: http://wiki.splunk.com/Community:TroubleshootingAlertScripts Is my scheduled search running? YES. I see it in scheduler.log Is my scheduled search generating the expected results? YES. I can watch the results come in on a real time search. I have also scheduled the alert to run "always" to make the search part of it irrelevant. Is my alert action being triggered? YES. I have added email actions and I get those emails, as well as watching it in https://splunkweb.domain.tld/en-US/app/launcher/job_management?savedSearch=script_test Is my alert script working? YES. I can run it from the command line, as the svc.splunk user, but when the alert triggers it, nothing happens. Reference I have been using the following documentation. I have read it and reread it. http://docs.splunk.com/Documentation/Splunk/6.1.3/alert/ConfiguringScriptedAlerts http://docs.splunk.com/Documentation/Splunk/6.1.3/Alert/Setupalertactions#Run_a_script http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/Savedsearchesconf http://wiki.splunk.com/Community:TroubleshootingAlertScripts I am out of rope on this one. Any help is appreciated. I have also done all of the above with a batch script as well with the same disappointing results. I have been testing and troubleshooting this from lots of different angles. I tried to break everything down to the simplest example, but my copy pasta above may still have gotten mixed up. If so please point it out and I will verify my testing results. EDIT: For posterity sake, I will explain the (very very very elementary and obvious) solution here. The script was writing out to $splunk_home\bin not $splunk_home\bin\scripts where the script was located. If you browse to the scripts location and run it, it obviously works, but when Splunk runs it, those relative paths do not exist from bin/ The other quirk I have noticed is that when Splunkd is running as LocalSystem, it can execute batch scripts but not powershell. When Splunk is running as a service account the powershell scripts work fine. To wrap your ps1 up into a bat you should put the folllowing in the batch script. C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -File .\location\ofyour\script.ps1 The above will look for $splunk_home\bin\location\ofyour\script.ps1 which is what we want. ... View more

When use the delta command I get results like this Value delta(Value) what-I-want-it-to-be 1 0 / 1 -1 0 2 -3 1 5 -4 3 9 / 4 Here, delta(n) is value(n)-value(n+1).. that is to say it is calculating the difference of the next value, not the previous. I want it to be delta(n) = value(n)-value(n-1). This is normally what I think of when someone says "delta"... the change since the last value, not the change that is about to happen. Am I using delta wrong? Is there away to use it to calculate past change, not future change? Returns negative values looking ahead, sorted oldest to newest eventype=myevents | delta Value | sort + _time Returns positive values looking back eventype=myevents | sort + _time | delta Value ... View more