I am running following search query to obtain history of triggered alerts (time, name, severity), manually:   index=_audit action=alert_fired ss_app=* | eval ttl=expiration-now() | search ttl>0 | table trigger_time ss_name severity | rename trigger_time as "Alert Time" ss_name as "Alert Name" severity as "Severity"   I am observing following strange behavior: I am running this for "all time", yet I can never match timestamps for two searches. What I mean by this is that if I search for value of `trigger_time` for alert that triggered in May, and search for that very same value in all time export generated in June, I will not find it. It is not a single occurrence. Whichever record I randomly select, I am unable to find in other files exporting seemingly the same data. In case it is important: I have 3 SeachHeads in a cluster. Is this a bug? (v8.0.2.1) If this is not a bug, how do I make Splunk to provide precise time of when alerts were triggered? ... View more

I am using EC2 to run my search heads. I want to configure Splunk Addon to start delivering alerts when they trigger to specific SNS Topic (ideally on per-alert basis). Where do I do this? ... View more

Architecture: 3 search-heads in a cluster, captain dynamically elected, 1 search head deployer that is not one of the search head servers. I go to $SPLUNK_HOME/etc/shcluster/apps/, create my app, add local/data/ui/{nav/default.xml,views/(empty)} that is valid (per https://dev.splunk.com/enterprise/docs/developapps/createapps/addnavsplunkapp/) Push type is merge_to_default - so when I run splunk apply shcluster-bundle command, it should merge the local folder into default and deploy that. That doesn't happen, however. When I check the app on SearchHeads I see that the app is deployed, but there are only two folders inside: default with app.conf:     [ui] is_visible = 1 show_in_nav = 1 label = Testing Label [launcher] author = myname description = Some label version = 1.0 [install] install_source_checksum = <digest>     The second folder metadata contains single file metadata.local with nothing particularly interesting (except version that I am running on - 8.0.2). Note default.xml (or any other UI stuff) is nowhere to be found. When I open the app on the search heads UI, I can select it from navigation and I am automatically redirected to Appname/Inputs despite this is not defined anywhere in my default.xml that I created on SHD.   I dig a little bit and found that /Inputs are defined in Splunk for AWS Addon's default.xml that is apparently being used for the application. I can also make my search head app to use my default.xml if I manually create local folder on each of the search heads - but I don't find this approach good from the maintenance perspective. My questions: 1: Where should I put my default.xml on SH deployer for it to be deployed and used on Search Heads? 2: Why does SearchHead automatically assigns default.xml from different app? Can I configure this somewhere? Can I maybe create a new default.xml to be selected to be used for this app (and future apps I create)?   Thank you for your time! ... View more

I have 3 search heads, 1 search head deployer. I need to create "a new app"  for which I will create a role that will be able to query few indices that are assigned to it in resources tab. I need to create roles on SearchHead, because I have SAML auth setup there. When I create this new app on one search head (via UI -> Manage Apps -> Create App), I do not see it available on other search heads, or in the list of resources when I try to create the role. When I create this new app on the Search Head Deployer, I cannot see it on SHs when creating role. I cannot make `splunk shbundle-deploy` (or whatever the command actually is), when I try, it fails, because splunk tries to create something in `/dev/null/.splunk` (it runs under system account which does not have shell access; so I guess that's related to that and it cannot change). Where do I create the app and how do I push it to search heads, so I can see it there? Thanks for your time. ... View more

Hello, I have a Heavy Forwarder on which I receive logs via Splunk for AWS addon as they appear in my S3 bucket. I know I will be receiving log files ending with `*_connectionlog_*.gz`, `*_userlog_*.gz` and `*_useractivitylog_*.gz`. My current input definition looks like this: [aws_sqs_based_s3://MyApp_RedshiftAuditLogs] aws_account = redacted index = myapp_redshiftindex interval = 300 s3_file_decoder = CustomLogs sourcetype = aws:cloudtrail sqs_batch_size = 10 sqs_queue_region = redacted sqs_queue_url = https://redacted.url disabled = 0   The problem is that there is a `sourcetype` field, that has single value here. I would like to assign this value based on whether there is useractivitylog, connectivitylog or userlog in the filename that just came in on the heavyforwarder. The vision here is that I will further research how to properly parse and extract each of these types of logs, so when I will be eventually searching the index, I will have extracted some (if not all) of the fields - but I am not at this stage yet. Questions: 1. Am I approaching this correctly by wanting to assign different source types to files that are structured differently? 2. How do I do this assigning thing? 3. Will the path that you propose enable me to write some parsing/extraction logic later down the road? Thank you for your time! ... View more

I have taken over deployed Splunk with Master node, several indexers and search heads. I want to update TLS cert for web splunk we have, so I place them into folder wherethey belong and want to restart splunkweb only. I run ps aux | grep "splunk"  to see what user the splunk services run under - it's splunkadmin. I navigate to $SPLUNK_HOME/bin and try running both of these:     sudo ./splunk restart splunkweb # prompts for authentication (which I do with administrator account I confirm that exists in $SPLUNK_HOME/etc/passwd # gives me simple output: Can't create directory "/root/.splunk": Permission denied sudo -u splunkadmin ./splunk restart splunkweb # i authenticate as above and receive: # Can't create directory "/dev/null/.splunk": Not a directory       Can you think of a different way to restart only splunkweb? And if not, can you help me figure out what is the problem here? Where do I find the logs that tell me more about the error that I get?   Thank you for your time and help. ... View more