Hey Splunk developers,   Quick question for you guys, I'm curious if it would be considered against best practices, or an anti-pattern to add your own code to the props.conf / transforms.conf file of a Splunk supported technical addon in order to add support for your own custom sourcetypes? Specifically, The Splunk Addon for Microsoft Cloud Services. My use-case is this, I'm utilizing a "custom" event delivery mechanism to send Azure Diagnostic Logs generated by my resources in my Azure tenant to Splunk over HTTP Event Collection (HEC). The mechanism consists of an Azure Event Hub that receives diagnostic events from my resources in Azure. Downstream an Azure Function App monitors the upstream event-hub namespace for incoming events, parses those events, constructs a source-type based on the data in the parsed event, and then wraps the event payload in a HTTP request and sends it to Splunk. The code for the Azure Function App can be found here.    The general source-typing logic for the function application is below.   Functions that collect diagnostic log data attempt to construct a sourcetype based on the resourceId of the event. The logic for this sourcetype construction can be found in the getSourceType function in the ./helpers/splunk.js file. The following steps are used to construct the sourcetype: A regular expression is used to extract two groups after the text /PROVIDERS Example /PROVIDERS/MICROSOFT.RESOURCES/DEPLOYMENTS/ Periods (.) and forward slashes (/) are replaced with colons (:) The event category is appended Example An event with a resourceId of /SUBSCRIPTIONS/subscription ID/RESOURCEGROUPS/group/PROVIDERS/MICROSOFT.RESOURCES/DEPLOYMENTS/FAILURE-ANOMALIES-ALERT-RULE-DEPLOYMENT-12345678 will have a sourcetype of azure:resources:deployments:administrative If a sourcetype cannot be constructed from the event, the specified default sourcetype entered at setup will be used.     Here's my dilemma, the team that administers our corporate Splunk instance would like to utilize as many TA's as possible to ensure CIM compliance of on-boarded data with as little work as possible, completely understandable. Therefore, for certain event sources I've hard-coded the source-type logic in my function app to match several of the pre-defined source-types found in the TA I linked above ("azure:monitor:aad" & "mscs:azure:security:alert" as examples). However, we're at the point where I'm trying to on-board data that does not have a pre-defined sourcetype within the TA's we're utilizing. As an example, Azure Front Door Firewall Logs which will sourcetype to "azure:network:frontdoors:frontdoorwebapplicationfirewalllog". To work around this, I've suggested to our content development team that we begin adding logic to the props.conf and transforms.conf of the TA listed above that will match the sourcetypes my function-app is sending over to Splunk and I've essentially been told no, as this isn't best practice. Is my suggestion truly against best practice, or an anti-pattern? In my mind the only extra overhead to take into consideration is merging our custom props.conf / transforms.conf with the base TA whenever an update for the TA is released.   What's my best path forward here? ... View more

Afternoon Splunk Community,   I'm currently in charge of helping  sunset an old subsidiary of ours and putting their infrastructure out to pasture. As part of the decommission process I need to backup indexed data from their Splunk instance for long term retention for the purpose of restoring the data should we ever need to view it for any reason. The Splunk indexing tier consists of six total indexers across two sites, three indexers in site 1 and the other three indexers in site 2. The cluster master for the site has a replication factor of "origin:2 total:3" which means that each indexer in the site should contain two copies of each bucket that originated within the site, and a single copy of each bucket that did not originate within the site.  In an ideal world I think this would mean that I should only need to backup the data volume of a single indexer in the cluster to have a copy of all indexed data. However I've read that in taking a backup of a clustered indexer that there is no guarantee that a single indexer contains all data within the cluster, even with replication enabled. I have several questions: What is the best practice for archiving indexed data for potential restoration into another Splunk instance at a later date? Each of my indexer's "/var/lib/splunk" directory symlinks to a unique AWS EBS volume. Should I simply retain the AWS EBS volume from each of my six indexers, one indexer from each of my two sites, or can I retain a single EBS volume from one of my indexers and discard the rest?  My thought process behind this method is that if for any reason I ever needed to restore the archived data for searching I could simply setup a new Splunk indexer, attach the archived EBS volume, and point a search head to the new indexer in order to search the data. Is a better approach to simply stop splunkd on my indexer, create an archive of /var/lib/splunk using an archive utility and then restoring that archive to /var/lib/splunk on a new indexer at a later date if we need to restore the data for any reason? As a last resort, I could run a search against each of my indexes and export this data in a human readable format (.csv, XLS...etc.). I've already checked in all of my knowledge artifacts (apps, configuration files..etc.) in Git for record keeping. However, Is there any other portions of my Splunk deployment that I should consider backing up? For example, should I retain a copy of my cluster master's data volume for any reason? If I do need to restore archived data, can I restore the archived data to a standalone indexer, or do I need to restore the data to an indexer cluster? This one seems rhetorical to me, but I figured I should ask none the less.    Resources: Splunk - Back up Indexed Data  Splunk Community - How to backup/restore Splunk db to new system  ... View more

Hey guys,   I need some quick help creating a nested stats table and grouping by multiple values within that table. My data contains the following data points that I am trying to correlate / visualize: Client IP Address, Unique Hash ID, Unique Document ID, and the count that shows the number of times an IP Address accessed a Unique Hash ID, and Doc ID.   An example data set is:   192.168.1.1 (client IP), abcdefg1 (hash 1),  12948(DocID1), 129584(DocID2), 1029384(DocID3) 192.168.1.1(Client IP), abcdefg2 (hash 2), 10294 (DocID 1), 192.168.1.5(Client IP), abcdefg1 (hash1), 12948(DocID1), 1029484(DocID2)   I'm looking to create the following table to help visualize these relationships | Client IP Unique Hash Document ID Count     192.168.1.1 abcdefg1 12948 5         129584 10         1029384 15                   abcdefg2 12948 2         1029484 3                 192.168.1.5 abcdefg1 12948 1         1029484 4                   I've created nested tables before but I'm really stumping myself on this one. Any advice? ... View more

Good Afternoon Splunkers, Let me start by saying that I hope this is the right sub-forum for this question. I'm working on a dashboard within Splunk to visualize our AWS Web Application Firewall data. The purpose of this dashboard is to show general statistics and information about the requests our AWS WAF Solution is processing. Ultimately, we would like to use this dashboard to debug and tune our WAF solution as we move our WAF into enforcement mode. One of the many charts / tables I'm trying to put together is a list of AWS WAF Rule-sets, and their sub-rules that have been triggered, by website our WAF is monitoring. A concrete example of what I'm looking to create would be: Webpage WAF Rulegroups Triggered Sub-Rules Triggered Count SomeWebpage.com AWSManagedCommonRuleSet         GenericRFI_Body 5     SomeOtherVuln 10     NoUserAgent_HEADER 15           AWSAnonymousIpList         HostingProviderIpList 20   the biggest issue I'm currently facing is that the AWS WAF data, while in JSON format from AWS, does not follow proper JSON, Key/Value pairings, and has nested arrays containing multiple types of information. Specifically the nested array that contains the rule evaluation information for a particular request contains all of the rules evaluated, even if the rules did not match, or no sub-rules were fired. Example below,       ruleGroupList: [ [-] { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesCommonRuleSet terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesSQLiRuleSet terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesLinuxRuleSet terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesAdminProtectionRuleSet terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesKnownBadInputsRuleSet terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesAmazonIpReputationList terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesAnonymousIpList terminatingRule: { [-] action: BLOCK ruleId: HostingProviderIPList ruleMatchDetails: null } } ]       As you can see, even though only one AWS Rulegroup fired for this request "AWSManagedRulesAnonymousIpList", and within that group, the sub-rule "HostingProviderIpList" fired, all of the rule-groups assigned to the WAF are present within the array. Therefore, if I were to search for something like     $search stats count by nonTerminatingMatchingRules{}.ruleId, ruleGroupList{}.terminatingRule.ruleId | stats list(ruleGroupList{}.terminatingRule.ruleId), list(count), by nonTerminatingMatchingRules{}.ruleId     I would get back a list of each rule-set but I would also get back each sub-rule that has fired as well, even if the sub-rule is not part of the rule-set that fired. What commands can I use to transform this data into proper key-value pairs on a per rule-group basis? Based on what I've read I think I want to use "Spath" and "mvexpand", I'm just not sure of the best path forward. For full transparency, here's an entire WAF log in JSON format, so you can see all of the fields. Here's the guide for understanding these fields as well.     { [-] action: ALLOW formatVersion: 1 httpRequest: { [-] args: clientIp: 8.8.8.8 country: CA headers: [ [-] { [-] name: Authorization value: SomeToken } { [-] name: User-Agent value: Site24x7 } { [-] name: Cache-Control value: no-cache } { [-] name: Accept value: */* } { [-] name: Connection value: Keep-Alive } { [-] name: Accept-Encoding value: gzip } { [-] name: Content-Type value: application/json; charset=UTF-8 } { [-] name: X-Site24x7-Id value: Redacted } { [-] name: Content-Length value: 1396 } { [-] name: Host value: mywebpage.com } ] httpMethod: POST httpVersion: HTTP/1.1 requestId: RedactedID uri: /big/uri/path } httpSourceId: Redacted ID httpSourceName: ALB labels: [ [-] { [-] name: awswaf:managed:aws:anonymous-ip-list:HostingProviderIPList } ] nonTerminatingMatchingRules: [ [-] { [-] action: COUNT ruleId: AWSCommonRuleSet ruleMatchDetails: [ [-] ] } { [-] action: COUNT ruleId: AWSAnonymousIpList ruleMatchDetails: [ [-] ] } ] rateBasedRuleList: [ [-] ] requestHeadersInserted: null responseCodeSent: null ruleGroupList: [ [-] { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesCommonRuleSet terminatingRule: { [-] action: BLOCK ruleId: GenericRFI_BODY ruleMatchDetails: null } } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesSQLiRuleSet terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesLinuxRuleSet terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesAdminProtectionRuleSet terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesKnownBadInputsRuleSet terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [-] ] ruleGroupId: AWS#AWSManagedRulesAmazonIpReputationList terminatingRule: null } { [-] excludedRules: null nonTerminatingMatchingRules: [ [+] ] ruleGroupId: AWS#AWSManagedRulesAnonymousIpList terminatingRule: { [-] action: BLOCK ruleId: HostingProviderIPList ruleMatchDetails: null } } ] terminatingRuleId: Default_Action terminatingRuleMatchDetails: [ [-] ] terminatingRuleType: REGULAR timestamp: 1629751363362 webaclId: RedactedID }     ... View more

Hello Splunk Community, I could certainly use you help getting myself out of a rather large jam I'm in. I need guidance on how to properly re-route a subset of events generated by one of our applications running within a Docker Container before it is indexed by Splunk. Our development team updated the logging mechanism of this application to no longer write this subset of events to a flat log file. The subset  of events is now being written out to STDOUT/STDERR within the container itself. This means the specific events I need to extract are now being lumped into the same index as every other container processes events running on the same Docker Swarm. This is causing a problem for me as I am new to administering Splunk and struggling to understand if it's going to be possible to extract this specific subset of events I need while filtering out the noise of every other container process writing events to the same index.  To get data in from Docker, we are using the Splunk Logging Driver for Docker and have replaced "/etc/docker/daemon.json" config file with the following configuration file on all of our Docker Clusters.       { "metrics-addr": "0.0.0.0:9323", "experimental": true, "log-driver": "splunk", "log-opts": { "splunk-format": "json", "splunk-verify-connection": "false", "splunk-token": "TOKEN", "splunk-url": "URL-TO-SPLUNK", "splunk-insecureskipverify": "true", "tag": "{{.DaemonName}}/{{.Hostname}}/{{.Name}}/{{.ID}}" } }     The HEC Token we have configured for Docker events is as follows     [http://docker] disabled = 0 index = main sourcetype = hec:swarm token = $HEC-Token useACK = 0     Since we use the same HEC Token across all of our Docker Swarms, we have some Props.conf and Transforms.conf stanzas that were configured by an old co-worker to modify the index these events flow to based on source-type.   #props.conf [hec:swarm] SEDCMD-0_hec_tags_rename = s/tag/tags/g TRANSFORMS-0_hec_indexer = docker_change_index TRANSFORMS-1_docker_sourcetyper = docker_sourcetyper TRANSFORMS-9_clean_fields = clean_docker_sourcetypes KV_MODE = json ANNOTATE_PUNCT = false [(::){0}json:platform:*] ANNOTATE_PUNCT = false KV_MODE = json REPORT-vp_app_extract = vp_app_extract REPORT-docker_msghdr_extract = docker_msghdr_extract REPORT-docker_auth_events_extract = docker_auth_events_extract REPORT-docker_kv_extract = inline_kv_extract EVAL-action = case(event == "Authentication failed", "failure", event == "Authentication success", "success", true(), null())       #Transforms.conf # Extract Message Header [docker_msghdr_extract] REGEX = \,\d{3} [^A-Z]+(?<level>\S+)(?:[^\[]+\[){2}\d+m(?<process>[^\\]+) # Extarct Authentication Events [docker_auth_events_extract] REGEX = (?<event>Authentication [^\:]+) # Extract KV pairs [docker_kv_extract] REGEX = (\S+) = [']*([^,"']*) FORMAT = $1::$2 # Extract platform application [vp_app_extract] #REGEX = tags\":\"docker(?:[^_]+)_[^_]+_(?<app>[^\.]+) REGEX = tags\":\"docker\/(?:[^\/]+)\/(?<stack>[a-zA-Z-]+)_(?<app>[^\.]+) ####### SOURCETYPERS ######################### [docker_sourcetyper] DEST_KEY = MetaData:Sourcetype REGEX = tags\":[^\/]+\/(?:[^-]+-){2}([^-]+) FORMAT = sourcetype::json:platform:$1 ####### CHANGE INDEX ############################ [docker_change_index] DEST_KEY = _MetaData:Index REGEX = \"tags\":\"docker/([^-]+)-([^-]+)-([^-]+) FORMAT = $1$2_$3 ####### CLEAN ############################ # Remove '-' from sourcetype [clean_docker_sourcetypes] INGEST_EVAL = sourcetype=replace(sourcetype, "(-|_)", "")     The event subset I'm trying to re-route to a different index contains information in the following format. This is a raw example pulled from Splunk after the event was indexed, and has been sanitized as much as possible.           {"line":"$dockerContainerID,$YYYY-$MM-$DD:$HH:$MM:$SS,$AlphaNumericString,$AlphaString,$AlphaString,$AlphaString,$numericString,$AlphaString,$numericString,,,$DollarAmount,$AlphaString,$AlphaString,$AlphaString,$FloatingPointString,$FloatingPointString,$NumericString,$NumericSring","source":"stderr","tags":"docker/$DockerNodeHostName/$DockerStack_$StackApplication.1.0fj4pexdb3m16giqp1atrfco5/47fb3b6218d5"}     Based on our current configuration could anyone lend a guiding hand on my best path forward to extract this subset of events out and redirect them to their own index, if what I'm attempting is at all possible. I fear that it is not because every container process is writing to the same index, has the same, host, source, and source-type. Based on my understanding of how Props.conf stanzas are defined I'm not sure that I can define any rules that won't effect every event.   [<spec>] * This stanza enables properties for a given <spec>. * A props.conf file can contain multiple stanzas for any number of different <spec>. * Follow this stanza name with any number of the following setting/value pairs, as appropriate for what you want to do. * If you do not set a setting for a given <spec>, the default is used. <spec> can be: 1. <sourcetype>, the source type of an event. 2. host::<host>, where <host> is the host, or host-matching pattern, for an event. 3. source::<source>, where <source> is the source, or source-matching pattern, for an event. 4. rule::<rulename>, where <rulename> is a unique name of a source type classification rule. 5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed source type classification rule. These are only considered as a last resort before generating a new source type based on the source seen.   ... View more

Evening Splunk community, My organization practices Blue / Green data-centers and requires us to switch production data centers every quarter.  In my environment I manage two standalone Search Heads. One Search Head in each data-center, separated by region. I'm trying to determine a clean solution for keeping user knowledge artifacts (saved searches, reports, alerts..ect) synced across the two Search Heads without having to implement Search Head Cluster Replication. Does anyone have any tips, advice, or general best practices when it comes to keeping knowledge objects synced between two or more standalone Search Heads? I've read a few forum posts that cover this topic and I've detailed some of the solutions I'm brainstorming, but wanted to get everyone's opinion before I start down the wrong path. For starters I believe the knowledge artifacts on the Search Head reside under the following directories. Not including the saved searches within /etc/apps.     $SPLUNK_HOME/etc/system/local/authentication.conf $SPLUNK_HOME/etc/system/local/authorize.conf $SPLUNK_HOME/etc/users/* Everything in local folders and local.meta files under splunkhome/etc/apps   Take a complete backup of /opt/splunk and restore it to the standby Search Head as needed. Implement an rsync script on each search head I'd create an environment variable that indicates if the search head is currently the active search head for production workloads. If yes, push that SH's configurations / changes to the standby Search Head. Implement Search Head Clustering. While I believe this is the intended solution for what I need to achieve. I'd like to avoid this if possible as I'm our only Splunk administrator and from what I've been told there's a fair bit more management overhead to tackle. Implement a CI / CD pipeline that checks user knowledge artifacts into git as they change and then pushes the data out to the environment as needed I think this would be the ultimate goal for me as we're working to eliminate as much toil and as many landmines in our environment as possible. Are there any good blog posts or guides concerning managing and deploying via CI / CD in regards to Splunk? Create an EFS mount on both Search Heads and point the directories that could contain Splunk User Data to the EFS mount for both Search Heads to share.     References - https://community.splunk.com/t5/Deployment-Architecture/What-is-the-best-practice-to-backup-Splunk/m-p/201159 - https://community.splunk.com/t5/Splunk-Enterprise-Security/What-is-considered-a-full-back-up-of-Search-Head/m-p/351484 - https://lantern.splunk.com/hc/en-us/articles/360043720673-Managing-backup-and-restore-processes- - https://community.splunk.com/t5/All-Apps-and-Add-ons/Has-anyone-attempted-to-run-Splunk-with-indexes-stored-on-AWS/td-p/274270 - https://community.splunk.com/t5/Splunk-Search/How-to-get-all-users-searches-and-lookups-synchronized-to-both/td-p/394614 - https://community.splunk.com/t5/Deployment-Architecture/I-have-a-Splunk-Enterprise-Search-Head-in-a-Production-and-a/td-p/507154      ... View more

Evening Splunk Community, I'm a new Splunk administrator (<6MO) and I have a couple of questions I could really use your help finding the solutions to. Over the past several months my team and I have been working to lift and shift our production environments hosted in physical data centers up to AWS. We use AWS Cloudformation to spin up these environments from top to bottom. Ideally with as little manual work as possible. Ultimately I'm having trouble finding solutions on how to keep my Splunk deployment and configurations data-center agnostic as possible to achieve this goal. We currently have the following multi-site architecture deployed in AWS. Each Region consists of 2 primary VPCs, a production app VPC, and a shared services VPC. With the exception of some heavy forwarders the Splunk infrastructure in these environments is housed in the shared services VPC: Region A (Active): 1x Standalone Search Head 1x Combined Deployment Server / License Server / Monitoring Console 1x Cluster Master 3x Clustered Indexers (Clustered with region B for event replication) 4x Load Balanced Syslog-NG Collectors / Heavy Forwarder Combo Region B (Failover): 1x Standalone Search Head (Offline unless needed) 1x Combined Deployment Server / License Server / Monitoring Console (Offline unless needed) 1x Cluster Master (Offline Unless Needed) 3x Clustered Indexers (Clustered with region A for event replication) 4x Load Balanced Syslog-NG Collectors / Heavy Forwarder Combo Here are my questions: Our environment architect would prefer that there are no external dependencies in our production app tier. Currently the heavy forwarders in my app tier communicate to the primary cluster master to receive a list of available indexers to output their events to. Would it be wise of me to write a custom outputs.conf that manually defined the indexers, essentially eliminating the need for my HFs to contact the cluster master on startup? What would the cons be to this approach? It's worth noting that my HF's are only able to forward events to the indexers within their own region. In a similar vein to question one is it possible to have more than one license server / deployment server active at a time? We would like to eliminate as many of Splunk's dependencies on the opposite region as much as possible. Ideally region A could have it's own DS / LS / MC and never need to contact region B's DS / LS / MC or visa versa. Is there any clean way to handle index replication for data DR / HA purposes across indexers that are not in a multi-site cluster? My problems seem like they would be solved if I simply split our individual AWS regions into their own independent Splunk deployments. Would something like having each region back their events up to AWS Smart Store / S3 then pulling the data into the other regions indexers work?    Any advice you could provide me would be more than appreciated. I'm here to learn. Thank you all for your time. ... View more