Hi, I deployed Splunk distributed topology. Now my server Search Head has issue: KVStore is on failed state (it make app "Enterperise Security" failed too). I checked "/opt/splunk/var/log/splunk/splunkd.log" and found the below logs: ========================================== 10-13-2021 18:14:03.127 +0700 ERROR DataModelObject - Failed to parse baseSearch. err=Error in 'inputlookup' command: External command based lookup 'correlationsearches_lookup' is not available because KV Store initialization has failed. Contact your system administrator., object=Correlation_Search_Lookups, baseSearch=| inputlookup append=T correlationsearches_lookup | eval source=_key | eval lookup="correlationsearches_lookup" | append [| `notable_owners`] | fillnull value="notable_owners_lookup" lookup | append [| `reviewstatuses`] | fillnull value="reviewstatuses_lookup" lookup | append [| `security_domains`] | fillnull value="security_domain_lookup" lookup | append [| `urgency`] | fillnull value="urgency_lookup" lookup 10-13-2021 18:14:30.350 +0700 ERROR KVStorageProvider - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling ismaster on '127.0.0.1:8191'] 10-13-2021 18:14:30.350 +0700 ERROR KVStoreAdminHandler - An error occurred. =============================== Could anyone help me to troubleshoot this issue to solve it? Thanks so much!  

Dear Splunk Community, I have the following statistics table and corresponding column chart that show the amount of errors per server: And Right now I am changing the colors on the column chart based on a static value like so: index="myIndex" host="myHostOne*" OR host="myHostTwo*" source="mySource" ERROR NOT WARN CTJT* | table host, errors | eval errors = host | stats count by host | eval redCount = if(count>50,count,0) | eval yellowCount = if(count<=50 AND count>25,count,0) | eval greenCount = if(count <=25,count,0) | fields - count | dedup host | rex field=host mode=sed "s/\..*$//" | sort host asc | rename host AS "Servers" | rename count AS "Foutmeldingen" The above uses a time range of "last 24 hours". I would like to change the colors of the bars (green, yellow, red) when a certain percentage of errors has been reached, based on the average of last week. To summarize, I would like to: Somehow get the average amount of errors per server per day from the last 7 days Then specify a percentage for each color (e.g if the amount of errors today are 25% more than the average of last week, make the bar red) I have no idea on how to do this, can anyone help? Thanks in advance.    

Greetings!! I need your help on where I can find the retention configuration for Splunk syslog receiver through command line and how can change to only receive the logs for 1day not 2days, here I mean to do retention so that I can change where it is receiving and store logs for 2 days, NOW I WANT to do retention to only receive for 1 day only, KINDLY HELP ME AND GUIDE ME HOW I CAN DO THIS?   Splunk receiver it is in opt directory where i receive the syslog logs for different network  devices , storing those logs for days then after logs are deleted, BUT I want only to receive all logs coming from different devices and when day finished at midnight can delete that logs in splunk receiver after being indexed into indexers. Kindly help me on this as I want to avoid that the receiver storage run out of space. Thank you in advance!

Hi team, I have below kind of data in splunk, it contains 3 fields ISRF, DSRF and DSFF.  they are all multi-value fields.   2021-10-13 19:26:46,813 ISRF="[fullName,managerFullName,title,userName,division,department,location]" DSRF="[fullName,managerFullName,title,userName,division,department,location]" DSFF="[managerFullName,division,department,location,jobCodereasonForLeaving]" 2021-10-12 19:32:31,504 ISRF="[fullName,managerFullName,userName,division,department,location]" C_DSRF="[fullName,managerFullName,title,userName,division,department,location]" DSFF="[managerFullName,division,department,location,custom05,jobCode,riskOfLoss,impactOfLoss,reasonForLeaving]" ...... ......     I expect the report like below format: fields count Of ISRF count Of DSRF count of DSFF fullName 2 2 0 managerFullName 2 2 2 title 1 2 0 ......       ......       resonForLeaving 0 0 1   I am trying below queries, and I am blocked how to continue for getting expected format table.   <baseQuery> |eval includeSearchResultField=replace(replace(C_ISRF,"\[",""),"\]",""), defaultSearchResultField=replace(replace(C_DSRF,"\[",""),"\]",""), filterFields=replace(replace(C_DSFF,"\[",""),"\]","") |makemv delim="," includeSearchResultField |makemv delim="," defaultSearchResultField |makemv delim="," filterFields    

Hello All,  Can someone help me to build a search query for the below use case ?   My use case is to detect if any S3 buckets have been set for Public access via PutBucketPolicy event.  I need this query to show results only if  the fields  Effect and Principal both have values  "Allow"  and " *  or {AWS:*} "  respectively for the same SID.   Basically the following 2 conditions must be met for a particular SID. Effect: Allow Principal:  *  OR {AWS:*} ----------------------- The Raw event data however has 2 SIDs  ( MustBeEncryptedInTransit and Cloudfront Access)  as shown below and each one has conflicting values of Effect & Principal.     eventName": "PutBucketPolicy" "awsRegion": "us-east-1" "sourceIPAddress": "x.x.x.x" "userAgent": "[<some agent>]" "requestParameters": {"bucketPolicy": {"Version": "2012-10-17" "Statement": [{"Sid": "MustBeEncryptedInTransit" "Effect": "Deny" "Action": "s3:*" "Resource": ["arn:aws:s3:::<Bucket_Name>/*" "arn:aws:s3:::<Bucket_Name>"] "Principal": "*" "Condition": {"Bool": {"aws:SecureTransport": ["false"]}}} {"Sid": "Cloudfront Access" "Effect": "Allow" "Action": "s3:*" "Resource": "arn:aws:s3::<Bucket_Name>/*" "Principal": {"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity XXXXXX"}}]} "bucketName": "<Bucket_Name>" "Host": "<SomeHost_Name>" "policy": ""}     Now, if i try the below search, it generates False Positives because the raw data has everything in the same event:   Effect = Allow , Effect = Deny, Principal = *   and 2 values of SID   sourcetype=aws:cloudtrail eventName IN(PutBucketPolicy) userName="abcd" requestParameters.bucketPolicy.Statement{}.Effect = "Allow" requestParameters.bucketPolicy.Statement{}.Principal = "*" requestParameters.bucketPolicy.Statement{}.Sid = "Cloudfront Access"   I am just lost as in how to build an eval statement to check if SID = CloudFront Access or SID!=MustBeEncryptedInTransit  only then check for values of Effect and Principal. Hope i am clear.  If you all have better suggestions to check for pubic access using Putbucketpolicy or ACL let me know

Hello Splunk Community, Can anyone help me build a query based on the below; I have a batch job that has multiple steps logged as separate events. How can I calculate the total duration of the batch job (Step 1 Start - Step 5 End). Example of my output format (Dummy Data Used): Step Start_Time End_Time Duration (Hours) 1 2021-09-11 22:45:00 2021-09-11 22:45:01 00:00:01 2 2021-09-11 22:45:01 2021-09-11 22:45:20 00:00:19 3 2021-09-11 22:45:20 2021-09-11 22:58:15 00:12:55 4 2021-09-11 22:58:15 2021-09-11 22:58:39 00:00:24 5 2021-09-11 22:58:39 2021-09-11 24:20:31 01:21:52   THANK YOU!

Hello Splunk Community,  Can anyone help me build a query based on the below; I want to convert a field (Fri Oct 8 23:15:05 AEDT 2021) into time format & then calculate the duration by subtracting the end time by the start time.  Appreciate your help

Hi All, Can someone help me with the following  ColY represents multi-value field. I want to search all rows which have null, 0 and someother values in ColY Based on the below example output rows should be for A123456, A123461  ColX ColY A123456 null 0 56789 987654 A123457 4332 A123458 54322 0 A123459 null 0 A123460 2345667 7665443 A123461 null 788765 0 A123462 876543 null    

I want to add a dropdown selection menu inside a dashboard panel using html and then based on the selected item from the dropdown , need to change the search query accordingly and results are updated in the chart visualization. Example:  Select Month:   All, Jan, Feb, ...Dec. Now when user selects all, the query will display results for entire year (jan -dec) and when jan is selected then chart is displayed only showing the entire january data. Similarly for others. I have other different panels within the dashboard under differet rows, and  I need to insert the drop down menu only for a specific panel. Also, my base search remains same irrespective of the drop-down selection, only the panel search string changes with change in the dropdown selection. - any suggestion or guide would be a great help. It would be nice if it can be done with no or minimal use of JavaScript. TIA thanks

Hi All,            As part of one of my SRE objectives I was trying to find out the following in splunk.  The High(Max) count of ERRORs within a given time period (1hr / 24hr/ 144hr) compared to monthly 99 Percentile I was starting off with baby steps assuming that the count is obviously zooming in on anything 'ERROR' index=myIndex ERROR source="/test.log" | timechart count by status | addtotals | addtotals fieldname=ERROR | eval ErrorRate=round(Errors/Total*100,2) | fields _time 5* ErrorRate But that doesn't seem to even work.  Help would be really appropriated .   Thanks in advance team !