Hello Splunkers, I faced the following issue : I deployed an app on a UF, this app should monitor a specific file in my machine let's say /<my_file> The thing is I'm running Splunk service as a non root user (splunk user) and this user does not have permission to read this file. I know how to solve this with setfacl command, but how could I spot this issue in the first place ? I thought that this permission error would have been visible in splunkd.log but it's not the case... I am trying to find a way to monitor the other possible "permissions denied" errors without manually log in as the splunk user and try to open the specific files. Thanks a lot, GaetanVP

i need to extract fields which are in json format i have been trying using spath command for extracting the following fields which are under log. But not able to fetch it. I am failing somewhere. Here is the example of my data: {"log":"[18:15:21.888] [INFO ] [] [c.c.n.t.e.i.T.ServiceCalloutEventData] [akka://MmsAuCluster/user/$b/workMonitorActor/$M+c] - channel=\"AutoNotification\", productVersion=\"2.3.3-0-1-eb5b8cadd\", apiVersion=\"V1\", uuid=\"0b8549ff-1f14-4fd5-99c5-b3f2240d7da8\", eventDateTime=\"2023-01-06T07:15:21.888Z\", severity=\"INFO\", code=\"ServiceCalloutEventData\", component=\"web.client\", category=\"integrational-external\", serviceName=\"Consume Notification\", eventName=\"MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT_REQUEST\", message=\"Schedule Job start, getNotification request\", entityType=\"MNDT\", externalSystem=\"SWIFTPAG\", start=\"1672989321888\", url=\"https://sandbox.swift.com/npp-mms/v1/subscriptions/29fbe070057811eca4fa68aa418f5c2a/notifications\", swiftMessagePartnerBIC=\"RESTMP01\", messageIdentification=\"e1f24a3b8d9111edb3368d1476d87136\", subscriptionIdentification=\"29fbe070057811eca4fa68aa418f5c2a\" producer=com.clear2pay.na.mms.au.notification.batch.GetNotificationService \n","stream":"stdout","docker":{"container_id":"89efc58c0a343ee01daa2fcdeadb3b952599f0c142fb7041f95a9d6702fe49d2"},"kubernetes":{"container_name":"mms-au","namespace_name":"msaas-t4","pod_name":"mms-au-b-1-54b4589f89-g74lp","container_image":"pso.docker.internal.cba/mms-au:2.3.3-0-1-eb5b8cadd","container_image_id":"docker-pullable://pso.docker.internal.cba/mms-au@sha256:9d48d5af268d28708120ee3f69b576d371b5e603a0e0c925c7dba66058654819","pod_id":"b474ec16-fc9f-4b7a-9319-8302c0185f83","pod_ip":"100.64.87.219","host":"ip-10-3-197-177.ap-southeast-2.compute.internal","labels":{"app":"mms-au","dc":"b-1","pod-template-hash":"54b4589f89","release":"mms-au"},"master_url":"https://172.20.0.1:443/api","namespace_id":"48ee871a-7e60-45c4-b0f4-ee320a9512f5","namespace_labels":{"argocd.argoproj.io/instance":"appspaces","ci":"CM0953076","kubernetes.io/metadata.name":"msaas-t4","name":"msaas-t4","platform":"PSU","service_owner":"somersd","spg":"CBA_PAYMENTS_TEST_COORDINATION"}},"hostname":"ip-10-3-197-177.ap-southeast-2.compute.internal","host_ip":"10.3.197.177","cluster":"nonprod/pmn02"}   i need to extract few events which are under log.Can anyone help me on this.   Thanks in Advance

Hello, I'm using stats list() to merge all my value into one field, but I want them to seperate with each other by ";" instead of space. Example USER_PHONE 123 456 789   When I use |stats list(USER_PHONE) the result I receive( in the csv that I output) was 123 456 789 The result that I want is 123;456;789 I try to use ... | rex mode=sed field=USER_PHONE "s/ /;/g" But it have no effect, what should I do?

so I configured a 4 member search head cluster successfully with a captain and a deployer server however when I got to deploy a simple app such as the GlobalBanner it aint working per this doc Use the deployer to distribute apps and configuration updates  I create the following subdirectory and conf file on my Deployer server C:\Program Files\Splunk\etc\shcluster\apps\GlobalBanner\local\global-banner.conf   and I push the bundle using this command from the Deployer server splunk apply shcluster-bundle -target https://MySearchHeadCaptain.MyDomain.net:8089   and the file ends up in the following directory on the search head cluster members C:\Program Files\Splunk\etc\apps\GlobalBanner\default\gloval-banner.conf Why does it end up in the default sub folder?   My Deployer server is setup with the push mode set to full like so  [shclustering] pass4SymmKey = $7$SoMeJuNk= shcluster_label = shcluster1 deployer_push_mode = full   1,000 thanks to anyone who can set me straight

Failed to contact license manager: reason='Unable to connect to license manager=https://SplunkInstance01.MyDomain.net:8089 Error connecting:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name.', first failure time= [...] first mea culpa for asking this as I am sure it has been asked before but I couldn't quite understand how to use the openssl verify command, when I try to run it I get this error: C:\Program Files\Splunk\etc\auth\distServerKeys>openssl verify -CAfile private.pem trusted.pem WARNING: can't open config file: C:\\jnkns\\workspace\\build-home/ssl/openssl.cnf Error loading file private.pem I also tried to run it from the bin subdirectory, home of the openssl utility  C:\Program Files\Splunk\bin>openssl verify -CAfile "C:\Program Files\Splunk\etc\auth\distServerKeys\private.pem" "C:\Program Files\Splunk\etc\auth\distServerKeys\trusted.pem" WARNING: can't open config file: C:\\jnkns\\workspace\\build-home/ssl/openssl.cnf Error loading file C:\Program Files\Splunk\etc\auth\distServerKeys\private.pem I suspect this private \ public key pair combination may still be the stale default self signed combination cause Splunk to frown upon it, however what is perplexing to me is that it works on the other 20 plus servers, so I throw yourself upon your mercy for help please note we have over 2 dozen Splunk servers running version 9.0.0 all on Windows platforms and this is the only server getting this error, all servers use our own Microsoft CA internal Enterprise certificates based on a two tier (RootCA \ IntermediateCA architecture) so I think I know what I am doing ha ha at least in terms of certing