All Topics

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Topics

Hi all, I am deploying the Hipster Store application as part of the Guided Onboarding in the Splunk Observability Cloud. However, many of the pods are stuck in `CrashLoopBackOff` status. Environmen... See more...
Hi all, I am deploying the Hipster Store application as part of the Guided Onboarding in the Splunk Observability Cloud. However, many of the pods are stuck in `CrashLoopBackOff` status. Environment Details - Platform: Minikube   - Minikube Version: v1.34.0 (running on macOS 14.5, arm64)   - Docker Version:    - Client: v27.4.0     - Server: v27.2.0 (Community Edition)   - Helm Version: version.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"clean", GoVersion:"go1.22.7"} Command Used for Deployment helm install my-sixth-splunk-o11y-otel-collector --set="splunkObservability.realm=us1,splunkObservability.accessToken=XXX_XXXXXXXXXXXXXXXXXX,clusterName=my-sixth-otel-collector-cluster" splunk-otel-collector-chart/splunk-otel-collector Error Observed  Pods remain in the `CrashLoopBackOff` state. Below is the output of `kubectl get pods`: NAME                                                              READY   STATUS             RESTARTS        AGE adservice-cbbc87864-fbt7d                                         0/1     Running            3 (45s ago)     7m51s cartservice-797fcdd44b-67ctq                                      0/1     CrashLoopBackOff   6 (97s ago)     7m52s checkoutservice-7c5955d5b9-gksms                                  0/1     CrashLoopBackOff   6 (2m11s ago)   7m52s ... (other entries truncated for brevity) Logs from Affected Pods 1. `checkoutservice` Logs:    {"message":"failed to start profiler: project ID must be specified in the configuration if running outside of GCP","severity":"warning","timestamp":"2025-01-06T22:58:09.648950291Z"}    {"message":"sleeping 10s to retry initializing Stackdriver profiler","severity":"info"} 2. `productcatalogservice` Logs:    {"message":"failed to start profiler: project ID must be specified in the configuration if running outside of GCP","severity":"warning","timestamp":"2025-01-06T22:59:15.143462044Z"}    {"message":"sleeping 20s to retry initializing Stackdriver profiler","severity":"info"} 3. `shippingservice` Logs:    {"message":"failed to start profiler: project ID must be specified in the configuration if running outside of GCP","severity":"warning","timestamp":"2025-01-06T22:58:59.462097926Z"}    {"message":"sleeping 10s to retry initializing Stackdriver profiler","severity":"info"} Troubleshooting Attempts 1. Tried adding dummy Google Project IDs (I researched how they should formatted) in various configuration files.   2. Disabled Tracing, Profiling, and Debugging at different times.   3. Confirmed Helm chart installation was successful.   Questions 1. How can I resolve the `CrashLoopBackOff` issue?   2. Is there a way to bypass or properly configure the Stackdriver profiler to avoid requiring a Google Project ID?   3. Are there any additional configurations required to run this demo outside of GCP? Thank you.
I am getting result like this.   query: index="webmethods_prd" host="USPGH-WMA2AISP*" source="/apps/WebMethods/IntegrationServer/instances/default/logs/SmartIST.log" |stats count by SmartI... See more...
I am getting result like this.   query: index="webmethods_prd" host="USPGH-WMA2AISP*" source="/apps/WebMethods/IntegrationServer/instances/default/logs/SmartIST.log" |stats count by SmartISTINTERFACE instead of above I want a report like this:    
Hello all, I'm trying to reset a host under the DHM tab in TrackMe so I can remove a sourcetype it's not finding that no longer exists. However, when I do this I receive the error below. I have alrea... See more...
Hello all, I'm trying to reset a host under the DHM tab in TrackMe so I can remove a sourcetype it's not finding that no longer exists. However, when I do this I receive the error below. I have already checked my permissions and as an admin have the correct role to admin the app as well. 
All, In the metric browser, I see a DB appear under Backends|Discovered backend call...  I also see the same backend under Overall Application Performance|My Tier|External Calls|Call-JDBC to Discove... See more...
All, In the metric browser, I see a DB appear under Backends|Discovered backend call...  I also see the same backend under Overall Application Performance|My Tier|External Calls|Call-JDBC to Discovered backend... The calls per minute graphs are approximately the same shape, but counts are not even close.  Under backends, the counts are much higher, like 5x higher.  This app only has 1 active tier. Why is there such a large difference in counts?  I would like to get a breakdown of DB calls per tier, but the numbers seem low. thanks
I am looking to have the middle row of this table be in the left instead. I think something in the query is off and causing it to have a weird behavior.  index=main host=* sourcetype=syslog ... See more...
I am looking to have the middle row of this table be in the left instead. I think something in the query is off and causing it to have a weird behavior.  index=main host=* sourcetype=syslog process=elcsend "\"config" $city$ | rex "([^!]*!){16}(?P<MEMGB>[^!]*)!" | chart count by MEMGB | addcoltotals label=Total labelfield=MEMGB | sort count desc This is the current search query. The rex provides the data in the MEMGB column.
How do I return field values from a specific max(eventnumber)? This was helpful but did not solve my issue Solved: How to get stats max count of a field by another f... - Splunk Community We are ... See more...
How do I return field values from a specific max(eventnumber)? This was helpful but did not solve my issue Solved: How to get stats max count of a field by another f... - Splunk Community We are ingesting logs from test devices. Each log has an event number, which I can search on to find the most recent event. When the devices disconnect from our cloud instance, they cache events which are transmitted at a lower priority (newest to oldest) than real time events. For example: event #100 connected to cloud, event 101-103 disconnected from cloud and cached, events, #104 re-connected to cloud (latest status) received, then event 103 is transmitted, then 102, so using latest/earliest or first/last does not return the most recent status The logs consist of an event number and boolean (true/false) fields. Searching for max(event number) and values(boolean field value) results in both true/false for any time picker period that has multiple events, for example: | stats max(triggeredEventNumber) values(isCheckIn) values(isAntiSurveillanceViolation) BY userName userName                 max(triggeredEventNumber)      values(isCheckIn)      latest(isAntiSurveillanceViolation) NS2_GS22_MW    92841                                                   false true                       FALSE In the example the actual value of isCheckIn was true. Here is a complete example event: { "version": 1, "logType": "deviceStateEvent", "deviceSerialNumber": "4234220083", "userName": "NS2_GS22_MW", "cloudTimestampUTC": "2025-01-06T18:17:00Z", "deviceTimestampUTC": "2025-01-06T18:16:46Z", "triggeredEventNumber": 92841, "batteryPercent": 87, "isCheckIn": true, "isAntiSurveillanceViolation": false, "isLowBatteryViolation": false, "isCellularViolation": false, "isDseDelayed": false, "isPhonePresent": true, "isCameraExposed": false, "isShutterOpen": false, "isMicExposed": false, "isCharging": false, "isPowerOff": false, "isHibernation": false, "isPhoneInfoStale": false, "bleMacAddress": "5c:2e:c6:bc:e4:cf", "cellIpv4Address": "0.0.0.0", "cellIpv6Address": "::" }
Hello all !  I need to get data from Splunk Observability (list of synthetics tests) into Splunk cloud. I have tried to use this observability API :  curl -X GET "https://api.{REALM}.signalfx.com/... See more...
Hello all !  I need to get data from Splunk Observability (list of synthetics tests) into Splunk cloud. I have tried to use this observability API :  curl -X GET "https://api.{REALM}.signalfx.com/v2/synthetics/tests" \ -H "Content-Type: application/json" \ -H "X-SF-TOKEN: <value>"  Then, I attempted to execute a cURL query in Splunk Cloud like this : | curl method=get uri=https://api.xxx.signalfx.com/v2/synthetics/tests?Content-Type=application/json&X-SF-TOKEN=xxxxxxxxxxx | table curl* but i am getting the following error : HTTP ERROR 401 Unauthorized. Thanks for any help !    
Hi, Could you pls let me know in what scenario would we use eventstats vs stats?
We are planning to upgrade our Splunk hardware. We currently have below(multisite indexer cluster with independant search head clusters) and we are facing problems with low cpu count and high disk la... See more...
We are planning to upgrade our Splunk hardware. We currently have below(multisite indexer cluster with independant search head clusters) and we are facing problems with low cpu count and high disk latency(we currently have HDDs). We primarily index data through HEC.   Type Site Number of nodes CPU p/v (per node) memory GB (per node) SH cluster 1 4 16/32 128 Indexer cluster 1 11 4/8 64 Indexer manager/License master 1 1 16/32 128 SH cluster 2 4 16/32 128 Indexer cluster 2 11 4/8 64 Indexer manager/License master 2 1 16/32 128   Daily indexing/license usage 400-450GB which may grow further in near future Search concurrency example for one instance from 4 node SH cluster   We are trying to come up with the best hardware configuration that can support such load.   Looking at Splunk recommended settings, we have comeup with below config. Can someone shed more light on if this is an optimal config and also advise on the number of SH machines and indexer machines needed with such new hardware Site1: 3 node SH clusters, 7 node idx cluster Site2:  As we are using site2 for searching and indexing only during unavailability of site1, may be it can be smaller? Role CPU (p/v) Memory Indexer 24/48 64G Non indexer 32/64 64G
I am working om creating a dashboard to display data from my app-I have a dropdown where you select which environment you want to see data for-I need to set 2 values based on this dropdown: 1.connec... See more...
I am working om creating a dashboard to display data from my app-I have a dropdown where you select which environment you want to see data for-I need to set 2 values based on this dropdown: 1.connection for db queries 2. host for logs based queries I searched many option but couldn't get any to work:I am trying to do  <fieldset submitButton="false"> <input type="dropdown" token="connection"> <label>Select Region</label> <default>dev-platform-postgres</default> <choice value="dev-platform-postgres">US</choice> <choice value="dev-platform-postgres-eu">EU</choice> <change> <condition label = 'dev-platform-postgres'> <set token="host">eks-prod-saas-ue1-*</set> </condition> <condition label = 'dev-platform-postgres-eu'> <set token="host">prd-shared-services-eu-eks*</set> </condition> </change> </input> </fieldset> and then be able to use both $host$ and $connection$ tokens in the dashboard but I cant get $host$ initialized correctly any help would be appreciated also -side note I am getting a warning "Expected at most 1 children of fieldset in dashboard, instead saw 2"-how am I supposed to handle a case where I want 2 selections -one of date and one of connection?
Can i do the wildcard matching in lookup? |makeresults |eval ip=192.168.101.10 |lookup ip.csv ip output host In my lookup i have two entry ip=192.168.101.10 & ip=192.168.101.10/24. How can i add... See more...
Can i do the wildcard matching in lookup? |makeresults |eval ip=192.168.101.10 |lookup ip.csv ip output host In my lookup i have two entry ip=192.168.101.10 & ip=192.168.101.10/24. How can i add wildcard (*) for match and i should get two entry.  
Hello, I'm setting up StatsD to send custom metrics from an AWS EC2 instance, where the Splunk OpenTelemetry Collector is running to Splunk Observability Cloud. I've configured StatsD as a receiver... See more...
Hello, I'm setting up StatsD to send custom metrics from an AWS EC2 instance, where the Splunk OpenTelemetry Collector is running to Splunk Observability Cloud. I've configured StatsD as a receiver using guidelines from the https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver. Here's my configuration for StatsD configured in the agent_config.yaml file. receivers: statsd: endpoint: "localhost:8125" aggregation_interval: 60s enable_metric_type: false is_monotonic_counter: false timer_histogram_mapping: - statsd_type: "histogram" observer_type: "histogram" histogram: max_size: 50 - statsd_type: "distribution" observer_type: "histogram" histogram: max_size: 50 - statsd_type: "timing" observer_type: "summary" The GitHub documentation provides exporter configurations, but I'm unsure how to implement them effectively. As per the github document below is mentioned. exporters: file: path: ./test.json service: pipelines: metrics: receivers: [statsd] exporters: [file] Below is the receivers configuration which I am setting in the service configuration section in the in the agent_config.yaml as mentioned below: service: pipelines: metrics: receivers: [hostmetrics, otlp, signalfx, statsd] processors: [memory_limiter, batch, resourcedetection] exporters: [signalfx] When I add "statsd" ("receivers: [hostmetrics, otlp, signalfx, statsd]" and "exporters: [signalfx]") as one of the more receivers as mentioned above and restart the "systemctl restart splunk-otel-collector.service", splunk otel collector agent stop sending any metric to the Splunk Observability Cloud and when I remove statsd (receivers: [hostmetrics, otlp, signalfx]) then splunk otel collector agent starts sending any metric to the Splunk Observability Cloud. What should be correct/supported ad receiver/exporter to be configured in the service section for the statsd? Thanks
HI all I have a scenario where i have to find the difference of two field value (string) for example fileda="raj", "rahul", "rohan" filedb="rahul", "rohan" i need to have a third field as differe... See more...
HI all I have a scenario where i have to find the difference of two field value (string) for example fileda="raj", "rahul", "rohan" filedb="rahul", "rohan" i need to have a third field as difference of the above two filed fieldc="raj" I am running out of ideas as how to do it. Can someone please help in this  
My requirement is to pass the tokens via drill down from parent dashboard to drill down dashboard (which is created with Java scripts) From parent dashboard, I tried to pass the token via drill down... See more...
My requirement is to pass the tokens via drill down from parent dashboard to drill down dashboard (which is created with Java scripts) From parent dashboard, I tried to pass the token via drill down URL to the java scripted dashboard, but that did not work out. Can anyone please help me in passing the tokens via drill down to the target dashboard which is created with Java script?
Hi everyone, want to ask why i cannot download data from dashboard. if i make it 4 hours it can, and if i want download in another date ex (1 Jan 2025) its fine, it is only happen when i want downloa... See more...
Hi everyone, want to ask why i cannot download data from dashboard. if i make it 4 hours it can, and if i want download in another date ex (1 Jan 2025) its fine, it is only happen when i want download data in (4 Jan 2025).  Since i cannot access any website / media to screenshot from client so i only can photo.  Thank you
Back again with another question. I'm still playing with my search and whle this is an issue I've managed to work around, the fact that I need to work around it without knowing the why behind it eats... See more...
Back again with another question. I'm still playing with my search and whle this is an issue I've managed to work around, the fact that I need to work around it without knowing the why behind it eats at me. I have a search that pulls data from two different sourcetypes, and each of those sourcetypes have a src_mac field(the data in these fields is identical except for the letter case). To rectify the issues this causes when attempting to call the field in a search, I use eval to create two new fields with the sourcetype of each event so that the field names are now unique(in addition to fixing the letter case mismatch). Specifically, this creates two fields named "src_mac-known_devices" and "src_mac-ise:syslog" | eval src_mac-{sourcetype}=src_mac, src_mac=upper(src_mac) | where upper("src_mac-*") = upper("src_mac-*") However, in the WHERE command, I'm only able to call these two new fields when I use a wildcard. I can't actually put in | WHERE upper("src_mac-bro_known_devices") = upper("src_mac-ise:syslog") The command just doesn't work for some reason, and I get zero hits despite *knowing* I should get plenty of hits. In other words, it works fine when I use the wildcard and not at all when I use anything else. Even attempting to do something like | where upper("src_mac-b*") = upper("src_mac-c*") doesn't work.  I have read through the wiki articles on proper naming practices for fields, so I know my two fields contain illegal characters. I also know the : is used when trying to search indexed fields, but I thought I could use single or double quotation marks to work around that limitation or maybe using the / to escape the special characters.....but none of that has worked. At this point, I just want to understand *why* it isn't working. Thank you for any help anyone can provide.
Hello everyone, I am facing an issue with the alerts triggered by the "Set Default PowerShell Execution Policy To Unrestricted or Bypass" (Correlation Search) rule in Splunk, as many alerts are bein... See more...
Hello everyone, I am facing an issue with the alerts triggered by the "Set Default PowerShell Execution Policy To Unrestricted or Bypass" (Correlation Search) rule in Splunk, as many alerts are being generated unexpectedly. After reviewing the details, I added the command `| stats count BY process_name` to analyze the data more precisely. After executing this, the result was 389 processes within 24 hours. However, it seems there might be false positives and I’m unable to determine if this alert is normal or if there’s a misconfiguration. I would appreciate any help in identifying whether these alerts are expected or if there is an issue with the configuration or the rule itself. Any assistance or advice would be greatly appreciated. Thank you in advance.  
From where we can see the actual score of any Splunk exam. Because from Splunk website we can only get certification and from Pearson Vue we can only see report which says congratulations you're pass... See more...
From where we can see the actual score of any Splunk exam. Because from Splunk website we can only get certification and from Pearson Vue we can only see report which says congratulations you're passed and doesn't mention any actual score.
We have a 5 node Splunk forwarder cluster to handle throughput of multiple servers in our datacenter.  Currently our upgrade method is keeping the the Deployment server as mutable where we just run c... See more...
We have a 5 node Splunk forwarder cluster to handle throughput of multiple servers in our datacenter.  Currently our upgrade method is keeping the the Deployment server as mutable where we just run config. changes via Chef, and update it.  But, the 5 node forwarders are being treated as fully replaceable with Terraform and Chef. Everything is working, but I notice the Deployment server holds onto forwarders after Terraform destroys the old one, and the new one pings home on a new IP(currently on DHCP), but with the same hostname as the destroyed forwarder.  Would replacing the forwarders with the same static IP and Hostname resolve that, or would there still be duplicate entries? Deployment server: Oracle Linux 8.10 Splunk-enterprise 8.2.9 Forwarders: Oracle Linux 8.10 Splunkforwarder 8.2.9
Bom dia! No cenário apresentado abaixo, não consigo associar os itens em uma tabela dentro do campo DbrfMatrial: EngineeringCode, ItemDescription, ItemQty, SolutionCode   Usei o... See more...
Bom dia! No cenário apresentado abaixo, não consigo associar os itens em uma tabela dentro do campo DbrfMatrial: EngineeringCode, ItemDescription, ItemQty, SolutionCode   Usei o índice abaixo! index=analise Task.TaskStatus="Concluído" Task.DbrfMaterial{}. SolutionCode="410 TROCA DO MOD/PLACA/PECA" State IN ("*") CustomerName IN ("*") ItemCode("*") | mvexpand Task.DbrfMaterial{}. Código de Engenharia| pesquise Task.DbrfMaterial{}. CódigoDeEngenharia="*" | contagem de estatísticas por Task.DbrfMaterial{}. Código de Engenharia| renomear contagem como Quantidade | cabeça 20 | tabela Task.DbrfMaterial{}. Quantidade do código de engenharia| ordenar -Quantidade | appendcols [ search index=brazilcalldata Task.TaskStatus="Concluído" Task.DbrfMaterial.SolutionCode="410 TROCA DO MOD/PLACA/PECA" CustomerName IN ("*") State IN ("*") Task.DbrfMaterial.EngineeringCode="*" ItemCode = "*" | stats count, sum(Task.DbrfMaterial.ItemQty) as TotalItemQty by Task.DbrfMaterial.EngineeringCode Task.DbrfMaterial.ItemDescription | renomeie Task.DbrfMaterial.EngineeringCode como Item, Task.DbrfMaterial.ItemDescription como Descricao, TotalItemQty como "Qtde Itens" | table Item Descrição "Qtde Itens" count | sort - "Qtde Itens" ] | eval TotalQuantity = Quantity + 'Qtde Itens' | pesquise Task.DbrfMaterial{}. Código de Engenharia!="" | tabela Task.DbrfMaterial{}. EngineeringCode Quantidade "Qtde Itens" TotalQuantity