The internal logs flow to splunk UI but the applications logs are not flowing to splunk UI. We have a cluster with several different components. We are facing the above issue with only one of the component, although, the splunk configuration for all the components are same except the host differs.

We're trying to integrate splunk APM to a python based app but the service doesn't appear in the APM list.  The integration works locally, but not in the same service deployed in our Kubernetes cluster We have added the following env variables into the deployment manifest of the application: - name: SPLUNK_OTEL_AGENT    valueFrom:         fieldRef:              fieldPath: status.hostIP - name: OTEL_EXPORTER_OTLP_ENDPOINT    value: "http://$(SPLUNK_OTEL_AGENT):4317" - name: OTEL_SERVICE_NAME    value: "my-app" - name: OTEL_RESOURCE_ATTRIBUTES    value: "deployment.environment=development" We also tried to add these additional variables to try to send the data directly, but it still didn't work - name: SPLUNK_ACCESS_TOKEN    value: "******" - name: OTEL_TRACES_EXPORTER    value: "jaeger-thrift-splunk" - name: OTEL_EXPORTER_JAEGER_ENDPOINT    value: "https://ingest.us1.signalfx.com/v2/trace"

Hi, I have created multi select dropdown and when I tried to be dependable by passing dropdown token, it doesn't shows any value. <form> <label>TEST- Multi Select with distinct value</label> <fieldset submitButton="false"> <input type="multiselect" token="flow"> <label>Select Flow</label> <choice value="*">All</choice> <default>*</default> <delimiter>,</delimiter> <fieldForLabel>FLOW</fieldForLabel> <fieldForValue>FLOW</fieldForValue> <search> <query>| loadjob savedsearch="Test_Data" | search adt="$adt$"</query> <earliest>0</earliest> <latest></latest> </search> <prefix>IN(</prefix> <suffix>)</suffix> <valuePrefix>"</valuePrefix> <valueSuffix>"</valueSuffix> </input> <input type="multiselect" token="adt"> <label>Select ADT</label> <choice value="*">All</choice> <default>*</default> <delimiter>,</delimiter> <fieldForLabel>adt</fieldForLabel> <fieldForValue>adt</fieldForValue> <search> <query>| loadjob savedsearch="Test_Data" | search flow="$flow$"</query> <earliest>0</earliest> <latest></latest> </search> <prefix>IN(</prefix> <suffix>)</suffix> <valuePrefix>"</valuePrefix> <valueSuffix>"</valueSuffix> <initialValue>*</initialValue> </input> </fieldset> <row> <panel> <table> <search> <query>| loadjob savedsearch="Test_Data" | where FLOW $flow$ and adt $adt$ | table adt, FLOW, Date, NbRecordsOKFCR, CMTotal, NbRecordsOKCM, NBIntFile, NB1, NB2, NB3, NbErrorsCM, Alert | fields Date, adt, FLOW, CMTotal</query> <earliest>0</earliest> <latest></latest> </search> <option name="count">10</option> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> </form>

Hello, I am using 2 multi select dropdown. When its on  the default value  'ALL' then it doesn't show any value in the table, after selection it works.    After open in search, it shows "*" in the value. | loadjob savedsearch="TEST" | where FLOW IN("*") and adt IN("*") | table adt, FLOW, Date | fields Date, adt, FLOW, CMTotal ------------------------------------------------------------------ Original Query-  <form>   <label>AAA_Test</label>   <fieldset submitButton="false">     <input type="multiselect" token="flow">       <label>Select Flow</label>       <choice value="*">All</choice>       <default>*</default>       <delimiter>,</delimiter>       <fieldForLabel>FLOW</fieldForLabel>       <fieldForValue>FLOW</fieldForValue>       <search>         <query>| loadjob savedsearch="TEST" | dedup FLOW</query>         <earliest>0</earliest>         <latest></latest>       </search>       <prefix>IN(</prefix>       <suffix>)</suffix>       <valuePrefix>"</valuePrefix>       <valueSuffix>"</valueSuffix>       <initialValue>*</initialValue>     </input>     <input type="multiselect" token="adt">       <label>Select ADT</label>       <choice value="*">All</choice>       <default>*</default>       <delimiter>,</delimiter>       <fieldForLabel>adt</fieldForLabel>       <fieldForValue>adt</fieldForValue>       <search>         <query>| loadjob savedsearch="TEST" | dedup adt</query>         <earliest>0</earliest>         <latest></latest>       </search>       <prefix>IN(</prefix>       <suffix>)</suffix>       <valuePrefix>"</valuePrefix>       <valueSuffix>"</valueSuffix>       <initialValue>*</initialValue>     </input>   </fieldset>   <row>     <panel>       <table>         <search>           <query>| loadjob savedsearch="TEST"           | where FLOW $flow$ and adt $adt$           | table adt, FLOW, Date, NbRecordsOKFCR, CMTotal, NbRecordsOKCM, NBIntFile, NB1, NB2, NB3, NbErrorsCM, Alert           | fields Date, adt, FLOW, CMTotal</query>           <earliest>0</earliest>           <latest></latest>         </search>         <option name="count">10</option>         <option name="drilldown">none</option>         <option name="refresh.display">progressbar</option>       </table>     </panel>   </row> </form> Please suggest.

I'm using Ansible to try to configure SPlunk Enterprise and more specifically- I want to create a user for the Splunk add-on TA-jira-service-desk-simple-addon However, I'm getting this error when trying when trying to run my Ansible  "Status code was -1 and not [200]: Connection failure: [Errno 104] Connection reset by peer"   Is this because I'm using a free-trial for Splunk? I deployed Splunk using the Splunk Enterprise API on AWS and I can connect to Splunk web no problem. Here's my Ansible playbook: --- - name: Create Jira Service Desk User in Splunk hosts: splunk_sh gather_facts: false tasks: - name: Create user uri: url: "http://ec2-44-212-47-250.compute-1.amazonaws.com:8089/servicesNS/nobody/TA-jira-service-desk-simple-addon/ta_service_desk_simple_addon_account" method: POST user: "admin username" password: "admin password" body: "name=svc_jira&jira_url=test.url.com&username=test_username" status_code: 200 I redacted my admin username and password, but I tried using the URL above (Which is the DNS name) and I tried using just the IPv4 address with port 8089 and the endpoint and it gave me the same error. I made sure that port 8089 is also open on my AWS Security Group. What could be causing the issues?

I have a search along these lines     "duration: " | rex field=host "(?P<host_type>[my_magic_regex])" | rex "duration: (?P<duration_seconds>[0-9]+)" | chart count by duration_seconds host_type limit=0 span=1.0     This is working exactly as expected. However, since I am doing count by ... for each host type, the histograms constructed for each host_type vary wildly.  The lines have such a different scale that overlaying them on the same axis is worthless. I need to either 1. create a different chart for each host_type (and not worry about the actual value of count) 2. normalize the y axis so that instead of the literal count, the max peak for all histograms is 1 (or 100 or whatever) I think I'll need a foreach command somewhere, but not sure what's the best route forward. Maybe there's a command similar to count that I should be using instead.

Is there a way to find out what the oldest events are, by index,  in the local cache when running SmartStore? I am able to ssh in and look at the local buckets, but is there a way to see it in the monitoring console or by query?   Thanks!   Joe

Is there a way to one-click reset all inputs back to their default values in dashboard studio? I have 7 different inputs (dropdowns and text) that are being used as filter criteria for a table. I would like a way to click "something" and have them all set back to their respective default values. I have done something similar for another dashboard that resets tokens that have been set based on clicked rows in charts/tables (just using a single value panel and setting all of the tokens), but I don't see a way to do this for inputs. Reloading the dashboard doesn't set them back to default either. It requires exiting the dashboard and relaunching. Thanks Craig

Hello from Splunk Data Manager Team, We are excited to announce the preview of the new Kubernetes Navigator for Splunk Observability Cloud. Before you search through previous conversations looking for assistance, we want to provide you with some basic information and quick resources. Want to access product docs? The Infrastructure Monitoring User Manual  offers detailed guidance on the interfaces provided by the new Kubernetes Navigator Want to request more features? Add your ideas and vote on other ideas in the Infrastructure Monitoring category via the Splunk Ideas Portal Please reply to this thread for any questions or get extra help!

Numeral system macros for Splunk Examples of Single Value panel and Table. Hello, Just an announcement. I have created macros that converts a number into a string with a language specific expressing (long and short scales, or neither). It was released on splunkbase. https://splunkbase.splunk.com/app/6595 Language-specific expressions may be useful when displaying huge numbers on a dashboard to make their size easier to understand. Or it may help us to mutually understand how numbers are expressed in other languages. Ref.: About long and short scales (numeration system). https://en.wikipedia.org/wiki/Long_and_short_scales Example of Use: Sample for English speakers | makeresults | eval val=1234567890123, val=`numeral_en(val)` | table val 1 trillion 234 billion 567 million 890 thousand 123 Provided macros: numeral_en(1) : Short Scale for English speaker numeral_metric_prefix(1) : Metric prefix. kilo, mega, giga, tera, peta, exa, zetta, yotta numeral_metric_symbol(1) : Metric symbol. K, M, G, T, P, E, Z, Y numeral_jp(1) : 万進法 for Japanese speaker. 千, 万, 億, 兆 numeral_kr(1) : for Korean speaker. 千, 萬, 億, 兆 numeral_cn_t(1) : Chinese with Traditional Chinese characters. 千, 萬, 億, 兆 numeral_cn(1) : Chinese with Simplified Chinese characters. 千, 万, 亿, 兆 numeral_in_en(1) : for India, South Asia English. thousand, lakh, crore, lakh crore numeral_in_en2(1) : for India, South Asia English. thousand, lakh, crore, arab numeral_nl(1) : Long Scale for Nederland. duizend, miljoen, miljard, biljoen numeral_fr(1) : Long Scale for French. mille, million, milliard, billion numeral_es(1) : Long Scale for Spanish speaker. mil, millón, millardo, billón numeral_pt(1) : Long Scale for Portuguese speaker. mil, milhão, bilhão, trilhão Followings also provided since v1.1.1 numeral_binary_symbol(1) : Binary symbol. KiB, MiB, GiB, TiB, PiB, EiB, ZiB, YiB, RiB, QiB numeral_binary_symbol(2) : Binary symbol with arg for rounding digits.     See Next article "How to convert large bytes to human readable units (e.g. Kib, MiB, GiB)" More details See Details tab on https://splunkbase.splunk.com/app/6595 Install this add-on into your search heads. Advanced examples Sample usage for using all provided macros. With rounding lowest 3 digit if over 6 digit. | makeresults count=35 | streamstats count as digit | eval val=pow(10,digit-1), val=val+random()%val.".".printf("%02d",random()%100) | foreach metric_prefix metric_symbol binary_symbol en es pt in_en in_en2 jp kr cn_t cn nl fr [eval <<FIELD>>=val] | table digit val metric_prefix metric_symbol binary_symbol en es pt in_en in_en2 jp kr cn_t cn nl fr | fieldformat val=tostring(val,"commas") | fieldformat metric_prefix=`numeral_metric_prefix(if(log(metric_prefix,10)>6,round(metric_prefix,-3),metric_prefix))` | fieldformat metric_symbol=`numeral_metric_symbol(if(log(metric_symbol,10)>6,round(metric_symbol,-3),metric_symbol))` | fieldformat binary_symbol=printf("% 10s",`numeral_binary_symbol(binary_symbol,2)`) | fieldformat en=`numeral_en(if(log(en,10)>6,round(en,-3),en))` | fieldformat es=`numeral_es(if(log(es,10)>6,round(es,-3),es))` | fieldformat pt=`numeral_pt(if(log(pt,10)>6,round(pt,-3),pt))` | fieldformat in_en=`numeral_in_en(if(log(in_en,10)>6,round(in_en,-3),in_en))` | fieldformat in_en2=`numeral_in_en2(if(log(in_en2,10)>6,round(in_en2,-3),in_en2))` | fieldformat jp=`numeral_jp(if(log(jp,10)>6,round(jp,-3),jp))` | fieldformat kr=`numeral_kr(if(log(kr,10)>6,round(kr,-3),kr))` | fieldformat cn_t=`numeral_cn_t(if(log(cn_t,10)>6,round(cn_t,-3),cn_t))` | fieldformat cn=`numeral_cn(if(log(cn,10)>6,round(cn,-3),cn))` | fieldformat nl=`numeral_nl(if(log(nl,10)>6,round(nl,-3),nl))` | fieldformat fr=`numeral_fr(if(log(fr,10)>6,round(fr,-3),fr))` The results of this search will look like the table in the top image of this article.

Hello, I have installed the SCOM app (version 430) on my Splunk Heavy Forwarder (903) The Windows SCOM infrastructure exists  of one managementgroup with 2 management servers. managementgroup = SC-PROD consisting of 2 management SC-PRD1 and SC-PRD2 The reason for this, is when server 1 is down, data is still collected thru the second node. I have configured  the managemt servers in the Splunk SCOM app. I have only to option to connect to the management servers and not to a group. I connect with  a URL , with a Service acount that exist on the server My problem is that i get double data, from both managemtn servers. I am looking for a smart way to connect, like a cluster Has anybody experience and advise in this scenario? Any advise is appriceated Regards, Harry

Installing the forwarder manually works fine, installing it automatically with the same user account fails with a 1603 error. Installer logs snippet:   MSI (s) (B8:FC) [09:22:23:304]: Note: 1: 2203 2: C:\Windows\Installer\inprogressinstallinfo.ipi 3: -2147287038 MSI (s) (B8:FC) [09:22:23:304]: Note: 1: 2205 2: 3: LaunchCondition MSI (s) (B8:FC) [09:22:23:304]: Note: 1: 2228 2: 3: LaunchCondition 4: SELECT `Condition` FROM `LaunchCondition` MSI (s) (B8:FC) [09:22:23:304]: APPCOMPAT: [DetectVersionLaunchCondition] Failed to initialize pRecErr. MSI (s) (B8:FC) [09:22:23:304]: PROPERTY CHANGE: Adding ACTION property. Its value is 'INSTALL'. MSI (s) (B8:FC) [09:22:23:304]: Doing action: INSTALL MSI (s) (B8:FC) [09:22:23:304]: Note: 1: 2205 2: 3: ActionText Action start 9:22:23: INSTALL. MSI (s) (B8:FC) [09:22:23:320]: Running ExecuteSequence MSI (s) (B8:FC) [09:22:23:320]: Doing action: SetAllUsers MSI (s) (B8:FC) [09:22:23:320]: Note: 1: 2205 2: 3: ActionText MSI (s) (B8:EC) [09:22:23:320]: Invoking remote custom action. DLL: C:\Windows\Installer\MSI5F93.tmp, Entrypoint: SetAllUsersCA MSI (s) (B8:F8) [09:22:23:320]: Generating random cookie. MSI (s) (B8:F8) [09:22:23:320]: Created Custom Action Server with PID 976 (0x3D0). MSI (s) (B8:3C) [09:22:23:335]: Running as a service. MSI (s) (B8:3C) [09:22:23:335]: Hello, I'm your 64bit Impersonated custom action server. Action start 9:22:23: SetAllUsers. SetAllUsers: Debug: Num of subkeys found: 1. SetAllUsers: Info: Previously installed Splunk product is not found. SetAllUsers: Error: Failed SetAllUsers: 0x2. SetAllUsers: Info: Leave SetAllUsers: 0x80004005. CustomAction SetAllUsers returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox) Action ended 9:22:23: SetAllUsers. Return value 3. Action ended 9:22:23: INSTALL. Return value 3.