About michel_wolf

michel_wolf · ‎08-07-2023

Thanks for the clearification, so it´s looks like post-processing searches and chained searches, doesn´t provide a SID so it´s not possible to access the refine results here. This is okay for me then I will work on antoher way.

michel_wolf · ‎08-07-2023

I'm currently working on an XML dashboard in Splunk where I've set up a chained search that builds upon a base search. My objective is to retrieve the SID (Search ID) for the chained search itself, rather than just obtaining the SID of the base search, which currently happens when I use the addinfo command. When I apply the addinfo command within the chained search, it only provides me with the SID of the base search, and I'm looking to access the SIDs associated with the extended search queries within the chained search. How can I effectively retrieve the SIDs for each component of the chained search, including the extended queries, using the addinfo command or any alternative methods? Sample <form theme="dark" version="1.1"> <label>test</label> <search id="baseSearch"> <query> index="test" | table A B C D E F _time </query> <earliest>-7d@d</earliest> <latest>now</latest> </search> <table> <search base="baseSearch"> <done> <set token="job_exportTocsv">$job.sid$</set> </done> <query>| stats count by A | addinfo </query> </search> <option name="drilldown">none</option> <option name="refresh.display">progressbar</option> </table> </panel> </row> </form> The job.sid you can see or which is added from addinfo shows only the results from the basesearch in this example, if you make a |loadjob $job.sid$ which is provided by the chained search you will see the results from the basesearch | table A B C D E F _time instead of the |stats count by A. So it looks like the chained searches handels different instead of a basesearch, it was also not possible for me to find the chained search in Activity --> Jobs or access this search via REST Endpoint. Any ideas here two access the results from the chained search?

michel_wolf · ‎08-06-2021

You can go to Settings --> "Health Report Manager" and then just search for iowait here you can enable the alert or edit the thresholds. Or you can use the health.conf and copy the stanza to system local like: [feature:iowait] display_name = IOWait indicator:avg_cpu__max_perc_last_3m:description = This indicator tracks the average IOWait percentage across all CPUs on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the percentage exceeds 1% and Red if it exceeds 3% during this window. indicator:avg_cpu__max_perc_last_3m:red = 3 indicator:avg_cpu__max_perc_last_3m:yellow = 1 indicator:single_cpu__max_perc_last_3m:description = This indicator tracks the IOWait percentage for the single most bottle-necked CPU on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the percentage exceeds 5% and Red if it exceeds 10% during this window. indicator:single_cpu__max_perc_last_3m:red = 10 indicator:single_cpu__max_perc_last_3m:yellow = 5 indicator:sum_top3_cpu_percs__max_last_3m:description = This indicator tracks the sum of IOWait percentage for the three most bottle-necked CPUs on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the sum exceeds 7% and Red if it exceeds 15% during this window. indicator:sum_top3_cpu_percs__max_last_3m:red = 15 indicator:sum_top3_cpu_percs__max_last_3m:yellow = 7

michel_wolf · ‎08-01-2021

Nope for me I decide to adjust the threshold or disable this health check because it´s meaningless without documentation to find out how and how often splunk will check this iowait. In generell I have no problems with my systems my searches are fast and I have no indexing delay so I think if you are interested in this the best option is to create a support case. I can´t find any searches which creates this data I think there is a "hidden" scripted input for this.

michel_wolf · ‎07-12-2021

Hi Matt, I tried this again with your new data and for me the search was the same to receive the results, maybe you have other field names for this here |rename latency_info{}.started as started, latency_info{}.task as task just look in some of your data and make sure if you look to the interesting fields the name for the started and task field are the same as in my rename command here, if not you have just to edit the rename command.

michel_wolf · ‎07-12-2021

If you just want to group all events by the same ID in your timerange you can use |stats values(*) as * by ID If you need to group them in different days you can to do: |bin span=1d _time |stats values(*) as * by ID _time

michel_wolf · ‎07-12-2021

Hi Bradd, every time I have this usecase I try to use 2 time picker for 2 different Inputs to build an inner and outer search with a | append command to compare two different time frames. Maybe this idea will help you. Michel

michel_wolf · ‎07-12-2021

Hey Johnson, I would try something like this index=<your_index> ID=* |bin span=1d _time |fields <all your fields> _time |stats values(*) as * by ID

michel_wolf · ‎07-12-2021

Hi korstiaans, you can try this: |makeresults |eval sample_field="[ \"842cef72-745d-463c-8b49-ce16ccc5ebd2\" ]" |rex field=sample_field "\[\s\"(?<new_field>.*)?\"" I don´t know if you have realy spaces between the [ and " Michel

michel_wolf · ‎07-12-2021

Hi Matt, maybe you can try something like this: source="test.json" host="splunk-aio01" sourcetype="_json" |rename latency_info{}.started as started, latency_info{}.task as task | eval temp_field=if(isnotnull(task),mvzip(started, task),"") | mvexpand temp_field | makemv temp_field delim="," | eval started=mvindex(temp_field,0) | eval task=mvindex(temp_field,1) |stats sum(started) by task Michel

michel_wolf · ‎07-12-2021

Hi, I think you need to ask your splunk admin, maybe you have some role restrictions here like:

michel_wolf · ‎07-12-2021

Hi Felix, you need to extract these requirements from your date field to filter it. You can perform your search for example like index=.... earliest=-7d@d latest=now to get all your data from the last 7 days. Then you can add something like this | eval day=strftime(_time,"%u") | search day IN (1 2 3 4 5) | eval timestampStart="05:30:00" | eval timestampEnd="22:30:00" | eval timestampDate = strftime(_time,"%H:%M:%S") | where timestampDate >= timestampStart AND timestampDate <= timestampEnd Michel

michel_wolf · ‎06-30-2021

Unfortunately no, in this configuration you can only see what the indicator should stand for, but not how the data is collected and evaluated, but I have made some progress and I was able to find out that in the app splunk_instrumentation the following searches are for it: instrumentation.usage.healthMonitor.report instrumentation.usage.healthMonitor.currentState And in the currentState search is the join to the data from the health.log for the Treshhold index=_internal earliest=-1d source=*health.log component=PeriodicHealthReporter node_path=splunkd.resource_usage.iowait In this case you can replace the iowait with the feature you want to look at in more detail The last step that is still missing is how splunk generates the health.log, since the state is already created in this case for the evaluation.

michel_wolf · ‎06-29-2021

Yes I think you have 2 options. 1. Route your data to your HF and drop the events there before you upload it to splunk cloud 2. If this is not possible you have to create a private app with your props and transforms which drops those events and try to upload this app to splunk cloud via the process in the docs

michel_wolf · ‎06-29-2021

I think what Splunk support means is that you can create your app locally as you described and then upload it accordingly. There is this article about that: https://docs.splunk.com/Documentation/SplunkCloud/8.2.2105/Admin/PrivateApps#Install_private_apps_on_Splunk_Cloud

michel_wolf · ‎06-29-2021

Yes this is ok, but if I take a look to /en-US/manager/system/health/manager or the documentation I found nothing about the calculation for the threshold and the link to the feature monitorings also don´t provide any information these data and calculation or I am blind. Maybe two examples: 1. https://docs.splunk.com/Documentation/Splunk/8.1.2/DMC/Usefeaturemonitoring#1._Review_the_splunkd_health_report In this Screenshot you can see the an error and some related INFO messages for this behavior to understand why this indicator is yellow or red. So I expect there is an alert savedsearch or something else like index=_internal sourcetype=splunkd component="CMMaster" "streaming error" | stats count as count | eval thresshold=if(count<10,"yellow","red") If you don´t have the information why the indicator is red or yellow the meaning of the indicator can be everything 2. You got an error like this: So how do you troubleshoot this warning or error? The only information you have is "System iowait reached yellow threshold of 1" , but I can not find anything on which data splunk calculates this information or how this data was generated. The only thing I can find is the settings of the thresholds, but nothing about the calculation for these threshholds what makes an alert for me meaningsless

michel_wolf · ‎06-29-2021

Hi all, in splunk there is always this icon next to your user for the "Health of Splunk Deployment". You can change these indicators and futures or their teshholds, but I can't find anything about what splunk actually does in the background to collect these values. You can find something like this in health.conf: [feature:iowait] display_name = IOWait indicator:avg_cpu__max_perc_last_3m:description = This indicator tracks the average IOWait percentage across all CPUs on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the percentage exceeds 1% and Red if it exceeds 3% during this window. indicator:avg_cpu__max_perc_last_3m:red = 3 indicator:avg_cpu__max_perc_last_3m:yellow = 1 indicator:single_cpu__max_perc_last_3m:description = This indicator tracks the IOWait percentage for the single most bottle-necked CPU on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the percentage exceeds 5% and Red if it exceeds 10% during this window. indicator:single_cpu__max_perc_last_3m:red = 10 indicator:single_cpu__max_perc_last_3m:yellow = 5 indicator:sum_top3_cpu_percs__max_last_3m:description = This indicator tracks the sum of IOWait percentage for the three most bottle-necked CPUs on the machine running the Splunk Enterprise instance, over the last 3 minute window. By default, this indicator will turn Yellow if the sum exceeds 7% and Red if it exceeds 15% during this window. indicator:sum_top3_cpu_percs__max_last_3m:red = 15 indicator:sum_top3_cpu_percs__max_last_3m:yellow = 7 I can´t find out how splunk generate this data and how this alert or indicator is created. There must be a kind of process like scripted input which execute a top command to look for the cpu wait time write it to the health.log in splunk ingest this log and a search which provide the information for these indicator.

michel_wolf · ‎06-07-2021

Hi Splunkers, When I jump from the Service Analyzer to a Deep Dive, ITSI uses a standard configuration to visualise the KPIs. I would like to change this for all services and possibly also set it as the default setting for new services. As an example, here is an option that I would like to change: In the URL of the Deep Dive, a DeepDiveID is specified, but this cannot be found in the Rest API. http://localhost:8000/en-US/app/itsi/deep_dive?savedDeepDiveID=60bdc692ae19663ea44e4aa3&earliest=1623013540&latest=1623056740&owner=nobody curl -k -u admin:password https://localhost:8089/servicesNS/nobody/SA-ITOA/itoa_interface/deep_dive/ Where are these settings set in the ITSI, or are they stored directly in javascript?

michel_wolf · ‎01-13-2020

Hello guys, I checked the splunk answers but I can´t find a solution for my problem. I have an indexer cluster with 2 idx and 2 sites and for my _internal index I get many small buckets. In the answers I found some notes about connection issues, but in this environment I don´t have connection problems. All splunk instances are installed in 7.3.3 I get the following error: I checked with the |dbinspect my _internal index and didn´t find any issues here: |dbinspect index=_internal |fields - splunk_server |table startEpoch endEpoch * Do you have any hints for me why new buckets are generated instead of using the existing one? I don´t change many things on the default configuration for the _internal index: /opt/splunk/bin/splunk cmd btool indexes --debug list _internal /opt/splunk/etc/slave-apps/customer_all_indexes/local/indexes.conf [_internal] /opt/splunk/etc/system/default/indexes.conf archiver.enableDataArchive = false /opt/splunk/etc/system/default/indexes.conf archiver.maxDataArchiveRetentionPeriod = 0 /opt/splunk/etc/system/default/indexes.conf assureUTF8 = false /opt/splunk/etc/system/default/indexes.conf bucketRebuildMemoryHint = auto /opt/splunk/etc/slave-apps/customer_all_indexes/local/indexes.conf coldPath = volume:main/_internaldb/colddb /opt/splunk/etc/system/default/indexes.conf coldPath.maxDataSizeMB = 0 /opt/splunk/etc/system/default/indexes.conf coldToFrozenDir = /opt/splunk/etc/system/default/indexes.conf coldToFrozenScript = /opt/splunk/etc/system/default/indexes.conf compressRawdata = true /opt/splunk/etc/system/default/indexes.conf datatype = event /opt/splunk/etc/system/default/indexes.conf defaultDatabase = main /opt/splunk/etc/system/default/indexes.conf enableDataIntegrityControl = false /opt/splunk/etc/system/default/indexes.conf enableOnlineBucketRepair = true /opt/splunk/etc/system/default/indexes.conf enableRealtimeSearch = true /opt/splunk/etc/system/default/indexes.conf enableTsidxReduction = false /opt/splunk/etc/system/default/indexes.conf fileSystemExecutorWorkers = 5 /opt/splunk/etc/system/default/indexes.conf frozenTimePeriodInSecs = 2592000 /opt/splunk/etc/slave-apps/customer_all_indexes/local/indexes.conf homePath = volume:main/_internaldb/db /opt/splunk/etc/system/default/indexes.conf homePath.maxDataSizeMB = 0 /opt/splunk/etc/system/default/indexes.conf hotBucketTimeRefreshInterval = 10 /opt/splunk/etc/system/default/indexes.conf indexThreads = auto /opt/splunk/etc/system/default/indexes.conf journalCompression = gzip /opt/splunk/etc/system/default/indexes.conf maxBloomBackfillBucketAge = 30d /opt/splunk/etc/system/default/indexes.conf maxBucketSizeCacheEntries = 0 /opt/splunk/etc/system/default/indexes.conf maxConcurrentOptimizes = 6 /opt/splunk/etc/system/default/indexes.conf maxDataSize = 1000 /opt/splunk/etc/system/default/indexes.conf maxGlobalDataSizeMB = 0 /opt/splunk/etc/system/default/indexes.conf maxGlobalRawDataSizeMB = 0 /opt/splunk/etc/system/default/indexes.conf maxHotBuckets = 3 /opt/splunk/etc/system/default/indexes.conf maxHotIdleSecs = 0 /opt/splunk/etc/system/default/indexes.conf maxHotSpanSecs = 432000 /opt/splunk/etc/system/default/indexes.conf maxMemMB = 5 /opt/splunk/etc/system/default/indexes.conf maxMetaEntries = 1000000 /opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroups = 8 /opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroupsLowPriority = 1 /opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedNoAcks = 300 /opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedWithAcks = 60 /opt/splunk/etc/system/default/indexes.conf maxTotalDataSizeMB = 500000 /opt/splunk/etc/system/default/indexes.conf maxWarmDBCount = 300 /opt/splunk/etc/system/default/indexes.conf memPoolMB = auto /opt/splunk/etc/system/default/indexes.conf minHotIdleSecsBeforeForceRoll = auto /opt/splunk/etc/system/default/indexes.conf minRawFileSyncSecs = disable /opt/splunk/etc/system/default/indexes.conf minStreamGroupQueueSize = 2000 /opt/splunk/etc/system/default/indexes.conf partialServiceMetaPeriod = 0 /opt/splunk/etc/system/default/indexes.conf processTrackerServiceInterval = 1 /opt/splunk/etc/system/default/indexes.conf quarantineFutureSecs = 2592000 /opt/splunk/etc/system/default/indexes.conf quarantinePastSecs = 77760000 /opt/splunk/etc/system/default/indexes.conf rawChunkSizeBytes = 131072 /opt/splunk/etc/slave-apps/_cluster/default/indexes.conf repFactor = auto /opt/splunk/etc/system/default/indexes.conf rotatePeriodInSecs = 60 /opt/splunk/etc/system/default/indexes.conf rtRouterQueueSize = 10000 /opt/splunk/etc/system/default/indexes.conf rtRouterThreads = 0 /opt/splunk/etc/system/default/indexes.conf selfStorageThreads = 2 /opt/splunk/etc/system/default/indexes.conf serviceInactiveIndexesPeriod = 60 /opt/splunk/etc/system/default/indexes.conf serviceMetaPeriod = 25 /opt/splunk/etc/system/default/indexes.conf serviceOnlyAsNeeded = true /opt/splunk/etc/system/default/indexes.conf serviceSubtaskTimingPeriod = 30 /opt/splunk/etc/system/default/indexes.conf splitByIndexKeys = /opt/splunk/etc/system/default/indexes.conf streamingTargetTsidxSyncPeriodMsec = 5000 /opt/splunk/etc/system/default/indexes.conf suppressBannerList = /opt/splunk/etc/system/default/indexes.conf suspendHotRollByDeleteQuery = false /opt/splunk/etc/system/default/indexes.conf sync = 0 /opt/splunk/etc/system/default/indexes.conf syncMeta = true /opt/splunk/etc/slave-apps/customer_all_indexes/local/indexes.conf thawedPath = $SPLUNK_DB/_internaldb/thaweddb /opt/splunk/etc/system/default/indexes.conf throttleCheckPeriod = 15 /opt/splunk/etc/system/default/indexes.conf timePeriodInSecBeforeTsidxReduction = 604800 /opt/splunk/etc/system/default/indexes.conf tsidxReductionCheckPeriodInSec = 600 /opt/splunk/etc/system/default/indexes.conf tsidxWritingLevel = 1 /opt/splunk/etc/slave-apps/customer_all_indexes/local/indexes.conf tstatsHomePath = volume:main/_internaldb/datamodel_summary /opt/splunk/etc/system/default/indexes.conf warmToColdScript = Thanks Michel

michel_wolf · ‎10-02-2018

No I don´t use SAML for identities in db connect I only use this to create a SSO with the permission (roles) db_connect_admin and db_connect_user. Now i found out if a Splunk user has a \ symbol in his logon name the app doesn´t work it makes no different if this is a SAML user or you create a domain\test user in your splunk enivornment directly. If you copy the domain\test user and name it domain.test with the same permission it works perfectly. So it looks for me like a bug in the db connect app.

michel_wolf · ‎10-02-2018

Hello, I have a problem with the Splunk DB connect app. I have two different types of users. For example, UserA (normal Splunk user) and domain\UserB which is a user I got from my SAML SSO configuration. Both users are in the same groups and have the same permissions. So now, I found out if I got a user with an . The DB Connect App doesn't work. I have the same issue if I create a local Splunk user domain\UserC. Is there something I can change in the db connect app or is there another work around? Michel

michel_wolf · ‎08-15-2018

Hi jkat54, thanks for your answer this will work for me in my scenario.

michel_wolf · ‎08-15-2018

Hello, Here is my scenario server: Splunk_A has index_a index_b and index_c Splunk_B has Index_d index_e and index_f Is it possible to copy only index_f from Splunk_B to Splunk_A and configure forwarding and receiving only for index_f on Splunk_B?

michel_wolf · ‎06-21-2018

Hi, I have a problem with a LDAP configuration I know there is a limit by 1000 users so I have change the following configuration authentication.conf sizelimit = 10000 limits.conf [ldap] max_users_to_precache = 10000 but it looks like this hasn´t impact of the max size of groups because it stops every time at 1000 groups. Any ideas what to do? Michel

michel_wolf · ‎10-05-2017

Hello guys, I have a problem with the "Cluster Map" so I have add a log 2 weeks ago and when I do a search about the last 6 weeks the "Cluster Map" shows me the right results, but if i lower the time range picker for example to the last 15 seconds where cant be any results, because I upload a static log and have no activ monitor on this file. I got the same results which I had before I decrease the time range picker and the job inspector says to me there are no results in this time range. This search has completed in 0.276 seconds, but did not match any events. The terms specified in the highlighted portion of the search: search (source="otrs.log" remote_addr:) | rex field=remote_addr "(? \d+.\d+.\d+).\d+" | iplocation remote_addr | geostats count over the time range: 10/5/17 4:33:27.000 PM - 10/5/17 4:33:42.000 PM But I get this map: If i extend the time pick ranger to "all time" and decrease it again to the last 15 seconds it will show me the results from the "all time" search, but if I decrease it to the last 6 weeks like in my sample picture I got the right results like above. For me it looks like the visualisation dosnt change if I got zero events. So I dont know if it is a bug in 6.6.3 and 7.0 or if something wrong with my search.

Posts	25
Solutions	2
Karma Given	12
Karma Received	13
Member Since	‎10-05-2017

Online Status	Offline
Date Last Visited	‎12-04-2025 11:24 PM

How can I effectively retrieve the SIDs for each c...

Dokumentation for "Health of Splunk Deployment" ca...

Splunk ITSI DeepDive default "Graph Rendering Opti...

The percentage of small buckets exceeded

Can you help me with a problem I'm having with the...

Can you forward specific indexed data from one ser...

LDAP with more then 1000 groups

Cluster Map doesnt show correct events

Re: Chained Search SID and Access in XML Dashboard

How can I effectively retrieve the SIDs for each c...

Re: Dokumentation for "Health of Splunk Deployment...

Re: Dokumentation for "Health of Splunk Deployment...

Re: Parse json array and plot to a line chart

Re: Returning events that have matching fields but...

Re: yesterday & last week data

Re: Returning events that have matching fields but...

Re: Replace square brackets and leave original val...

Re: Parse json array and plot to a line chart

Re: Search Help

Re: How could I look only at a defined time span i...

Re: Dokumentation for "Health of Splunk Deployment...

Re: Dropping Logs Sent Straight to Splunk Cloud

Re: Dropping Logs Sent Straight to Splunk Cloud

Re: Dokumentation for "Health of Splunk Deployment...

Dokumentation for "Health of Splunk Deployment" ca...

Splunk ITSI DeepDive default "Graph Rendering Opti...

The percentage of small buckets exceeded

Re: Can you help me with a problem I'm having with...

Can you help me with a problem I'm having with the...

Re: Can you forward specific indexed data from one...

Can you forward specific indexed data from one ser...

LDAP with more then 1000 groups

Cluster Map doesnt show correct events

Join the Conversation