About yshen

yshen · ‎06-22-2023

@yuanliu Thanks for concurring @VatsalJagani 's approach, especially optimizing the multiple parallel sub-searches by the constraints of earliest and latest! I think your suggestion may be the best solution that I learned so far. I'm using this more convoluted (challenging) problem to help me to learn of the nuances in Splunk query execution. Unfortunately, I find it extremely hard to find more in depth discussion of Splunk queries' execution behavior. The Splunk's own documentation is too sketchy of the nuances. It makes too easy for toy problems. What is your recommendation to learn more of Splunk queries for such more nuanced behaviors/performance. Besides, I'd like to leave behind a query expression that is more intuitive, more readable, when I'm no longer around to maintain it. This map expression is a lot intuitive than the convoluted list gymnastics to deal with the results of lookup by location with multiple time windows of overheat for a same location. We want more than doable, but performant, and readable code!

yshen · ‎06-21-2023

@yuanliu Thank you for very detailed review and explanation! Your understanding of the my requirements is almost perfect, except some minor naming difference. I had an almost identical solution as yours. I was worrying the first part of the query for all locations, and for all the times (theoretically speaking the overheat could last from the beginning of the ingested data to the moment of computing its average, but mostly likely, the overheat should be rare and of short duration, not too long before the current time of calculation. It seems the blind query for all temperatures would be rather wasteful. Especially, the performance might get worse over the increasing time of the temperature observations collected. Besides, I find dealing with both the list of start_time_secs, end_time_secs rather complicated after the lookup operation, albert doable. Here is my solution. In order to reduce the impact of excessive delays in query all temperature data, I'm limiting the query to only last 24 hours. Thus the implementation is correct for the overheat that had happened not earlier than 24 hours before. (I was just wondering if there were a better solution.) `bsm_battery_data_parsed_by_time_window(-24h, now)` | lookup bsm_string_new_overheat_records.csv CCU_location OUTPUT start_time_secs end_time_secs overheat_duration_minutes | where isnotnull(start_time_secs) | eval interval=mvzip(start_time_secs, end_time_secs) | mvexpand interval | rex field=interval "(?<start_secs>\S+),(?<end_secs>\S+)" | where (start_secs <= _time) AND (_time <= end_secs) | stats avg(CCU_AMBI_TEMP) as average_temperature_in_interval latest(*) AS * BY CCU_location start_secs end_secs | eval average_temperature_in_interval=round(average_temperature_in_interval, 2) | convert ctime(start_secs) AS start_time ctime(end_secs) AS end_time | outputlookup bsm_battery_string_overheat_records.csv append=true create_empty=true The definition of the macro is index="battery_data" sourcetype="battery_field_data" earliest=$earliest$ latest=$latest$ | rex field=Tag "^(?P<CCU_location>\w+)_BQMS\.\1\.((BMS_\1_(?P<tag_suffix>.*))|(MU(?P<MU_number>\d+)\.BMS_\1_MU\6_UNIT(?P<UNIT_number>\d+)_(?P<tag_type>.*)))" If this is the best with Splunk, then my quest for more perfection can be closed.

yshen · ‎06-20-2023

I think that this is the closest semantics to what I want. I'll give it a try. Note, maxcount should be maxsearches, and maxsearches=0 does not mean unlimited searches. maxsearches Syntax: maxsearches=<int> Description: The maximum number of searches to run. A message is generated if there are more search results than the maximum number that you specify. Zero ( 0 ) does not equate to unlimited searches. Default: 10 https://docs.splunk.com/Documentation/Splunk/9.0.5/SearchReference/Map

yshen · ‎06-10-2023

I have a lookup table bsm_string_new_overheat_records.csv: _time overheat_location start_CCU_AMBI_TEMP start_time_secs end_CCU_AMBI_TEMP end_time_secs overheat_duration_minutes 2023-06-10 11:41:44 A10 29 1686421604 23 1686422504 15 2023-06-10 11:42:44 A10 29 1686413444 23 1686422564 152 2023-06-10 11:43:44 A10 29 1686412844 23 1686422624 163 2023-06-10 11:44:44 A10 29 1686413984 23 1686422684 145 2023-06-10 11:45:44 A10 29 1686420584 23 1686422744 36 I also have an `index==“battery_data” sourcetype=“battery_field_data”` that contains the battery data. From this, I can compute the average temperature for a location (“A10”), with start_time_secs (1686421604), and end_time_secs (1686422504). Here is an example: index="battery_data" sourcetype="battery_field_data" | rex field=Tag "^(?P<CCU_location>\w+)_BQMS\.\1\.((BMS_\1_(?P<tag_suffix>.*))|(MU(?P<MU_number>\d+)\.BMS_\1_MU\6_UNIT(?P<UNIT_number>\d+)_(?P<tag_type>.*)))" | where CCU_location=="A10" | eval start_time_secs=1686421604, end_time_secs=1686422504 | where (tag_suffix == "CCU_AMBI_TEMP") AND (start_time_secs <= _time) AND (_time <= end_time_secs) | eval Value=round(Value*0.1, 2) | stats avg(Value) AS average_temperature_in_overheat latest(_time) AS _time latest(*) AS * BY CCU_location start_time_secs end_time_secs | table _time CCU_location start_time_secs end_time_secs average_temperature_in_overheat yielding: _time CCU_location start_time_secs end_time_secs average_temperature_in_overheat 2023-06-10 11:39:54.971 A10 1686421604 1686422504 17.45 I would like to compute the average temperature for each record in bsm_string_new_overheat_records.csv with the following query, | inputlookup bsm_string_new_overheat_records.csv | rename CCU_location AS overheat_location | appendpipe [ search index="battery_data" sourcetype="battery_field_data" | rex field=Tag "^(?P<CCU_location>\w+)_BQMS\.\1\.((BMS_\1_(?P<tag_suffix>.*))|(MU(?P<MU_number>\d+)\.BMS_\1_MU\6_UNIT(?P<UNIT_number>\d+)_(?P<tag_type>.*)))" | where CCU_location==overheat_location | where (tag_suffix == "CCU_AMBI_TEMP") AND (start_time_secs <= _time) AND (_time <= end_time_secs) | eval Value=round(Value*0.1, 2) | stats avg(Value) AS average_temperature_in_overheat latest(_time) AS _time latest(*) AS * BY CCU_location start_time_secs end_time_secs ] | table _time overheat_location CCU_location start_time_secs end_time_secs average_temperature_in_overheat start_CCU_AMBI_TEMP end_CCU_AMBI_TEMP overheat_duration_minutes I’d expect one additional record of average temperature for each overheat record (total 5+5=10 records). However, here is what I got: _time overheat_location CCU_location start_time_secs end_time_secs average_temperature_in_overheat start_CCU_AMBI_TEMP end_CCU_AMBI_TEMP overheat_duration_minutes 2023-06-10 11:41:44 A10 1686421604 1686422504 29 23 15 2023-06-10 11:42:44 A10 1686413444 1686422564 29 23 152 2023-06-10 11:43:44 A10 1686412844 1686422624 29 23 163 2023-06-10 11:44:44 A10 1686413984 1686422684 29 23 145 2023-06-10 11:45:44 A10 1686420584 1686422744 29 23 36 None of the average temperature results is there. What’s wrong with my above query?

yshen · ‎09-22-2022

I see an interesting Simple XML idiom below: <input type="multiselect" token="multiselect_lines" searchWhenChanged="true"> <label>Lines</label> <choice value="ACEKLMRSWY">All lines</choice> <choice value="A">A Line</choice> <choice value="C">C Line</choice> <choice value="E">E Line</choice> <choice value="K">K Line</choice> <choice value="L">L Line</choice> <choice value="M">M Line</choice> <choice value="R">R Line</choice> <choice value="S">S Line</choice> <choice value="W">W Line</choice> <choice value="Y">Y Line</choice> <default>ACEKLMRSWY</default> <prefix>regex Location="^[</prefix> <suffix>]"</suffix> <change> <eval token="form.multiselect_lines"> case( mvcount('form.multiselect_lines') == 2 AND mvindex('form.multiselect_lines', 0) == "ACEKLMRSWY", mvindex('form.multiselect_lines', 1), mvfind('form.multiselect_lines', "ACEKLMRSWY") == mvcount('form.multiselect_lines') - 1, "ACEKLMRSWY", true(), 'form.multiselect_lines')</eval> </change> </input> It seems updating the appearance of the multiselect field "multiselect_lines" so whenever the selections in the multiselect change, "form.multiselect_lines" will be updated accordingly. I guess that it is supposed to solve the deficiency of multiselect in Splunk that the option of "All" does not disappear automatically when a subset is selected, or when there is no more subset selected, "All" as default does not come back automatically. The above is my trying to understand to achieve the functionality. It works as hypothesized in a dashboard that I'm studying, but when I copied the mechanism to my dashboard, it has no effect in the behavior. So I under what the token with the pattern of form.<multiselect_input_token>, and what does it take to have the above mechanism work in auto removing and adding "All" in appearance? I know that there is a javascript solution by modifying the list of multiselect options on the fly through Javascript. But I don't have the admin privilege to add the javascript for my dashboard. So a solution without requiring admin privilege is handy.

yshen · ‎08-17-2022

I further confirmed that the problem can be reproduced reliably by reloading the web page of the dashboard. When there is already working visualization, once I reload, then the visualization will be gone.

yshen · ‎08-17-2022

I'm running into a strange behavior: For the first time opening my dashboard, the dashboard always shows no visualization for the data, as if the query for the dashboard has not been executed or the query produces no data. Simply reloading the page still does not show the visualization for the queried data. However, if I edit the input field from which the query uses the token value for the query, as long as the value entered is different from the existing value, the dashboard will show the expected visual. But if I just reload the page again, then the visual would disappear! The dashboard uses a Splunk extension written in Javascript&colon; But the same extension works with another dashboard without the reloading problem. How can I approach solving the mystery?

yshen · ‎07-15-2022

@gcusello Thanks for the pointers. They are indeed helpful. Lookup table or summary index, which is more appropriate? I'm not yet fluent in Splunk query programming. I wonder if Splunk custom alert action script would be an easier alternative to implement the evaluation logic? Any example of similar query programming, or of custom alert action script would be appreciated! Thanks again!

yshen · ‎07-14-2022

I need to first issue an alert for overheat temperature 24 hours in advance for the affected locations, for their forecast to be above 100 F (long term query). Then I need to query for the next 2 hours to 8 hours (for near term forecast), of the more recent temperature forecast for the same sets of locations. If the recent forecast for the same location has dropped below the threshed 100 F, I need to issue an alert to cancel the previous alert. If a location's recent forecast is above 100 F, but the prior forecast was below 100 F (no alert had been issued), I need to issue a new alert for the location. Effectively, the query for near term forecast needs to access the query results of the long term query (or redo a query for the previous long term query), to compare with the recent forecast results. (I'm especially not clear how to compare two queries' results with Splunk query.) I wonder how to implement a solution with Splunk? Thanks for pointers! Let's build an example to develop the solution. Assume the operation time in question is 8:00 AM on July 14, 2022, so the 24 hour in advance long term forecast should have been made at 8:00 AM on July 13, 2022 (long term forecast) The time window to make the short term forecast should be 0:00 AM (8-8) and 6:00 AM (8-2) (8 to 2 hours before) on the same day. Here is more concise requirements: 1. Hourly, the forecasts of 24 hours after for all locations shall be collected and evaluated. If the 24-hour-after temperature will be over the threshold (100 F), alert shall be sent for the to-be-overheat locations. 2. Also hourly, the forecasts for the window of next 2 hours to the next 8 hours should be collected and evaluated. Based on the evaluation of the 2-hours-8-hours-after forecast, revision shall be made according to the following rules: a. If a location’s 2-hours-8-hours-after forecast is below the threshold, while there had been an alert issued. A cancellation message shall be sent. b. If a location’s 2-hours-8-hours-after forecast is above the threshold, while there had not been alert sent, then a new alert shall be sent c. For the other case, no operation is needed 3. At 15 minutes interval, the real time temperature for the locations shall be collected and evaluated. . Based on the evaluation of the real-time temperature, revision shall be made according to the following rules: a. If a location’s real time temperature is below the threshold, while there had been an alert issued. A cancellation message shall be sent. b. If a location’s real time temperature is above the threshold, while there had not been alert sent, then a new alert shall be sent c. For the other case, no operation is needed

yshen · ‎07-14-2022

Thanks for the perfect solution!

yshen · ‎07-14-2022

I want to compare the daily temperature measurements at the same period, but different days by a stacked temperature time series for multiple days. Using timechart I have the following query to organize the data, as the _time value contains the date information, the resulted visualization yields no stacked but one after another. index="weather" sourcetype=publicweatherdata (Location=C60*) | fields _time, Location, Temperature | eval Date=strftime(_time, "%D") | timechart span=30m max(Temperature) AS Temperature BY Date I tried to only retain the hour, minutes in _time, resulting all _time value of the date of 20222-07-06, when I executed the query, I could have the time series chart stacked but it shows with much of the horizontal space blank! Here is the query alternative: index="weather" sourcetype=publicweatherdata (Location=C60*) | fields _time, Location, Temperature | eval Date=strftime(_time, "%D") | eval hour_min=strftime(_time, "%H:%M") | eval _time = strptime(hour_min, "%H:%M") | timechart span=30m max(Temperature) AS Temperature BY Date How can I improve the visualization to make time series stacked with x-axis free from the dates? Below are the charts needing improvement. Thanks!

yshen · ‎04-07-2022

I also note that with Splunk SDK (Python), at the end of the embedded query, using 'fields' to select the returned fields, it does not work as I desired with all fields returned. But 'table' would result in only the listed fields returned.

yshen · ‎04-07-2022

With hint by https://splunk-usergroups.slack.com/team/UB5DA9L02, it turns out that as the sourcetype is only known in the context of my application ics_analytics, in the service definition with SDK, I must indicate the application context with app= argument. Here is the corrected service definition: service = client.connect( host= 'splunk.bart.gov', app='ics_analysis', port = '8089', username = 'userid', password = 'secrete', ) once the sourcetype is properly declared to be known, the same code as above would be able to retrieve the field value of ENTRY. Here is the link to the relevant documentation: https://docs.splunk.com/DocumentationStatic/PythonSDK/1.6.16/client.html#splunklib.client.Service This is post is a capture of Slack discussion: https://splunk-usergroups.slack.com/archives/C04DC8JJ6/p1649351828984919?thread_ts=1649265592.685629&cid=C04DC8JJ6

yshen · ‎04-07-2022

When I used the following code to perform a query: service = client.connect( host= 'splunk.bart.gov', port = '8089', username = 'userid', password = 'secrete', ) query = "search index=slog_ics sourcetype=occ_mgr | table _time, ENTRY | head 3" query_results = service.jobs.oneshot(query) reader = res.ResultsReader(query_results) results = [] for item in reader: print(item) results.append(item) print("results[1]:") print(results[1]) In the above result, I cannot see the value for the field ENTRY. ENTRY is a field defined by the sourcetype occ_mgr in my application ics_analytics. While in Splunk web UI, in the context of the application ics_analytics using the same query, I can see the field value of ENTRY: index=slog_ics sourcetype=occ_mgr | fields _time, ENTRY | head 3 with the result: _time ENTRY 4/6/22 2:11:00.000 AM EOR. 4/6/22 1:48:00.000 AM (ref 0120) T203 released ATO, (762) second delay. 4/6/22 1:36:00.000 AM CORE Blanket established. What could be the root cause of the problem?

yshen · ‎02-12-2022

@gcusello Clever idea! I tried your suggestion. Here is the outcome: I use https://splunk.bart.gov/en-US/static/app/ics_analysis/../../weatherAssets/lowTemp.png it is translated by the browser (Chrome) or Splunk server as: https://splunk.bart.gov/en-US/static/weatherAssets/lowTemp.png again failed. I tried: https://splunk.bart.gov/en-US/static/app/ics_analysis/../weatherAssets/lowTemp.png it is converted as https://splunk.bart.gov/en-US/static/app/weatherAssets/lowTemp.png It looks that the browser performed the translation of the /.. literally before reaching the web server. Yes, I may try to manually upload the asset individually as last resort. Indeed, the manual approach works! Here is the proof: At https://splunk.bart.gov/en-US/static/app/ics_analysis/lowTemp.png The indented image is shown. Furthermore, the following also works in the dashboard: | eval coldImg = "/static/app/ics_analysis/lowTemp.png" Disappointed with Splunk's implementation, I guess, this is the best I can do. (Why the disappointment, we're working with a big app, there are 3 to 5 engineers improving it, if I have to manually add about a dozen icons to apps/ics_analysis/appserver/static, suffering from the unnecessary repetition. That directory will be a big soup of mess, that no one can understand the files left there completely.) Thanks again!

yshen · ‎02-11-2022

@gcusello Thank for your further help! From the information that you shared me, it seems that the image MUST be at /opt/splunk/etc/apps/ics_analysis/appserver/static/lowTemp.png ? I wonder if it will work as follows: /opt/splunk/etc/apps/ics_analysis/appserver/static/weatherAssets/lowTemp.png as the application is large. I hope to have better separation within. But even more unfortunate, the administrator is not responsive to help me at the moment. Therefore, I'm very interested in your mentioned "absolute path". I wish that it would refer and access to my image at its current, non-standard location. But I could not find any information of how to express the absolute path in the Simple XML. Could you elaborate it? Thanks again!

yshen · ‎02-11-2022

@gcusello thanks for helping! As I don't have the write access to the Splunk server, I had to ask the admin to place the folder weatherAssets for me. Now, it has been placed at /opt/splunk/etc/apps/ics_analysis/weatherAssets/lowTemp.png I wonder if there is a way to use the above path somehow still make the image accessible to the dashboard? Or do I have to place the image at /opt/splunk/etc/apps/ics_analysis/appserver/static/lowTemp.png to make it work? Where is the documentation for the rule of placing image? Thanks again!

yshen · ‎02-10-2022

I use the following to define an icon, to display on my dashboard: eval coldImg = "/weatherAssets/apps/ics_analysis/lowTemp.png" in the Simple XML for the dashboard. Here is the path for the image: /opt/splunk/etc/apps/ics_analysis/weatherAssets/lowTemp.pngwhere ics_analysis is the name of the app and weatherAssets is the folder for the icons. It used to display, when I had the following: eval coldImg = "https://image.flaticon.com/icons/png/512/1312/1312331.png" but now it only shows a broken image icon. What could be wrong? How can I debug the problem? It's frustrating that I don't know how to find out the error message to the issue. Do I have to restart the Splunk server, or bump my dashboard? (I just did reload the web page.) Thanks for your help!

yshen · ‎01-28-2022

Thanks for the solution! Here is a minimum working example: <form> <label>Select between Current and Forecast Weather</label> <fieldset submitButton="false"> <input type="radio" id="forecast" token="forecastToken" searchWhenChanged="true"> <label></label> <choice value="0">Current</choice> <choice value="1">Forecast</choice> <default>0</default> <change> <condition value="0"> <set token="showCurrent">1</set> <unset token="showForecast"></unset> </condition> <condition value="1"> <set token="showForecast">1</set> <unset token="showCurrent"></unset> </condition> </change> <initialValue>0</initialValue> </input> </fieldset> <row> <panel depends="$showCurrent$"> <single> <title>Weather</title> <search depends="$showCurrent$"> <query>| makeresults | eval weather="Current"</query> <earliest>-7d@h</earliest> <latest>now</latest> </search> <option name="drilldown">none</option> </single> </panel> <panel depends="$showForecast$"> <single> <title>Weather</title> <search depends="$showForecast$"> <query>| makeresults | eval weather="Forecast"</query> <earliest>-7d@h</earliest> <latest>now</latest> </search> <option name="drilldown">none</option> </single> </panel> </row> </form>

yshen · ‎01-28-2022

As follows if <condition-based-on-token-value> then query_1 else query_2where query_1 and query_2 may be a series of statements producing different sets of data. Given a search query can be embedded in a panel, it might achieve the equivalent results, if there were a way to have conditional panel selection: if <condition-based-on-token-value> then panel_with_query_1 else panel_with_query_2 (The above idea is inspired by @mmccul) but I don't know if it's possible to have such panel selection mechanism with Simple XML of Splunk?Or alternatively, if I could control the visibility of a panel based token value, then I might also achieve the panel selection mechanism: define two panels with visibility control by the token value the controls are mutually exclusive, so that only one panel will be shown I'd appreciate some pointers or examples. (edited)

Posts	57
Solutions	1
Karma Given	51
Karma Received	4
Member Since	‎06-18-2020

Online Status	Offline
Date Last Visited	‎06-23-2023 01:36 PM

What is wrong with my sub-pipeline in appendpipe?

What's the meaning and mechanism of form.multisele...

Having problem of not seeing the expected visualiz...

How to implement alert that need to consider state...

How to have multiple time series charts of multipl...

Fields defined by a sourcetype not being shown in ...

How to make local image accessible to dashboard?

Is it possible to have query/panel with conditiona...

How to compute the field value a time interval ear...

How can I dynamically overlay graphic objects (ico...

Re: What is wrong with my sub-pipeline in appendpi...

Re: What is wrong with my sub-pipeline in appendpi...

Re: What is wrong with my sub-pipeline in appendpi...

What is wrong with my sub-pipeline in appendpipe?

What's the meaning and mechanism of form.multisele...

Re: Having problem of not seeing the expected visu...

Having problem of not seeing the expected visualiz...

Re: How to implement alert that need to consider s...

How to implement alert that need to consider state...

Re: How to have multiple time series charts of mul...

How to have multiple time series charts of multipl...

Re: Fields vs table vs nothing?

Re: Fields defined by a sourcetype not being shown...

Fields defined by a sourcetype not being shown in ...

Re: How to make local image accessible to dashboar...

Re: How to make local image accessible to dashboar...

Re: How to make local image accessible to dashboar...

How to make local image accessible to dashboard?

Re: If it is possible to have query/panel with con...

Is it possible to have query/panel with conditiona...