We are planning upgrade our clustered deployment (SH-C + IDX-C) from 8.0.5 to 8.2.1 In case we plan to migrate the KVStore engine to WiredTiger, do we have to mandatorily go through the v8.1 step as mentioned in https://docs.splunk.com/Documentation/Splunk/8.2.1/Admin/MigrateKVstore ?  Per my understanding, the steps for this method stands like this Upgrade cluster from 8.0.5 to 8.1.5  Migrate the KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment Upgrade cluster from 8.1.5 to 8.2.1 In case we decide not to use WiredTiger for now and complete the upgrade to 8.2.1, can we migrate the KV  Store to WiredTiger at a later point of time?  To elaborate, will the following work:  Upgrade cluster from 8.0.5 to 8.2.1  At a later time, migrate the KV Store as instructed in Migrate the KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment We do not use KV Store at all now, apart from whatever internal functions that Splunk Enterprise uses it for (no ITSI or ES as well), but want to plan ahead in case we use it in the future.  Thanks in advance! ... View more

We are planning upgrade our clustered deployment from 8.0.5 to 8.2.1 In case we plan to migrate the KVStore engine to WiredTiger, do we have to mandatorily go through the v8.1 step as mentioned in https://docs.splunk.com/Documentation/Splunk/8.2.1/Admin/MigrateKVstore ?  Per my understanding, the steps for this method stands like this Upgrade cluster from 8.0.5 to 8.1.5  Migrate the KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment Upgrade cluster from 8.1.5 to 8.2.1 In case we decide not to use WiredTiger for now and complete the upgrade to 8.2.1, can we migrate the KV  Store to WiredTiger at a later point of time?  To elaborate, will the following work:  Upgrade cluster from 8.0.5 to 8.2.1  At a leter time, if WiredTiger is needed, migrate the KV Store as instructed in Migrate the KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment We do not use KV Store at all now, apart from whatever internal functions that Splunk Enterprise uses it for (no ITSI or ES as well), but want to plan ahead in case we use it in the future.  Thanks in advance! ... View more

Hello all, On the splunkd health report, what is the difference between Search Lag & Delay? [ref: https://docs.splunk.com/images/e/ee/Splunkd_health_report_8.0.0.png] Our deployment has a high number of savedsearches that trigger on cron (every 5m, 15m, 30m, 1h etc) and we are working to minimise the concurrency by introducing Scheduler Window & Skew. I know exactly which searches are triggering beyond the scheduled time (dispatch_time - scheduled_time from the scheduler.log) and which searches are skipping. But I do not understand what Splunk signifies as Lag & Delay in terms of searches.. I have gone through the $SPLUNK_HOME/var/log/health.log & server/health/splunkd/details API endpoints but they give the same messages as the Health Indicator.. Thanks in advance! ... View more

First question: If I run the below command, I get four different values (historical, historical batch, real-time, or real-time indexed) for data.search_props.mode index IN (_introspection) sourcetype=splunk_resource_usage component=PerProcess data.search_props.sid=* | stats count by data.search_props.type data.search_props.mode What exactly is the difference between "historical" & "historical batch"? When is "historical" mode used over "historical batch"? What impacts does these modes have over searches running on a multisite indexer cluster with a multisite stretched searchhead cluster? Splunk Docs has this https://docs.splunk.com/Documentation/Splunk/7.2.6/Troubleshooting/Sampleplatforminstrumentationsearches#Median_search_concurrency_by_search_mode_for_all_instances , but they didnt explain what the different modes signify. Second Question: I found different forms of search IDs in the introspection logs. Samples: - 1576061498.2185156_3863A2B4-1AB7-42DB-ACD3-4484EDD006D2 - 1576063020.97915 - userid__userid__search__search12_1576060747.2056595_CD71F91B-FF33-490B-8C4B-EE986A5C4E6F - subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3 - remote_hostname_1576063020.3359 - remote_hostname_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675 - remote_hostname_subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3 I understand that sids with userid__userid__search__search12 spawn from dashboard panels, subsearch_userid__userid_ spawn from subsearches in dashboard panels, sids like 1576063020.97915 are searches run from the search box, etc.. But the ones like 1576061498.2185156_3863A2B4-1AB7-42DB-ACD3-4484EDD006D2 , remote_hostname_1576063020.3359 , remote_hostname_subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3 ... - What are origins of these search sids? - How is any given sid generated? The idea is to come up with a way to correlate these sids from the _introspection logs to the _audit so we can identify the searches/dashboards/alerts that take up most resources on our Splunk platform and tune them. Third Question What is the use of data.pid ? How do I leverage this? Google didn't come up with anything solid. Please help me understand these. Any documentation links/answers would be greatly appreciated helpful. ... View more

Hello Splunkers, I am trying my hand at building modular inputs using the Splunk AoB, and following the walk-through examples provided for REST API at http://dev.splunk.com/view/addon-builder/SP-CAAAFCA . Followed the steps right down to each word to get a feel of the app.. I used the same set of API for NYTimes. The tests performed while building the input worked and provided output JSON. The Interval is set for 30s. I have created two inputs, indexing data to two separate indexes. However, data is not indexed. There is no activity logged by the two inputs in the _internal index as well. I have restarted Splunk, but no result as well. The AoB docs do not mention anything about indexing data via the created Add Ons Can someone please help? Note: I am running the created Add-On on the same instance as the AoB.. does taht make any difference? ... View more

Hello Splunkers!! Apologies for the wall of text below, but my urge to explain the situation has overcome everything else. Scenario: SQL Server stores factory floor data. New row is appended for each new Shift. So, we have unique rows for each DAY and each Shift that runs at the factory. Hence, each combo of DAY-Shift will be unique in the DB. For any changes in the data for a DAY_Shift, it is updated in the existing row. To allow for corrections, the data for any shift can be updated within a time window of 72hrs from the shift start, after which it is frozen and no further updates can be done. A timestamp column [col_TS, factory local time] holds the time of last change/update. Thus, this can be anytime within the next 72hrs of the shift start time. Hence, these updates can be within a span of 3 Days. DBConnect monitors a Rising Column [col_TS] and index time is also the col_TS timestamp, with properly configured time zones. The problem is with the sentence in bold: Hence, these updates can be within a span of 3 Days. By this logic, data for production day 2018-10-01 [DAY] can come in within the span of 2018-10-01 till 2018-10-04 [col_TS & index _time]. Thus I can have multiple entries for 2018-10-01 - Shift 1 over 2018-10-01, 2018-10-02, 2018-10-03 & 2018-10-04 and just pick the latest one. However, if a user using the Splunk Time Picker wants to see a report on a span of time where 2018-10-01 is the last day, the last three day's values will not be the latest. Because Time Picker will translate values into earliest and latest and run them on the first line, against the col_TS and hence filter out all data outside the selected time span. How do I get rid of this and use something like latest + 72hrs ?? . A dirty workaround that I have thought of is to provide two different time picker buttons that will return epoch time to the SPL query. The Start Time will be a straight conversion. The End Time will be User Selected Time + 72hrs converted to epoch. 😛 Regards & Thanks Anirban. EDIT: https://answers.splunk.com/answers/322027/splunk-db-connect-is-there-a-way-to-configure-a-da.html @rich7177 's answer is promising. putting that here for my reference. _time col_TS Day ShiftCode Quantity 2018-10-02 06:10:02 2018-10-02 06:10:02.0 2018-10-01 1 689.7239999999999 2018-10-02 06:10:02 2018-10-02 06:10:02.0 2018-10-01 1 689.7239999999999 2018-10-02 05:52:03 2018-10-02 05:52:03.0 2018-10-01 1 689.7239999999999 2018-10-02 05:37:05 2018-10-02 05:37:05.0 2018-10-01 2 3965.2490000000003 2018-10-02 05:37:05 2018-10-02 05:37:05.0 2018-10-01 2 3965.2490000000003 2018-10-02 05:24:03 2018-10-02 05:24:03.0 2018-10-01 2 3965.2490000000003 2018-10-02 05:24:03 2018-10-02 05:24:03.0 2018-10-01 1 689.7239999999999 2018-10-02 04:52:02 2018-10-02 04:52:02.0 2018-10-01 1 689.7239999999999 2018-10-02 04:52:02 2018-10-02 04:52:02.0 2018-10-01 2 3607.0090000000005 2018-10-02 04:24:02 2018-10-02 04:24:02.0 2018-10-01 1 689.7239999999999 2018-10-02 04:24:02 2018-10-02 04:24:02.0 2018-10-01 2 3245.649 2018-10-02 03:52:03 2018-10-02 03:52:03.0 2018-10-01 1 689.7239999999999 2018-10-02 03:52:03 2018-10-02 03:52:03.0 2018-10-01 2 2883.799 2018-10-02 03:24:02 2018-10-02 03:24:02.0 2018-10-01 1 689.7239999999999 2018-10-02 03:24:02 2018-10-02 03:24:02.0 2018-10-01 2 2883.799 2018-10-02 02:52:02 2018-10-02 02:52:02.0 2018-10-01 1 689.7239999999999 2018-10-02 02:52:02 2018-10-02 02:52:02.0 2018-10-01 2 2883.799 2018-10-02 02:24:02 2018-10-02 02:24:02.0 2018-10-01 1 689.7239999999999 2018-10-02 02:24:02 2018-10-02 02:24:02.0 2018-10-01 2 2521.799 2018-10-02 01:52:02 2018-10-02 01:52:02.0 2018-10-01 1 689.7239999999999 2018-10-02 01:52:02 2018-10-02 01:52:02.0 2018-10-01 2 2521.799 2018-10-02 01:24:02 2018-10-02 01:24:02.0 2018-10-01 1 689.7239999999999 2018-10-02 01:24:02 2018-10-02 01:24:02.0 2018-10-01 2 2521.799 2018-10-02 00:52:03 2018-10-02 00:52:03.0 2018-10-01 1 689.7239999999999 2018-10-02 00:52:03 2018-10-02 00:52:03.0 2018-10-01 2 2521.799 2018-10-02 00:24:03 2018-10-02 00:24:03.0 2018-10-01 1 689.7239999999999 2018-10-02 00:24:03 2018-10-02 00:24:03.0 2018-10-01 2 2163.3289999999993 2018-10-01 23:52:00 2018-10-01 23:52:00.0 2018-10-01 1 689.7239999999999 2018-10-01 23:52:00 2018-10-01 23:52:00.0 2018-10-01 2 1804.4389999999999 2018-10-01 23:24:00 2018-10-01 23:24:00.0 2018-10-01 1 689.7239999999999 2018-10-01 23:24:00 2018-10-01 23:24:00.0 2018-10-01 2 1442.249 2018-10-01 22:52:00 2018-10-01 22:52:00.0 2018-10-01 1 689.7239999999999 2018-10-01 22:52:00 2018-10-01 22:52:00.0 2018-10-01 2 1080.5890000000002 2018-10-01 22:24:00 2018-10-01 22:24:00.0 2018-10-01 1 689.7239999999999 2018-10-01 22:24:00 2018-10-01 22:24:00.0 2018-10-01 2 722.209 2018-10-01 21:52:01 2018-10-01 21:52:01.0 2018-10-01 1 689.7239999999999 2018-10-01 21:52:01 2018-10-01 21:52:01.0 2018-10-01 2 722.209 2018-10-01 21:24:00 2018-10-01 21:24:00.0 2018-10-01 1 689.7239999999999 2018-10-01 21:24:00 2018-10-01 21:24:00.0 2018-10-01 2 360.869 2018-10-01 20:52:00 2018-10-01 20:52:00.0 2018-10-01 1 689.7239999999999 2018-10-01 20:52:00 2018-10-01 20:52:00.0 2018-10-01 2 0.0 2018-10-01 20:24:00 2018-10-01 20:24:00.0 2018-10-01 1 689.7239999999999 2018-10-01 20:24:00 2018-10-01 20:24:00.0 2018-10-01 2 0.0 2018-10-01 19:52:00 2018-10-01 19:52:00.0 2018-10-01 1 689.7239999999999 2018-10-01 19:52:00 2018-10-01 19:52:00.0 2018-10-01 2 0.0 2018-10-01 19:24:00 2018-10-01 19:24:00.0 2018-10-01 1 689.7239999999999 2018-10-01 19:24:00 2018-10-01 19:24:00.0 2018-10-01 2 0.0 2018-10-01 18:52:00 2018-10-01 18:52:00.0 2018-10-01 1 689.7239999999999 2018-10-01 18:52:00 2018-10-01 18:52:00.0 2018-10-01 2 0.0 2018-10-01 18:24:00 2018-10-01 18:24:00.0 2018-10-01 1 689.7239999999999 ... View more

Hi Splunkers, I have a few saved searches that query SQL DB via dbxquery and perform some calculations. The search CRON schedule is set to */30 * * * * . The dbxquery is something like : SELECT top 20 * FROM tbl WHERE unitname LIKE '%integrated%' ORDER BY day DESC, shiftcode DESC I have kept them as scheduled reports so that I can quickly check the cached results and not have to wait for the query to execute every time. However, frequently the following happens: This scheduled report runs on cron schedule */30 * * * *. Its time range is last 60 minutes. The following results were generated an hour ago. This should not happen. As per the CRON exp., the results should always be generated less than 30mins back. I have checked scheduler.log, the reports run successfully every 30mins as per the CRON exp. A sample event in scheduler.log for this savedsearch is as follows [I am masking some sensitive data with ***]: 09-04-2018 09:00:43.048 +0000 INFO SavedSplunker - savedsearch_id="nobody;search;***IntegratedLines_30min", search_type="scheduled", user="***", app="search", savedsearch_name="***IntegratedLines_30min", priority=default, status=success, digest_mode=1, scheduled_time=1536051600, window_time=-1, dispatch_time=1536051604, run_time=8.906, result_count=35, alert_actions="", sid="***_at_1536051600_151", suppressed=0, thread_id="AlertNotifierWorker-0" host=M***1 source=/opt/splunk/var/log/splunk/scheduler.log sourcetype=scheduler The data that is generated is not refreshed as well. If I run the query manually, I get the refreshed data. Please note that the time range for the search does not matter since we are not working on indexed data. Is this something related to dbxquery, or am I missing out something? Let me know if you guys need more information. Thanks in advance! ... View more

Hello splunkers, We have to calculate some KPIs, on time-series data, aggregated by multiple factors, with time being the most important one. The most common time aggregations for us are monthly and weekly. Scenario: Calculate the weekly trend of a KPI for the month of May 2018 [ref image attached] How one would normally do it would be [assume week start on Sunday]: Week Date 1: 1 - 5 2: 6 - 12 3: 13 - 19 4: 20 - 26 5: 27 - 31 Thus there will be 5 data points. How do I implement this in Splunk? The way I figured out to do it now is use "bin span=1w _time", but it does not detect the week start. However, this approach plainly takes 7 days from the 1st of the month and the result is that we have skewed weeks leading to misleading KPI values. ... View more