Some facts beforehand DB View Setup We are in Timezone Europe/Zurich, that is CEST+0200 including daytime savings. Since the switch to daytime savings on 29. March, we're having issues with the time conversion on our Splunk DB connect inputs. We've never configured specific timezone on the db connect app or on the inputs. Normally it just worked fine. We have a multiple DB views, from which we read and then index into Splunk. This particual View has these four columns: EVENT_ID NUMBER LOG_LEVEL NUMBER EVENT_TIME TIMESTAMP EVENT CLOB It is a follow tail input, rising column set to EVENT_ID. The timestamp ist extracted from the EVENT_TIME column. The events are not updated, hence it's safe to index them with the follow tail method. DB Input Setup The input's SQL Query looks like this: SELECT * FROM (SELECT * FROM "ONL"."DBG_EVENT_LOG" WHERE EVENT_TIME >= sysdate-1) t WHERE EVENT_ID > ? ORDER BY EVENT_ID ASC WHERE EVENT_TIME >= sysdate-1 is there to improve query search time performance as it is a really big DB view. The EVENT column contains several lines that belong together. Each line starts with another timestamp. When splunk indexes the events, it writes all columns prefixed by a timestamp, which I assume is the timestamp from EVENT_TIME. This gets us events in Splunk with following structure: <timestamp>, EVENT_ID="<EVENT_ID>", EVENT_TIME="<EVENT_TIME>", EVENT="<EVENT>" For some reason it omits LOG_LEVEL, but this is no problem at all. Splunk Query for Analysing the Issue I have following Splunk query to compare the timestamps to each other: |search <base search for raw events> |eval indextime=strftime(_indextime, "%c") |eval time=strftime(_time, "%c") |rex field=_raw "(?<firstdate>\S+\s\S+)," |table time, EVENT_TIME, firstdate, indextime |sort by indextime DESC The |rex command just rexes the timestamp which prefixes all indexed events into Splunk field firstdate . I called it <timestamp> in the previous event structure. Timestamps So we have these four timestamps: _time as time _indextime as indextime EVENT_TIME (incorporated into the splunk event) firstdate (incorporated into the splunk event) Now on to the issues... Issue with no Timezones Set at all I haven't set any timezone configuration at all. I let Splunk automatically decide, what timezone the servers run on and how the events should be converted. So I get following timestamps: time EVENT_TIME firstdate indextime Mon Apr 20 13:01:04 2020 2020-04-20 12:01:04.382576 2020-04-20 12:01:04.382 Mon Apr 20 12:01:30 2020 Mon Apr 20 13:01:04 2020 2020-04-20 12:01:04.369736 2020-04-20 12:01:04.369 Mon Apr 20 12:01:30 2020 Splunk indexes the event chronically at the wrong time, see difference of 1 hour between time and indextime. Issue with all Timezones Set to Europe/Zurich Now to try with explicitly set timezones, I configured the following: Splunk starts with environment variable TZ="Europe/Zurich" set (in the systemd .service manifesto) Added option -Duser.timezone="Europe/Zurich" via GUI in Splunk DB Connect App -> Configuration -> Settings -> General to both Task Server JVM Options and Query Server JVM Options. Set DB Input Timezone to Europe/Zurich : +02:00 via GUI in Splunk DB Connect App -> Configuration -> Databases -> Connections -> Selecting the input and setting Timezone When searching for events now, I get following differing timestamps: time EVENT_TIME firstdate indextime Mon Apr 20 12:17:41 2020 2020-04-20 12:17:41.679789 2020-04-20 11:17:41.679 Mon Apr 20 12:18:02 2020 Mon Apr 20 12:17:41 2020 2020-04-20 12:17:41.599967 2020-04-20 11:17:41.599 Mon Apr 20 12:18:02 2020 Either firstdate misses the daytime savings configuration and hence is only +01:00, or it actually has the daytime savings but is indexes as UTC, hence again only +01:00. Questions I'd be happy with either solution to one of these questions: How to remove that prefixed timestamp? How to disable timezone conversion for that timestamp? Some Specs Splunk 7.3.3 (build 7af3758d0d5e) DBX 3.1.4 and 3.3.0 (after upgrade) ojdbc7.jar 12.1.0.2.0 java8 jdk1.8.0_202 OS Linux RHEL7.6 Sorry for the long post, I just wanted to provide all possibly relevant information. ... View more

In the documentation about using summary indexes it says at step 8: Select a summary index. The default summary index is named summary. The list only displays indexes to which you have permission to write. [...] It seems the last bit "only displays indexes to which you have permission to write" is a bit off. Our users see the correct summary index on a random basis: Sometimes they see them, sometimes not. We could not determine what's causing this undeterministic behaviour. So how do I configure a role to have permissions to write to a dedicated index? (obviously without allowing them to write to all other indexes) Our specs: Index cluster on version 7.1.2 (build a0c72a66db66) search head cluster on version 7.1.2 (build a0c72a66db66) summary indexes are defined on index & search head cluster ... View more

Hi We have some standalone searchheads. We used to version-control each app on them with subversion, which worked perfectly. Now we will migrate to a searchhead cluster <i>(shcluster)</i>. Hence version-control with subversion does not work anymore, not without recurring conflicts, which are not user-friendly for our developers. Then I saw your app, and it might solve our problem, were it not for some missing features. So my question is, are you going to implement following points: Version comparison of views: <code>/servicesNS/-/-/data/ui/views</code> Version comparison of lookup files: <code>/servicesNS/-/-/data/lookup-table-files</code> Version comparison more granular, say in the <code>ct_change_investigation</code> dashboard not only to compare date, but also datetime <i>( date = strftime(_time, "%m/%d/%Y %H:%M:%S") )</i>. My intention would be, to manually trigger a populating search <i>(see http://answers.splunk.com/answers/49891/can-i-manually-trigger-a-scripted-alert.html)&lt;/i> whenever a user wants to save the current state, e.g. when working on views. That would be very, very nice. Kind regards, Muryoutaisuu ... View more

Hi guys We are testing the Search Head pooling functionality. We have one dedicated deployer and 5 searchhead clustermembers. To deploy we execute following command: splunk apply shcluster-bundle --answer-yes -target https://[MEMBER_HOSTNAME]:8089 -auth [SPLUNK_USER]:[PW] Sometimes it works good. Sometimes not and then it has different errors. Error Nr. 1: Error while deploying apps to target=https://[MEMBER_HOSTNAME]:8089 with members=5: ConfDeploymentException: Error while updating app=XXX on target=https://[MEMBER_IP]:8089: Non-200/201 status_pre=500; {"messages":[{"type":"ERROR","text":"\n In handler 'localapps': Error during app install: failed to extract app from /appl/splunk/var/run/splunk/bundle_tmp/2753df224a95e6e5.bundle to /appl/splunk/var/run/splunk/bundle_tmp/1074b24058b88cde: No such file or directory"}]} Error Nr. 2: Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://[MEMBER_IP]:8089: Network-layer error: Connection reset by peer Error Nr. 3: Error while deploying apps to target=https://[MEMBER_HOSTNAME]:8089 with members=5: ConfDeploymentException: Error while fetching apps baseline on target=https://[MEMBER_IP]:8089: Network-layer error: Connection reset by peer Error Nr. 4: Error when getting master uri from target to do a rolling-restart Error connecting: Connection refused What astonishes me, what I do not understand, is: why does it sometimes work, and sometimes not? Sometimes I have to execute the deploy command more than 5 times consecutively! It begins to annoy me. Does somebody experience the same? Or does somebody even have a solution, or an explanation? Thanks - Muryoutaisuu ... View more