I'm trying to tail a database using a rising column labelled "ID". Splunk DB Connect is working in the sense that it can search the database and return results but it appears to be updating the checkpoint sporadically and out of order which is causing duplicate events to be indexed. The database is MS SQL. inputs.conf: [mi_input://my_querytable] connection = My_Database enable_query_wrapping = 1 index = my_index input_timestamp_column_fullname = (003) my_querytable.Time.datetime input_timestamp_column_name = Insert_Time interval = 120 max_rows = 100000 mode = advanced output_timestamp_format = yyyy-MM-dd HH:mm:ss query = SELECT ID,Time,Field2,Field2,Field3,Field4,Field5,Field56,Field7 FROM "my_table"."dbo"."my_querytable" WHERE ID > ? sourcetype = my_sourcetype tail_rising_column_checkpoint_value = 1675918831 tail_rising_column_fullname = (001) my_querytable.ID.bigint tail_rising_column_name = ID ui_query_catalog = my_catalog ui_query_mode = advanced ui_query_schema = dbo ui_query_table = my_querytable disabled = 0 fetch_size = 20000 Here are what the logs look like when the query runs every 2 minutes and returns 100000 records via 20000 row pulls. 2017-02-26T16:42:43+0000 [INFO] [mi_input.py], line 193: action=start_executing_dbinput dbinput="mi_input://my_querytable" 2017-02-26T16:43:01+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=20000 2017-02-26T16:43:01+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1401390595 2017-02-26T16:43:16+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=20000 2017-02-26T16:43:17+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1413595646 2017-02-26T16:43:19+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=20000 2017-02-26T16:43:20+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1399518391 2017-02-26T16:43:22+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=20000 2017-02-26T16:43:23+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1399707472 2017-02-26T16:43:25+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=20000 2017-02-26T16:43:26+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1524449693 2017-02-26T16:43:26+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=0 2017-02-26T16:43:26+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1524449693 2017-02-26T16:43:26+0000 [INFO] [mi_input.py], line 211: action=complete_dbinput dbinput="mi_input://my_querytable" 2017-02-26T16:44:43+0000 [INFO] [mi_input.py], line 193: action=start_executing_dbinput dbinput="mi_input://my_querytable" 2017-02-26T16:45:30+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=20000 2017-02-26T16:45:31+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1533335128 2017-02-26T16:45:33+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=20000 2017-02-26T16:45:34+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1533500333 2017-02-26T16:45:36+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=20000 2017-02-26T16:45:37+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1528798868 2017-02-26T16:45:39+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=20000 2017-02-26T16:45:40+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1680663600 2017-02-26T16:45:42+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=20000 2017-02-26T16:45:43+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1675416914 2017-02-26T16:45:43+0000 [INFO] [modular_input_event_writer.py], line 93 : action=print_csv_from_jdbc_to_event_stream dbinput="mi_input://my_querytable" input_mode=advanced events=0 2017-02-26T16:45:43+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1675416914 2017-02-26T16:45:43+0000 [INFO] [mi_input.py], line 211: action=complete_dbinput dbinput="mi_input://my_querytable" Here is the chronological order of just the checkpoint being updated. Notice how it jumps back and forth in the rising column: 2017-02-26T16:43:01+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1401390595 2017-02-26T16:43:17+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1413595646 2017-02-26T16:43:20+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1399518391 2017-02-26T16:43:23+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1399707472 2017-02-26T16:43:26+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1524449693 2017-02-26T16:43:26+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1524449693 2017-02-26T16:45:31+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1533335128 2017-02-26T16:45:34+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1533500333 2017-02-26T16:45:37+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1528798868 2017-02-26T16:45:40+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1680663600 2017-02-26T16:45:43+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1675416914 2017-02-26T16:45:43+0000 [INFO] [mi_input.py], line 109: action=rising_column_checkpoint_updated dbinput="mi_input://my_querytable" checkpoint=1675416914 I've tried changing the query to something like this but the query constantly times out even when only trying to return 100 rows. SELECT ID,Time,Field2,Field2,Field3,Field4,Field5,Field56,Field7 FROM "my_table"."dbo"."my_querytable" WHERE ID > ? ORDER BY ID I also tried changing the max row pull to 1000/1000 every two minutes so it was forced to only grab 1000 rows every two minutes and update the checkpoint value only once every two minutes. The checkpoint value still kept bouncing around and was not in a linear progression. Anyone have any idea why I'm seeing this kind of behavior and if there is anything I can do about it? ... View more

I'm using Splunk DB Connect to pull data from an MS SQL database that is sitting on a server in the US Eastern time zone. The Splunk server with Splunk DB Connect is configured in UTC time. The time stamp column I'm using to extract the time stamp from the event is in Eastern time. All of the data I forward to Splunk is interpreted into UTC time but I'm having issues getting this Eastern time stamp from the database to be correctly indexed with a UTC time stamp for _time. We also use an intermediary heavy forwarder to receive the events from the Splunk DB Connect server before the events are forwarded to the indexers. It looks something like this: MS SQL Database Server (EST) <--- Splunk DB Connect Server (UTC) ---> Intermediary Heavy Forwarder (UTC) ---> Indexer pool (UTC) I've tried adding the following props.conf stanzas to both the Splunk DB Connect server and the Heavy Forwarder server but the events are still being indexes with an Eastern timezone time stamp. [source::mi_input://database_input1] TZ = US/Eastern [source::mi_input://database_input2] TZ = US/Eastern We are using the output time stamp format of epoch with the following inputs.conf stanza. output_timestamp_format = epoch Could this be causing Splunk to automatically assume the epoch time is already in UTC? Perhaps I'm not fully understanding the the function of the TZ stanza. How can I get Splunk to index the event from the database with a converted timestamp from EST to UTC? Using Splunk 6.5.1 and Splunk DB Connect 2.4.0 ... View more

I'm trying to create a dashboard form with a text input box that populates a token that is used in a number of searches on the dashboard. I recently created a new panel on the dashboard and could not get it to work using the tokens from the text input. After some trial and error I found that the search doesn't want to start when the subsearch attempts to return two aliased fields. Here is what my search looks like: index=myindex [|inputlookup MyLookup.csv.gz | search Field=$token1$ | return customField1=$FieldA customField2=$FieldA] If I remove one of the aliased fields from the return command in the subsearch the search starts without issue and finishes. If I keep both aliased fields in the rerturn command as seen above the search never starts after inputting data in the text box. The rest of the panels on the dashboard will load but this panel will just sit there stating "Search is waiting for input...". If I run this search with a hardcoded value where the token is in a regular search window I have no problem running the search and it returns the results I want. Can you not return two aliased fields in a subsearch when it is part of a dashboard form or is this a bug? Has anyone else had this issue? I'm currently running 6.5.1 on the Search Head. ... View more

We recently upgraded to Splunk 6.5.1 and noticed a fairly large increase in our replicated knowledge bundle size from the Search head to our search peers. After doing some digging it appears that the splunk_archiver app has grown significantly in size since the last version we were on, 6.3.5. In Splunk 6.3.5 the splunk_archiver app was a total of 108K in the knowledge bundle tar. In Splunk 6.5.1 the splunk_archiver app is a total of 73M in size; mostly due to some large jar files. Since the splunk_archiver app is a native Splunk component and already lives on the Splunk Indexers, does the search head really need to be distributing the entire splunk_archiver app to the search peers? This is an additional 73M of data that doesn't seem to be necessary to distribute. Would it be safe to assume that we can blacklist the splunk_archiver app from distribution in the knowledge bundle without breaking anything? ... View more

While digging through my Search head logs, I stumbled upon some WARN messages from the DistributedBundleReplicationManager component regarding "Asynchronous bundle replication" "took too long (longer than 10 seconds)". The knowledge bundle the Search head is currently replicating is about 250mb in size. Doing a historical look back, I found that this error message was occurring approximately 50 times an hour for the last month or so, each message reporting that the configuration bundle was average 250mb in size. The majority of the files in the bundle that are decently large include lookup files, however, the majority of these lookup files remain static and do not change frequently or ever. It was my understanding that the Search head and search peers kept file hash records of the knowledge bundle components and only replicated the delta of that bundle. Looking at these error messages it appears that the Search head is replicating the 250mb knowledge bundle multiple times per hour. Is this the expected behavior? I know that all of /etc/apps for the most part is being replicated as well but nothing to my knowledge is changing on a regular enough basis that would require the Search head to send the entire replication bundle 50 times an hour. What is the threshold on blacklisting large lookup files? Do the search peers need the lookup files? I thought the lookup files were used at search time on the Search head. ... View more