topic Splunk DB Connect: If two Splunk servers create a database input from TableA, will this cause duplicates to be indexed? in Deployment Architecture

Splunk DB Connect: If two Splunk servers create a database input from TableA, will this cause duplicates to be indexed?

oraclebox — Mon, 09 Mar 2015 07:28:36 GMT

If two Splunk server create a Database input from table A, will this cause any conflict? Will both Server1 and Server2 import to index if a new record is inserted into table A?

Re: Splunk DB Connect: If two Splunk servers create a database input from TableA, will this cause duplicates to be indexed?

jcoates_splunk — Mon, 09 Mar 2015 16:53:50 GMT

you'll get a duplicate copy of your table.
DB1.TableA -> SplunkX.Input1A -> Index1
DB1.TableA -> SplunkY.Input1A -> Index1

Index1 now contains:
DB1.TableA.Row1
DB1.TableA.Row1
DB1.TableA.Row2
DB1.TableA.Row2
et cetera et cetera

Re: Splunk DB Connect: If two Splunk servers create a database input from TableA, will this cause duplicates to be indexed?

Richfez — Mon, 09 Mar 2015 17:01:38 GMT

Yes, both Splunk inputs will grab the same [full] set of data, though perhaps at different times. Inputs like these should be independent. See below for more explanation.

Assume you have a DB input on Splunk Server 1 that gets its input from, say, DBServerA, TableX. Then another DB input on Splunk Server 2 that gets its input from DBServerA, TableX as well.

Each Splunk server runs a SQL Query like select * from TableX {{ where $rising_column$ > ?}}

The first time through, the {{ ... }} is left off so the input can grab all the data. After that, each Splunk server will remember the last value for whatever has been declared as the $rising_column$ and use that for subsequent queries. Therefore, each server will get a copy of all rows that are newer than the last row that server had last requested, regardless of what other queries may have happened in the interim (applications, other Splunk servers, etc...)