I would start with Select columns from table where status=complete and date>$risingcol That requires the status incomplete to status complete transition to be one-way and permanent, but it would allow you to use time for rising relatively safely (given the caveats in the documentation). If that’s not true, you’ll need to batch the whole table in regularly and use a lookup to manage a view from a couple of Splunk saved searches. In addition to high complexity, this could be a non-starter in terms of license impact if the table is large and collection is frequent. The lookup thing could also hurt performance if you have a lot of volume, as you’re effectively storing the desired subset of the original table in Splunk’s memory or as a KV store table in Splunk. ... View more

yes, it's based on the input name. It'll always be the same file, the rotation is just to drop old lines and add new ones. It doesn't gzip off old copies. ... View more

I’m assuming you’re talking about role based access control to the dB connect app. You can’t use SAML auth for the database service account. The scenario you describe should work, since it doesn’t you will need to contact support and probably file a bug. ... View more

1) I doubt the Java thing is actually what people think it is. Certainly wasn't last time. https://blog.joda.org/2018/08/java-is-still-available-at-zero-cost.html and That said, Splunk does need to invest in modernizing their Java support. 2) Not using Java to talk to RDBMS's is technically feasible but often unpleasant, because it's not very standard or performant. For instance you can use ODBC against MS, but it stinks against anything else. Lots of databases have native clients that will be great for one offs, but it won't be the standardized experience of DB Connect. ... View more

http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/Createandmanagedatabaseinputs#Choose_input_type Note that timestamps are not ideal for rising columns, though they often are the best available choice. Using a timestamp for rising column can produce the following problem conditions: A high rate of event generation can cause data duplication or loss, because checkpointing in a stream of timestamp-distinguished records assumes there is never more than one row created in a given time. If you set the time to a one second level of resolution and get five records per second, you lose or duplicate four records from every run. Clock skew, NTP skew corrections, physical moves between timezones, and daylight savings events can cause data mis-ordering, duplication, or loss. If the skew is towards the future, then the resulting checkpoint value may temporarily or permanently stop data collection. Non-numeric datetime values cannot be evaluated numerically, and lexical sorting can produce unpredictable results. If time series data is ordered lexically, then the resulting checkpoint value may temporarily or permanently stop data collection. ... View more

Short version: upgrade to DBX version 3. It no longer acts that way. ... View more

I’m not sure if it will be possible, it’s either easy or not possible. DBX sends JSON to a local HEC. There’s two ways to get indexed extractions from HEC. If using the correct endpoint and sourcetype, it could just mean configuration flags. Or it needs manipulating the source data, in which case you’re out of luck. http://dev.splunk.com/view/event-collector/SP-CAAAFB6 I don’t recall if DBX is using the raw endpoint; if so, this won’t work. If it is using the events endpoint and if it’s forming the JSON as expected, then you might be able to just turn on indexed extractions in props.conf for the sourcetype. Good luck! ... View more

HEC needs its own port ... View more

That makes this your next step then http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/TroubleshootHTTPEventCollector ... View more

I think you've properly diagnosed that it's unable to write into HEC -- can you write any input from db connect? The setup should have created an HEC input, is it there and enabled in Splunk inputs? ... View more

@vishaltaneja07011993 is correct. If you can’t go that far, you can modify your SQL query to CAST or CONVERT the column to a type that Splunk can handle. ... View more

It can’t read the configuration. Check file permissions, app permissions, and configuration file errors. ... View more

Hi, if you scroll down to Rising Input here, there’s some discussion of the problem: http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/Createandmanagedatabaseinputs You’re most likely seeing records come in at higher resolution than DBX can handle. In other words, if you get 5 records per second and the checkpoint is at a 1 second interval, then you’ll lose 4 records from each execution. ... View more

no problem! ... View more

I suspect you're running into trouble because it's using current index time to set _time instead of your column, even though you're doing something to name a column _time which isn't the same thing. Can you check the setting referenced in the doc? ... View more

I think you need to do an input instead to control time, http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/Createandmanagedatabaseinputs Have you tried naming the time column to _time in your SQL? ... View more

there's always a maximum somewhere... 3 million is large enough to produce bundle replication challenges for instance. DBX doesn't have a built-in limit that I'm aware of, but you could certainly produce a query that's bigger than the database wants to return. ... View more

Hi, you should read the first part of this page to make sure you're selecting the right design: http://docs.splunk.com/Documentation/DBX/3.1.3/DeployDBX/Createandmanagedatabaselookups With such a large lookup, you're going to end up with an index anyway, either an automatic summary or a KV store... which effectively means "copy from one database into another database". ... View more

Hi, free license won't work with DB Connect, because it needs RBAC and that isn't available. If you're a licensed customer, you can have it contact your license master or you can contact support to get a forwarder license. I'm not sure that message is related, but I am pretty sure that setup can't complete without an Enterprise license because it needs to do permissions-related activities. ... View more