Getting MongoDB data into Splunk on Date rising column

[zvaseqi]

Update on June 24, 2015

I've tried switching to _id column from my mongo db as my rising column; it again works the first round but doesn't incrementally get data on the given intervals. It's passed the 15 minute interval I specified and I'm not seeing any new data coming in or the state.xml under persistentstorage/dbx/... being updated either!

I can see the the state.xml for my data input is marking my rising column as "string" which is probably why Splunk is not able to incrementally get data after the first round. Likely Splunk expects "int", "Data" or similar data types to increment.

---

Update on June 23, 2015

I managed to get mongo db data into splunk by adding my strptime pattern to prps.conf and indicating a query like
SELECT TIME_COLUMN, COLUMN2, COLUMN3 FROM MY_TABLE {{ WHERE $rising_column$ > ?}}
rising column= TIME_COLUMN

along with checking "output timestamp" checkbox and setting it to CAST(TIME_COLUMN, 'TIMESTAMP') inside my data input form.

However, I'm currently having an issue where the data only pours in one round. Looks like it doesn't run tail data every 1h interval.
I think the main problem is that the rising column format is "string" so it fails on comparing rising column which is why it's not able to get the data more than just once.

Any ideas of what could be causing this?

---

I'm trying to get data from Mongo DB into splunk using "Tail" input type so that I can get data in an incremental fashion on a daily basis.
I have installed JDBC driver for MongoDB to be able to use a SQL query for pulling data.

I'm using the following query to get data in
SELECT CAST(TIME_COLUMN, 'TIMESTAMP') AS TIME_TEST, COLUMN2, COLUMN3 FROM MY_TABLE {{ WHERE $rising_column$ > ?}}

and rising_column is set to TIME_TEST

Note: TIME_COLUMN from MongoDB is of type ISODATE.

When using the above command in a new database input (using Splunk DB Connect App) I get the following error:

ERROR AdminManagerExternal - Received malformed XML from external handler:\nFailed to validate: com.splunk.config.SplunkConfigurationException: Error validating dbmonTail for monitor=dbmon-tail://MY_DataBase/MY_INPUT_NAME: java.lang.Throwable: **JDBC for MongoDB Driver: Functions in projections are not supported.** with query = MY_QUERY_FROM_ABOVE  \r\r\n<eai_error><recognized>false</recognized><type><class 'spp.java.bridge.JavaBridgeError'></type><message>Command com.splunk.dbx.monitor.DatabaseMonitorValidator returned status code 17</message><stacktrace>Traceback (most recent call last):\r\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\admin.py", line 70, in init\r\n    hand.execute(info)\r\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\admin.py", line 526, in execute\r\n    if self.requestedAction == ACTION_CREATE:   self.handleCreate(confInfo)\r\n  File "C:\Program Files\Splunk\etc\apps\dbx\bin\spp\config.py", line 230, in handleCreate\r\n    self.handleModification("create", output)\r\n  File "C:\Program Files\Splunk\etc\apps\dbx\bin\spp\config.py", line 218, in handleModification\r\n    id, props = self.process_modification(id, props, type=type, output=output)\r\n  File "C:\Program Files\Splunk\etc\apps\dbx\bin\rest_handler_dbmon.py", line 95, in process_modification\r\n    self.validateConfig(stanza, props)\r\n  File "C:\Program Files\Splunk\etc\apps\dbx\bin\rest_handler_dbmon.py", line 126, in validateConfig\r\n    executeBridgeCommand("com.splunk.dbx.monitor.DatabaseMonitorValidator", args, checkStatus=True)\r\n  File "C:\Program Files\Splunk\etc\apps\dbx\bin\spp\java\bridge.py", line 182, in executeBridgeCommand\r\n    raise JavaBridgeError("Command %s returned status code %s" % (cmd, ret))\r\nJavaBridgeError: Command com.splunk.dbx.monitor.DatabaseMonitorValidator returned status code 17\r\n</stacktrace></eai_error>\r\n

[richgalloway]

Have you tried setting rising_column to TIME_COLUMN?

[zvaseqi]

I had to actually set rising column to TIME_COLUMN and adjust "output timestamp" with CAST(TIME_COLUMN, 'TIMESTAMP')
that took me one step closer but still not importing data on given intervals!