Getting Data In

Getting MongoDB data into Splunk on Date rising column


Update on June 24, 2015

I've tried switching to _id column from my mongo db as my rising column; it again works the first round but doesn't incrementally get data on the given intervals. It's passed the 15 minute interval I specified and I'm not seeing any new data coming in or the state.xml under persistentstorage/dbx/... being updated either!

I can see the the state.xml for my data input is marking my rising column as "string" which is probably why Splunk is not able to incrementally get data after the first round. Likely Splunk expects "int", "Data" or similar data types to increment.

Update on June 23, 2015

I managed to get mongo db data into splunk by adding my strptime pattern to prps.conf and indicating a query like
rising column= TIME_COLUMN

along with checking "output timestamp" checkbox and setting it to CAST(TIME_COLUMN, 'TIMESTAMP') inside my data input form.

However, I'm currently having an issue where the data only pours in one round. Looks like it doesn't run tail data every 1h interval.
I think the main problem is that the rising column format is "string" so it fails on comparing rising column which is why it's not able to get the data more than just once.

Any ideas of what could be causing this?

I'm trying to get data from Mongo DB into splunk using "Tail" input type so that I can get data in an incremental fashion on a daily basis.
I have installed JDBC driver for MongoDB to be able to use a SQL query for pulling data.

I'm using the following query to get data in

and rising_column is set to TIME_TEST

Note: TIME_COLUMN from MongoDB is of type ISODATE.

When using the above command in a new database input (using Splunk DB Connect App) I get the following error:

ERROR AdminManagerExternal - Received malformed XML from external handler:\nFailed to validate: com.splunk.config.SplunkConfigurationException: Error validating dbmonTail for monitor=dbmon-tail://MY_DataBase/MY_INPUT_NAME: java.lang.Throwable: **JDBC for MongoDB Driver: Functions in projections are not supported.** with query = MY_QUERY_FROM_ABOVE  \r\r\n<eai_error><recognized>false</recognized><type><class ''></type><message>Command com.splunk.dbx.monitor.DatabaseMonitorValidator returned status code 17</message><stacktrace>Traceback (most recent call last):\r\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\", line 70, in init\r\n    hand.execute(info)\r\n  File "C:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\", line 526, in execute\r\n    if self.requestedAction == ACTION_CREATE:   self.handleCreate(confInfo)\r\n  File "C:\Program Files\Splunk\etc\apps\dbx\bin\spp\", line 230, in handleCreate\r\n    self.handleModification("create", output)\r\n  File "C:\Program Files\Splunk\etc\apps\dbx\bin\spp\", line 218, in handleModification\r\n    id, props = self.process_modification(id, props, type=type, output=output)\r\n  File "C:\Program Files\Splunk\etc\apps\dbx\bin\", line 95, in process_modification\r\n    self.validateConfig(stanza, props)\r\n  File "C:\Program Files\Splunk\etc\apps\dbx\bin\", line 126, in validateConfig\r\n    executeBridgeCommand("com.splunk.dbx.monitor.DatabaseMonitorValidator", args, checkStatus=True)\r\n  File "C:\Program Files\Splunk\etc\apps\dbx\bin\spp\java\", line 182, in executeBridgeCommand\r\n    raise JavaBridgeError("Command %s returned status code %s" % (cmd, ret))\r\nJavaBridgeError: Command com.splunk.dbx.monitor.DatabaseMonitorValidator returned status code 17\r\n</stacktrace></eai_error>\r\n


Have you tried setting rising_column to TIME_COLUMN?

If this reply helps you, Karma would be appreciated.


I had to actually set rising column to TIME_COLUMN and adjust "output timestamp" with CAST(TIME_COLUMN, 'TIMESTAMP')
that took me one step closer but still not importing data on given intervals!

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...