According to the documentation you should package dependencies your app needs in its  /<appname>/bin  directory. ... View more

That is normal. We store the rising column checkpoint value in inputs.conf. ... View more

After many hours of debugging and going back and forth with Splunk Support here is what I did. First Question: Controlling data/time format of the data entering into a database as part of DB Output Operation? Short answer is this cannot be controlled from Splunk unless you follow suggestions from other users to this question and modify inputs.conf directly and not go thru the DB Output UI to update/modify your output. If you do not have access to input.conf then you need to change your column in Oracle to be of type "Timestamp with TImeZone" and update the splunk query to output the date time in the exact format Oracle wants which would depend on your installation. To find out the default Oracle format run the following SQL in Oracle: select * from nls_database_parameters where parameter='NLS_TIMESTAMP_TZ_FORMAT' The SQL above gave me the default date time format for my version or Oracle as "DD-MON-RR HH.MI.SSXFF AM TZR" - I ten used the Splunk function strftime(_time,"%d-%b-%y %I.%M.%S.%3N %p %z") to output the date time in the exact format Oracle is expecting. Second Question: You are tasked with exporting specific data out of Splunk to a database on regular basis and you have to make sure you do not export duplicates or miss exporting logs because Splunk was down for maintenance or the DB Output job did not run at the scheduled time for any reason. How can you ensure the next time DB Output operation runs it will output all records since the last export to the database? I looked around and the only solution I found that I thought would be reliable is to query the database and find out what was the date/time stamp of the last exported record and export all records with a date time stamp after that to the database. After some research I came up with the following Splunk Query to do this: index=main sourcetype=radius [| dbxquery connection="bidb" maxrows=1 shortnames=true query="SELECT to_char(MAX(START_TIME), 'mm/dd/yyyy:HH24:MI:SS') AS EARLIESTTIME FROM SUBSCRIPTIONS" | eval earliest=if(EARLIESTTIME=="NULL","-3mon@mon",EARLIESTTIME) | eval latest="now" | fields earliest, latest | format "(" "(" "" ")" "OR" ")"] Referring to the above Splunk query - the inner query run once and uses dbxquery to find the date time of the last records exported to database. Then it uses that to set the "earliest" value in the Splunk query along with "latest" set to "now". This way the search can be scheduled to run at whatever intervals you like since we no longer rely on specific duration or do we rely on the output job to run at an exact time every time. This approach worked great in Splunk UI while I went thru the DB Output Wizard and setup the operation. But once scheduled it failed every time with an HTTP 400 error in the logs. After spending hours trying to debug and going back and forth with Splunk support it turns out you cannot have a dbxquery in your DB Output operation even thou it works when you are setting up the operation. During setup the query is run under the DB Connect app and it succeeds but scheduled jobs which are run thru the python script are not run under the DB Connect app context and are run under the default search and reporting app context which does not know what dbxquery is and as such it fails. My conclusion is that DB Connect V2 Output Operation is NOT currently a reliable method of exporting data out of Splunk on regular basis. We decided to just write our own Java app that uses the Splunk Rest API to pull data out of Splunk and store in the database. ... View more