topic Re: DB Connect: How to index quoted string values from a database with a dbmon tail? in Deployment Architecture

DB Connect: How to index quoted string values from a database with a dbmon tail?

falkyre — Fri, 31 Oct 2014 19:39:36 GMT

I have some data in a database that contains quoted values like the following:

REPT ALM ONT "ONT-1-1-16-3-11:MJ,LEVELLO,SA,7-26,11-58-9: \"1490 Optical signal level too low\""

When I run a dbquery, the value comes out with no issues. However, when using a tail on the database, the data gets indexed as a key value pair and drops everything after first " and shows up in Splunk as REPT ALM ONT \/ (should be a forward slash as it looks like it's trying to escape the first double quote).

Is there a setting that I can use to get all of the data indexed as the line shows up in the database so it shows up in Splunk? Can I set KV_MODE=auto in the props.conf or transforms.conf for dbx?

Thanks,

Re: DB Connect: How to index quoted string values from a database with a dbmon tail?

pmdba — Sun, 02 Nov 2014 03:56:40 GMT

I would use a view to select your data instead of selecting it directly from the table. The view should use a "replace" function (or the equivalent for your particular database) to substitute some other character for the quotes, or to insert an escape that Splunk will recognize.

Re: DB Connect: How to index quoted string values from a database with a dbmon tail?

falkyre — Mon, 03 Nov 2014 02:42:02 GMT

Unfortunately your suggestion won't work as I am using the tail functionality to index the database tables on the fly. If I do a dbquery, I get the values correctly but when I'm using the tail function, I am indexing the data with key-value pairs and that's when it "breaks". I want to know how to get the tail to index the field with all of the quotes in it. The actual key value pair is as follows:

SUMMARY=REPT ALM ONT "ONT-1-1-16-3-11:MJ,LEVELLO,SA,7-26,11-58-9: "1490 Optical signal level too low""

Re: DB Connect: How to index quoted string values from a database with a dbmon tail?

pmdba — Tue, 04 Nov 2014 15:32:41 GMT

A view works just as well as a table for dbtail. My point was that with a view you could insert whatever escape character(s) Splunk needs into your data and possibly preserve the quotes that way. Another possibility is to define a new key/term within Splunk using the "extract fields" functionality.

Re: DB Connect: How to index quoted string values from a database with a dbmon tail?

falkyre — Tue, 04 Nov 2014 19:26:36 GMT

Did some more digging and the data is being indexed properly in Splunk (ie I can see it in the Events view) but what's happening is it's getting escaped out when the default table view is done. I followed your suggestion and created a new extraction based on the data that is showing up (using the extract fields functionality which is much nicer in Splunk 6.2). Doing that got me to where I needed to be.

Thanks.