Archive

Splunk DB Connect 1: Why are large events > 10000 bytes being truncated to only 10K?

New Member

Hi,

I have a DBX input as follow:

[dbmon-tail://HPNA-DB/HPNA-Configs]
host = HPNA-DB
index = hpnaconfigs
output.format = mkv
output.timestamp = 1
output.timestamp.column = LastSnapshotSuccessDate
output.timestamp.format = yyyy-MM-dd HH:mm:ss.SSS
query = with Configs as (\r\n select p.PrimaryIPAddress DeviceIP\r\n ,p.hostname DeviceName\r\n ,p.LastSnapshotSuccessDate\r\n ,ConfigTextId = (select top 1 dd.DeviceDataId from RN_Device d inner join rn_device_data dd ON   dd.DeviceID = d.DeviceID\r\n and d.DeviceID = p.deviceid \r\n and dd.BlockType = 'configuration' \r\n and dd.blockformat = 1\r\n    order by dd.LastModifiedDate desc\r\n                         )\r\n from RN_DEVICE p\r\n)\r\nselect   LastSnapshotSuccessDate\r\n ,DeviceName\r\n ,DeviceIP\r\n
,convert(varchar(50), dd.LastModifiedDate, 21) as LastModifiedDate\r\n  
,substring(DataBlock,1,100) as ConfigTextStart\r\n
,substring(DataBlock,datalength(DataBlock)-100,100) as ConfigTextEnd\r\n
,datalength(DataBlock) as ConfigTextLen1\r\n ,DataBlock as ConfigText\r\n
,datalength(DataBlock) as ConfigTextLen2\r\nfrom Configs c inner join rn_device_data dd on dd.DeviceDataId = c.ConfigTextId\r\n{{WHERE $rising_column$ > ?}}
sourcetype = dbmon:mkv
tail.rising.column = LastSnapshotSuccessDate
disabled = 0
interval = auto
table = HPNA-Configs

and the following props.conf stanzas in system/local, apps/dbx/local and apps/search/local:

[dbmon:mkv]
LINE_BREAKER_LOOKBEHIND = 100000
TRUNCATE = 0
MAX_EVENTS = 100000

However, when searching, events are being truncated after 10K.

Any idea?

0 Karma

Splunk Employee
Splunk Employee

I ran into a similar issue, was as if Splunk failed to honor the settings in props.conf. I ran across an answer (sorry can't find it now) that suggested using the tpl_*.dbmonevt source. It's solved my issue, could you try adding the following to your props.conf?

[source::...tpl_*.dbmonevt]
LINE_BREAKER_LOOKBEHIND = 100000
TRUNCATE = 0
MAX_EVENTS = 100000

0 Karma

New Member

Thanks for the suggestion, however it had no affect, the events are still capped at 10K exactly.

Note: the last column "ConfigTextLen2" in the query is never visible...

0 Karma

New Member

this is what the event is tag with as well:

host = HPNA-DB source = dbmon-tail://HPNA-DB/HPNA-Configs sourcetype = dbmon:mkv

0 Karma

Splunk Employee
Splunk Employee

A suggestion was made that if you are using the JDBC drivers that ship with DB Connect and this is MS SQL Server, to swap them out and use Drivers that are shipped directly from Microsoft.

0 Karma

New Member

Splunk indexer is running on Linux, i don't believe MS made an SQL driver for this OS.

0 Karma

Splunk Employee
Splunk Employee

yeah, they do make a Linux version -- you can get it here. http://www.microsoft.com/en-us/download/details.aspx?id=11774

I'm not positive that it's relevant to your problem, but we've found that it has fewer weirdnesses.

0 Karma

New Member

wow, that is a surprise. Maybe i should try it.

0 Karma

Splunk Employee
Splunk Employee

what type of database and which driver are you using?

0 Karma

New Member

I am using MS SQL Server and the Java driver that comes with Splunk

0 Karma