Problem with Tailing events from Splunk DB connect

[alenseb]

Hi guys,

I have indexed a table from a DB using Splunk DB Connect.

It's got 2 Million records, i have given a column(ID) as the Rising Column.

But when i look at the data which has come in, sometimes 2 events gets accumulated together into one, sometimes 3 events gets together.

Is this a specific Splunk DB Connect or am i missing something?

Please help.
Thanks!

[Dan · Accepted solution]

Hi alenseb,

The problem is that Splunk has certain heuristics for linebreaking. Normally, log file data has timestamps for each event. Splunk understands that well. If you have timestamps in your database rows, then you shouldn't have line breaking issues. Just be sure to set output timestamp and specify as timestamp column the column that, you know, has the timestamp.

If you don't have timestamps in your database rows, you have two options:

1. Click output timestamp and leave the timestamp column blank. Splunk will output the current time when indexing.

2. Use the default sourcetype in the input config. Just leave it blank and Splunk DB Connect will use dbmon:kv as the sourcetype (in the normal case where you're using the key-value output format). However, if you put something custom in the sourcetype field, you should then tell Splunk how to linebreak for that sourcetype. You can just copy over the props.conf settings for the default stanzas - specifically, add "SHOULD_LINEMERGE = false".

Hope this helps. If you can tell me which of these paths you went down to cause/solve the problem, it will help me set the product roadmap around this feature.

Cheers,

Dan Goldburt

dan@splunk.com

Product Manager, DB Connect

[srevelle16]

Dan - Using a custom sourcetype in props.conf with "SHOULD_LINEMERGE = False" worked for me.

[alenseb]

Thanks a lot Dan! 🙂

Since i wasn't using a default sourcetype, i went for the Timestamp solution where i specified a timestamp column & the required format.

Looking forward for a better version soon!
Good luck!