DB connect data missing from splunk

[rakesh44]

Hi Friends,

I have one query in DB connect app, which runs once in month and pull 24 months of data ( 2017 and 2018) from oracle database

I have two columns in events segment and type which contains its value, when searched for Period 2017 for both columns , specific value is missing from splunk. When i checked in oracle logs both columns and its value are present in oracle logs also if searched for Period Apr 2018 value is present for both columns. When i checked in internal logs there is no error for logs. Is this happening because glitch in query.

Index=test Period =Apr-2017 segment=1265 type=3746

Note: I have ran above query mutiple times due to retention period issue. In such case can we enable data integrity check in splunk, which will check if all data is Indexed or not.

[tiagofbmm]

First, trying to pull 28 million events from a database is not a very promissing scenario. I would limit maximum number or rows to retrieve, increase your fetch size and put a couple of minutes of frequency of poll and enable the checkpoint on a strictly increasing column.

Second, data integrity check will make splunk to SHA-256 slices of data, so I don't think you could know make a comparison of checksums of what is in your database to what reaches Splunk. This is used to ensure data integrity was no later violated by any actor after the data has been indexed.

[rakesh44]

When i checked in _internal log found error message

Status=FAILED read_count=28708000 write_count=28707000, does this mean it is related to fetch size ?

I have setup fetch size = 300

[rakesh44]

or it is related to timezone, i did not select any timezone while creating connection