topic Re: Why is the Splunk DB Connect not indexing data? in All Apps and Add-ons

Why is the Splunk DB Connect not indexing data?

apair — Mon, 28 May 2018 13:15:38 GMT

Hello,

I have a problem with Splunk Entreprise 6.5.2 et Splunk DB Connect 3.1.3 :
Splunk DB Connect don't index data from database.
In logs, I see :

2018-05-28 14:53:51.863 +0200  [QuartzScheduler_Worker-27] INFO  org.easybatch.core.job.BatchJob - Job 'testdbinput' finished with status: FAILED

2018-05-28 14:53:51.863 +0200  [QuartzScheduler_Worker-27] ERROR org.easybatch.core.job.BatchJob - Unable to write records
java.io.IOException: HTTP Error 400: Bad Request
    at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEventBatch(HttpEventCollector.java:112)
    at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEvents(HttpEventCollector.java:89)
    at com.splunk.dbx.server.dbinput.recordwriter.HecEventWriter.writeRecords(HecEventWriter.java:36
....

2018-05-28 14:53:51.863 +0200  [QuartzScheduler_Worker-27] ERROR c.s.d.s.task.listeners.RecordWriterMetricsListener - action=unable_to_write_batch
java.io.IOException: HTTP Error 400: Bad Request
    at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEventBatch(HttpEventCollector.java:112)
    at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEvents(HttpEventCollector.java:89)
    at com.splunk.dbx.server.dbinput.recordwriter.HecEventWriter.writeRecords(HecEventWriter.java:36)
...

2018-05-28 14:53:51.850 +0200  [QuartzScheduler_Worker-27] INFO  c.s.d.s.dbinput.recordwriter.HttpEventCollector - action=writing_events_via_http_event_collector record_count=5

When I configure my input, the request is OK :

I have disabled SSL, and I put a tcpdump in the server to see request :

{"time":"1527509442,533","event":"2018-05-28 14:10:42.533, action=\"SUPPRESSION_CONTRAT\"","host":"xxxxx","source":"testdbinput","sourcetype":"defautkv_xxxxx","index":"test"}

When I test to send this data with a curl :

curl -k https://127.0.0.1:8088/services/collector/event -H "Authorization: Splunk 761bdb35-0b8c-4780-xxxx-xxxxxx" -d '{"time":"1527509442,533","event":"2018-05-28 14:10:42.533, action=\"SUPPRESSION_CONTRAT\"","host":"xxxxx","source":"testdbinput","sourcetype":"xxxxx","index":"test"}'
{"text":"Error in handling indexed fields","code":15}

For me the field time isn't correct : 1527509442,533 ==> 1527509442.533

curl -k https://127.0.0.1:8088/services/collector/event -H "Authorization: Splunk 761bdb35-0b8c-4780-xxxx-xxxxxx" -d '{"time":"1527509442.533","event":"2018-05-28 14:10:42.533, action=\"SUPPRESSION_CONTRAT\"","host":"xxxxx","source":"testdbinput","sourcetype":"xxxxx","index":"test"}'
{"text":"Success","code":0}

Is it a bug in Splunk DB Connect ?

Thank you in advance,

Cordially

Re: Why is the Splunk DB Connect not indexing data?

apair — Tue, 29 May 2018 06:47:13 GMT

Sorry, I can't edit and I want to add this information :
=> I have tested with the version 2.4.1. It is OK, Data is indexed correctly...

So it is a bug in the 3.1.1 version ?

Re: Why is the Splunk DB Connect not indexing data?

PeterSkarmyr — Wed, 30 May 2018 12:16:45 GMT

I have the same problem. It would be much appriciated if you could update your post if you find a solution. Thanks

Re: Why is the Splunk DB Connect not indexing data?

PeterSkarmyr — Wed, 30 May 2018 13:44:02 GMT

I also have the issue with the metadata field "time" is not being formated correctly. It is using a comma instead of a dot. In the documentation, under metadata, it says it should be a dot with the default settings: https://docs.splunk.com/Documentation/Splunk/7.1.1/Data/FormateventsforHTTPEventCollector

Again, if you find a workaround it would be much appriciated if you let me know. Thanks.

Relevant event from my log where you see the event being created incorrectly with a badly formated time field:

2018-05-30 15:09:48.365 +0200 [QuartzScheduler_Worker-22] DEBUG c.s.d.s.dbinput.task.processors.EventMarshaller - action=finish_format_hec_events record=Record: {header=[number=2, source="blueprism", creationDate="2018-05-30 15:09:48.365"], payload=[{"time":"1527685788,365","event":"2018-05-30 15:09:48.365, resourceid=\"9EAD88A2-725A-4806-897F-8F1C8B1022AD\", name=\"NOLB2373_debug\", status=\"Ready\", processesrunning=\"0\", actionsrunning=\"0\", unitsallocated=\"0\", lastupdated=\"2018-05-09 14:12:21.64\", AttributeID=\"4\", diagnostics=\"0\", logtoeventlog=\"1\", FQDN=\"NOLB2373.mistral.mistralnett.com\", ssl=\"0\", userID=\"6D34DB81-1665-4324-89B4-21A0B878100B\"","host":"NOLB2373\\SQLEXPRESS","source":"blueprism","sourcetype":"blue_prism","index":"resources"}]}

Re: Why is the Splunk DB Connect not indexing data?

apair — Thu, 31 May 2018 08:45:30 GMT

At this time I have downloaded the version 2.4.1 and it's working properly but I would like to update to the latest version...

Re: Why is the Splunk DB Connect not indexing data?

qthalia — Wed, 06 Jun 2018 08:48:50 GMT

3.1.1 version works properly as well. But I had to completely remove the app in console first. After upgrade I see each time that task server cannot be run on port 9998 or any other free port.

Re: Why is the Splunk DB Connect not indexing data?

PeterSkarmyr — Wed, 06 Jun 2018 12:47:23 GMT

How did you get version 3.1.1? I can only download version 2.4.1 or 3.1.3 on splunkbase.
Thanks.

Re: Why is the Splunk DB Connect not indexing data?

qthalia — Sat, 09 Jun 2018 05:18:14 GMT

Let me know your email I'll send you a link to the file stored in my Google drive.

Re: Why is the Splunk DB Connect not indexing data?

kamil_rostecki — Thu, 28 Jun 2018 07:58:41 GMT

You have to change your locale environment variables:
LANG=C
LC_ALL=C

Re: Why is the Splunk DB Connect not indexing data?

jmzuccolini — Tue, 29 Sep 2020 21:02:44 GMT

I had the same issue, and your suggestion worked for me. My splunk user was using "fr_FR.UTF-8",
I changed with LANG=en_US.UTF-8 and LC_ALL=en_US.UTF-8
Thank you for your help

Re: Why is the Splunk DB Connect not indexing data?

astrid_h — Fri, 28 Dec 2018 12:35:31 GMT

Your suggestion worked for me too.

Re: Why is the Splunk DB Connect not indexing data?

Anonymous — Fri, 11 Jan 2019 12:56:40 GMT

Hi
Can anyone show an example of how to change the locale environment variables:
LANG=C
LC_ALL=C

Thanks for all reply

Re: Why is the Splunk DB Connect not indexing data?

Anonymous — Fri, 11 Jan 2019 12:57:44 GMT

Hi, is it possible to get a copy of the older version please?

Re: Why is the Splunk DB Connect not indexing data?

tecooper — Fri, 11 Jan 2019 14:23:35 GMT

In Linux, type locale at the prompt. I'm not sure how to do it in Windows.

Re: Why is the Splunk DB Connect not indexing data?

Anonymous — Fri, 11 Jan 2019 18:48:45 GMT

Thank you, I wil try to change this in windows for the user running splunk.