All Apps and Add-ons

Why are DBconnect 3 inputs unable to write records and giving an http 400: bad request error?

bkoehler4070
Explorer

I have DBconnect 3.1.3 running on a 7.0.1 instance with 3 DB Inputs, 2 of them work perfectly but the third one puts out the error below and then fails. Two of the connections are almost identical except they go to different DBs, but one works and one doesn't. I have tried restarting and reconfiguring the input.

2018-04-12 18:48:43.627 +0000 [QuartzScheduler_Worker-29] ERROR c.s.d.s.task.listeners.RecordWriterMetricsListener - action=unable_to_write_batch
java.io.IOException: HTTP Error 400: Bad Request
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEventBatch(HttpEventCollector.java:112)
at com.splunk.dbx.server.dbinput.recordwriter.HttpEventCollector.uploadEvents(HttpEventCollector.java:89)
at com.splunk.dbx.server.dbinput.recordwriter.HecEventWriter.writeRecords(HecEventWriter.java:36)
at org.easybatch.core.job.BatchJob.writeBatch(BatchJob.java:203)
at org.easybatch.core.job.BatchJob.call(BatchJob.java:79)
at org.easybatch.extensions.quartz.Job.execute(Job.java:59)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

1 Solution

bkoehler4070
Explorer

Found the problem, the HEC fails to write when a single event has 800000+ characters. Had 1 log message that was ~800x the size of a normal log message and that was causing the failure, when that message is skipped (by changing the rising column) everything goes back to working as intended.

View solution in original post

henriklund
Engager

I have this issue with my DB Connection on the customer environment. I tried the following:
- Add hardcoded metadata
- Downgrade "db connect" to earlier version
- Install splunk on other server
- The total size of all columns in the table is not bigger than 800000+ characters

Have anyone had this issue and solved it in another way?

sylim_splunk
Splunk Employee
Splunk Employee

alt text

0 Karma

sylim_splunk
Splunk Employee
Splunk Employee

Many cases turned out to be caused by lack of meta data. For example, in DB Connect 3 the data fetched from DB are forwarded via HEC configured in local.
It appears to get 404 error when the payload lacks meta data, like source, sourcetype or host.
You can get the payload and confirm it by enabling DEBUG for the inputs or connections.

alt text

Then you can find the payload being forwarded via HEC - check the fields for default metadata such as host, source and sourcetype, below is just for example purpose.

If any of default fields are not populated you need to add them through the configuration for metadata.

jfeitosa_real
Path Finder

Hi @andreacorvini ,

I am facing same issue. Can you please send me the link for dbconnect[previous version] as well.

jjfeitosa@gmail.com

Thanks

0 Karma

andreacorvini
Path Finder

I shared the app with you.

0 Karma

matteotrombetta
New Member

Can you send me the app as well? Thanks.
@andreacorvini

0 Karma

tdubicz
Engager

Dear Matt,
I have a copy of the splunk-db-connect_311. If you provide me an email address i'll gladly send it to you!
BR,
Tamas

0 Karma

matteotrombetta
New Member

Thanks for the quick reply. My Email is -> zauberpferd12@gmail.com
Thanks again.
Mat

0 Karma

jfeitosa_real
Path Finder

I'm going to install this version and report the result here.

Thank you very much!

0 Karma

jfeitosa_real
Path Finder

Hi @andreacorvini .

Really Worked!

Thank you!

0 Karma

qthalia
Explorer

I would suggest to use HEC /collector REST API http://docs.splunk.com/Documentation/SplunkCloud/7.0.3/Data/Sendmetricstoametricsindex to get server verbose response body and learn more about failure reason. For example :
curl -k https://localhost:8088/services/collector \
-H "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29" \
-d '{"time": 1486683865.000,"event":"metric","source":"disk","host":"host_99","fields":{"region":"us-west-1","datacenter":"us-west-1a","rack":"63","os":"Ubuntu16.10","arch":"x64","team":"LON","service":"6","service_version":"0","service_environment":"test","path":"/dev/sda1","fstype":"ext3","_value":1099511627776,"metric_name":"total"}}'

HEC authorization token can be found in Settings->Data Inputs->HTTP Event Collector->db-connect-http-input (or create your own token. Also uncheck 'Enable indexer acknowledgement' if the api returns code=10 in the response)
HTTP can be found in Settings->Data Inputs->Global Settings

0 Karma

gjanders
SplunkTrust
SplunkTrust

Had this exact same symptom but a very different cause in DB Connect 3.1.2

After debugging it was found that the host field on the input was empty (which is allowed as it defaults to the connection hostname), however since the connection was set to use the JDBC URL the hostname was blank there as well.

This appears to be only possible if while creating the connection the edit JDBC URL button is pressed and no hostname is left in the box.

I have case open to ensure the application gets updated to prevent this scenario but it was very difficult to find the issue, the root cause was DB Connect attempting to send a hostname of "null" via HTTP Event Collector which also has minimal logging available

thambisetty
SplunkTrust
SplunkTrust

after giving value to host field in the inputs its started writing/indexing records.
Thank you.

————————————
If this helps, give a like below.
0 Karma

dyuran
Explorer

Hi.
Tell me where can I download the version DB Connect 3.1.2?
I can not download it on the site (((
Could you send it to my e-mail dyuran@list.ru?

0 Karma

andreacorvini
Path Finder

I sent you an email with a link to download it.
Ciao
Andrea

0 Karma

marxsabandana
Path Finder

May I have that version as well? Please send it here: marxsabandana@gmail.com

0 Karma

dyuran
Explorer

Thank you, this version works for me.

0 Karma

LH_SPLUNK
Explorer

Hello @andreacorvini,
I've the same problem with 3.1.3.
Can you send the link to lukahoff@gmx.de please?
Thanks so much.
LH

0 Karma

andreacorvini
Path Finder

ok, shared.

0 Karma

LH_SPLUNK
Explorer

Thanks!
LH

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...