topic Re: Curl Command: Why is getting data into Splunk with Python https request so slow? in All Apps and Add-ons

Curl Command: Why is getting data into Splunk with Python https request so slow?

henne959 — Tue, 21 May 2019 18:12:58 GMT

Hi,

I am testing a simple Python script to load data into Splunk using the HTTP-Event-Collector.
The data comes in so this is not the problem but the time the script takes to send each datapoint.

My script:

import datetime
import random
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)



import requests

headers = {
    'Authorization': 'Splunk 7da83bdd-7d7c-4603-b814-fc505fec312a', 'Connection': 'close'
}

for i in range(1,20):
    t = datetime.datetime.now()
    timestamp = str(t.timestamp())
    data = '{"sourcetype":"_json_adv","time":"'+timestamp+'", "event":{"name":"uwu","value":"'+str(random.uniform(0,100))+'"}}'


    response = requests.post('https://localhost:8088/services/collector', headers=headers, data=data, verify=False)


    print(response.elapsed)

The statement print(response.elapsed) prints this out:

0:00:01.005077
0:00:01.004486
0:00:01.004662
0:00:01.004560
0:00:01.006372
0:00:01.008843
0:00:01.005079
0:00:01.015055
0:00:01.006656
0:00:01.004583
0:00:01.003251
0:00:01.003271
0:00:01.004073
0:00:01.007512
0:00:01.005102
0:00:01.006401
0:00:01.006413
0:00:01.005493
0:00:01.005120

The statement print(response.headers) prints this out:

{'Date': 'Tue, 21 May 2019 18:00:03 GMT', 
'Content-Type': 'application/json; 
charset=UTF-8', 
'X-Content-Type-Options': 'nosniff', 
'Content-Length': '27', 
'Vary': 'Authorization',
'Connection': 'Close', 
'X-Frame-Options': 'SAMEORIGIN', 
'Server': 'Splunkd'}

In another thread on Github (https://github.com/kennethreitz/requests/issues/4023) I stumbled upon the statement 'Connection': 'Close' in the request header to speed it up but that didn't shortened the time for each request.
I tried another request at some test website with response times <200ms.
For me it seems to be a problem with the splunk server.

Anybody know a solution to speed this up?

Thanks!

Re: Curl Command: Why is getting data into Splunk with Python https request so slow?

henne959 — Tue, 21 May 2019 18:16:16 GMT

In the end I want to read the data with the script from a PLC and forward it Splunk this way. Therefore multiple datapoints have to be sent in <1s.

Re: Curl Command: Why is getting data into Splunk with Python https request so slow?

henne959 — Tue, 21 May 2019 18:19:07 GMT

The header I send is
{'User-Agent': 'python-requests/2.18.4',
'Accept-Encoding': 'gzip, deflate',
'Accept': '/',
'Connection': 'close',
'Authorization': 'Splunk 7da83bdd-7d7c-4603-b814-fc505fec312a',
'Content-Length': '105'}

Re: Curl Command: Why is getting data into Splunk with Python https request so slow?

starcher — Wed, 22 May 2019 18:30:17 GMT

https://github.com/georgestarcher/Splunk-Class-httpevent

Re: Curl Command: Why is getting data into Splunk with Python https request so slow?

henne959 — Wed, 22 May 2019 19:18:15 GMT

Thanks, didn't find that on my own...
Oh and I see your the author, nice work!