All Apps and Add-ons

Curl Command: Why is getting data into Splunk with Python https request so slow?

henne959
Engager

Hi,

I am testing a simple Python script to load data into Splunk using the HTTP-Event-Collector.
The data comes in so this is not the problem but the time the script takes to send each datapoint.

My script:

import datetime
import random
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)



import requests

headers = {
    'Authorization': 'Splunk 7da83bdd-7d7c-4603-b814-fc505fec312a', 'Connection': 'close'
}

for i in range(1,20):
    t = datetime.datetime.now()
    timestamp = str(t.timestamp())
    data = '{"sourcetype":"_json_adv","time":"'+timestamp+'", "event":{"name":"uwu","value":"'+str(random.uniform(0,100))+'"}}'


    response = requests.post('https://localhost:8088/services/collector', headers=headers, data=data, verify=False)


    print(response.elapsed)

The statement print(response.elapsed) prints this out:

0:00:01.005077
0:00:01.004486
0:00:01.004662
0:00:01.004560
0:00:01.006372
0:00:01.008843
0:00:01.005079
0:00:01.015055
0:00:01.006656
0:00:01.004583
0:00:01.003251
0:00:01.003271
0:00:01.004073
0:00:01.007512
0:00:01.005102
0:00:01.006401
0:00:01.006413
0:00:01.005493
0:00:01.005120

The statement print(response.headers) prints this out:

{'Date': 'Tue, 21 May 2019 18:00:03 GMT', 
'Content-Type': 'application/json; 
charset=UTF-8', 
'X-Content-Type-Options': 'nosniff', 
'Content-Length': '27', 
'Vary': 'Authorization',
'Connection': 'Close', 
'X-Frame-Options': 'SAMEORIGIN', 
'Server': 'Splunkd'}

In another thread on Github (https://github.com/kennethreitz/requests/issues/4023) I stumbled upon the statement 'Connection': 'Close' in the request header to speed it up but that didn't shortened the time for each request.
I tried another request at some test website with response times <200ms.
For me it seems to be a problem with the splunk server.

Anybody know a solution to speed this up?

Thanks!

0 Karma
1 Solution

henne959
Engager

Thanks, didn't find that on my own...
Oh and I see your the author, nice work!

0 Karma

henne959
Engager

In the end I want to read the data with the script from a PLC and forward it Splunk this way. Therefore multiple datapoints have to be sent in <1s.

0 Karma

henne959
Engager

The header I send is
{'User-Agent': 'python-requests/2.18.4',
'Accept-Encoding': 'gzip, deflate',
'Accept': '/',
'Connection': 'close',
'Authorization': 'Splunk 7da83bdd-7d7c-4603-b814-fc505fec312a',
'Content-Length': '105'}

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...