Sending data to an index using an script

[MatheoCaneva1]

Hi everyone!

Quick question. I would like to know how can I send data to an index using a python script.

We need to ingest some data without using a forwarder and I would like to use an script for this reason.

Did anyone do this already?

Ty!

Regards.

[PrewinThomas]

@MatheoCaneva1 

You can send data directly to a Splunk index using a Python script by leveraging the HTTP Event Collector (HEC) or the Splunk SDK for Python. Both methods bypass the need for a forwarder

Option 1 - Send Data via HTTP Event Collector
-Enable HEC in Splunk
-Create script and send data

Option 2 - Use Splunk SDK for Python
-Install splunk SDK
-Create script using Splunk SDK and send data

Option 1 is lightweight, fast and easy.
Option 2 is having more functionalities, since you are interacting with full Splunk API.

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[PickleRick]

That's not entirely true. Typically you'd set up a HEC input on a HF layer. True, you can use HEC input directly on indexers but it's not the best solution typically. 

Also, what does "install Python SDK and write a script" mean? Have you ever done that? With SDK you can write a modular input which... tadaaaam! runs on a HF. Technically - again - you could run it on idx but that's an even worse idea.

[PrewinThomas]

@PickleRick 

Thanks for adding that extra context! Just to clarify, I wasn’t suggesting anything should run on an indexer(I believe you referenced this in earlier comment) — I was outlining the available ingestion methods (HEC and SDK). And yep, I completely agree: HEC is usually best deployed on a Heavy Forwarder, especially in production environments(Again it depends on the requirements/situation).

Also, I’ve actually built both standalone scripts and modular inputs using the Python SDK(If you have dev background, yep its straight forward) — so I meant that quite literally! It’s a solid way to integrate external sources without needing a forwarder
#https://dev.splunk.com/enterprise/docs/devtools/python/sdk-python/

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[PickleRick]

OK. Let me be a bit more precise.

SDK as such is not a "method". It's a Software Development Kit which can help you in writing your code but it still has to use one of the available methods.

In Splunk's case the ways to get data to Splunk are:

1) Pushing from remote via HEC (or generally, other inputs used by Splunk out of the box - writing to files and monitor them, sending via syslog and so on)

2) Writing own moduar inputs (that's generally where SDK helps).

Both of these methods need an input on Splunk's side. And the main point here is that you cannot go without a forwarder, unless you create an input directly on indexer(s) which is not advisable.

OK, technically, you could go via the "let's craft a search which will do something and call collect at the end" but it's an even worse idea so I will not even acknowledge that it exists.

There is no other way than through inputs to "get something into Splunk". And SDK is not a "method of getting the data in". It's just a component which helps you write Splunk-related code. It's a completely different layer.

[livehybrid · Accepted solution]

Hi @MatheoCaneva1 

You can send data to a Splunk index using a Python script via the HTTP Event Collector (HEC).

Yo uwill need to enable HEC in Splunk if not already done, create a token, and specify the target index in the token configuration.

Here's a basic Python example using the requests library to send a JSON event:

python
import requests
import json

# Replace with your values
splunk_host = "https://your-splunk-instance:8088"  # HEC endpoint (default port 8088)
hec_token = "your-hec-token-here"
index = "your_target_index"  # Ensure the token allows this index

# Sample event data
event_data = {
    "event": "This is a test event from Python",
    "sourcetype": "mysourcetype",
    "index": index,
    "fields": {
        "severity": "info"
    }
}

# Send the event
headers = {
    "Authorization": f"Splunk {hec_token}"
}
response = requests.post(f"{splunk_host}/services/collector/event", headers=headers, data=json.dumps(event_data))

print(response.status_code)
print(response.text)

This script sends a single event as JSON to the specified Splunk index, however you can send an array of events if needed.

Ensure the HEC token has permissions for the target index, and the Splunk instance is reachable (handle SSL if using HTTPS). I would recommend testing with small data volumes first.

Check out https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector for more info on HEC including setting up, as well as https://help.splunk.com/en/splunk-enterprise/get-data-in/collect-http-event-data/http-event-collecto... which covers further examples.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick]

You must get it to Splunk somehow. The easiest way would be to send events to a HEC input created on a HF or indexer.