How to upload log files to Splunk using REST API?

[dilippanwar]

Hi ,

I want to upload log files using Splunk Rest APIs. Can you please share how I can do that

[jrodman]

While it's possible to use the UI feature of "upload a file to splunk" and then review the pattern of splunkd_access.log files to see how it accomplishes this, I wouldn't recommend it for production workflow.

Why don't you simply transfer the files to a location that Splunk monitors on another host? If you want the data to go away when Splunk completes it, you can transfer the files into a monitored sinkhole.

[gblock_splunk]

Hi dilippanwar

HEC is not the best for uploading files. If you are using our JSON format, you need to parse your data and then turn it into our JSON event protocol. Our new Raw endpoint won't require that as it supports arbitrary text, but it is only available in cloud currently, and it has a default size limit of the payload being 1 meg.

A better option for file upload would be to use our one shot upload API as you can send it a file directly.

In terms of HEC in the cloud being enabled. You can enable it yourself in single instance or trial. For a clustered cloud config, you have to work with support to get the endpoint opened and for token management. You can ask support to open up our REST API (8089) and then use the Splunk CLI / REST API to also manage tokens.

Let me know if you have any questions.

[bizmate]

I downvoted this post because the oneshot endpoint is not for upload of data, as data should be already on the server in the form of a file

[bizmate]

Thanks for this suggestion but just to clarify the documentation for the one shot endpoint states 'The path to the file to be indexed. The file must be locally accessible by the server.' but the process/question is to upload a raw file from the client to the splunk server and not reference a file on the splunk server.

Assuming we want to use the oneshot endpoint I guess we need to upload the file first with another endpoint? I ll check the reference.

Going back to 'HEC in the cloud being enabled', pls see my example below. I was getting connection error though I have enabled it and generated a token.

[bmacias84]

Yes, you can upload log data via the API. Though I would use a forwarder or HTTPEvent collector.

To upload data you have to use the receivers/simple endpoint using the post method. The post body will contain the your event using an XML or Json format.

https://<host>:<mPort>/services/receivers/simple

Reciever Example
API summary

[leosanchezcasad]

I downvoted this post because it is about uploading log files, no log data in a specific format.

[bmacias84]

My post has nothing to do with format. It simply states that you can use the rest endpoint post your date. That endpoint is https://:/services/receivers/simple.

[yannK]

I am curious too, any success ?

I saw API methods to convert an uploaded file to a lookup
https://:/services/data/lookup-table-files

POST Create a lookup table file by moving a file from the upload staging area into $SPLUNK_HOME
http://docs.splunk.com/Documentation/Splunk/6.3.1/RESTREF/RESTknowledge

What is the method to upload the file to the staging area ?
according to this answer https://answers.splunk.com/answers/152485/can-you-create-modify-a-lookup-file-via-rest-api.html
"But you can't remotely upload a new lookup file with these REST endpoints , you'd need to create a Custom REST Endpoint to do this."

[bizmate]

I am also stuck, I would like to upload logs but I want to gather this data without using the Splunk Forwarder due to limitations on the machines. Any chance this can be achieved or are we still stuck. Look-up tables look like separate things than loading raw data with some tags (like source, type etc) to an index. But I might be wrong of course. Still learning about Splunk

[frobinson_splun]

Hi @bizmate,
You might want to see the documentation for the HTTP Event Collector:
http://dev.splunk.com/view/event-collector/SP-CAAAE6M

and associated REST endpoints:
http://docs.splunk.com/Documentation/Splunk/6.3.3/RESTREF/RESTinput#services.2Fcollector

http://docs.splunk.com/Documentation/Splunk/6.3.3/RESTREF/RESTinput#data.2Finputs.2Fhttp

Hope this helps!

[bizmate]

Hi @frobinson, your suggestion gave me some hope. I am trying splunk with a cloud instance until i can provision a local enterprise instance. I have enabled the token as suggested in the documentation.
See - http://dev.splunk.com/view/event-collector/SP-CAAAE7F

I have tried to upload my application logs

$ curl -ki https://prd-p-XXXXXXX.cloud.splunk.com:8088/services/collector -H 'Authorization: Splunk 61EC1DEF-XXXXXXXXXXXXXXXXXXXXX' -d @application201603031354.log 
curl: (7) Failed to connect to prd-p-XXXXXXX.cloud.splunk.com port 8088: Connection timed out

The strange thing is that when i set a token I dont get the same screen visible in the documentation, i.e. I dont see a confirmation of the hostname to send the request to. I have popped in on IRC to ask how i could get a confirmation of the hostname, if that was the problem. Also i confirm the EC is enabled in the global configuration so I am stuck right now. Can EC be enabled on the cloud?

[frobinson_splun]

Hi @bizmate,
I didn't realize until your most recent comment that you are on Splunk Cloud. Let me check with our engineering team to see what differences there are and what you can do. I'll report back!

[frobinson_splun]

As a follow-up--please see @gblock 's answer and one shot upload suggestion below 🙂