topic Re: How to upload log files to Splunk using REST API? in Getting Data In

How to upload log files to Splunk using REST API?

dilippanwar — Thu, 06 Nov 2014 17:56:58 GMT

Hi ,

I want to upload log files using Splunk Rest APIs. Can you please share how I can do that

Re: How to upload log files to Splunk using REST API?

yannK — Tue, 24 Nov 2015 18:24:31 GMT

I am curious too, any success ?

I saw API methods to convert an uploaded file to a lookup
https://:/services/data/lookup-table-files

POST Create a lookup table file by moving a file from the upload staging area into $SPLUNK_HOME
http://docs.splunk.com/Documentation/Splunk/6.3.1/RESTREF/RESTknowledge

What is the method to upload the file to the staging area ?
according to this answer https://answers.splunk.com/answers/152485/can-you-create-modify-a-lookup-file-via-rest-api.html
"But you can't remotely upload a new lookup file with these REST endpoints , you'd need to create a Custom REST Endpoint to do this."

Re: How to upload log files to Splunk using REST API?

bizmate — Wed, 02 Mar 2016 23:36:50 GMT

I am also stuck, I would like to upload logs but I want to gather this data without using the Splunk Forwarder due to limitations on the machines. Any chance this can be achieved or are we still stuck. Look-up tables look like separate things than loading raw data with some tags (like source, type etc) to an index. But I might be wrong of course. Still learning about Splunk

Re: How to upload log files to Splunk using REST API?

frobinson_splun — Wed, 02 Mar 2016 23:47:32 GMT

Hi @bizmate,
You might want to see the documentation for the HTTP Event Collector:
http://dev.splunk.com/view/event-collector/SP-CAAAE6M

and associated REST endpoints:
http://docs.splunk.com/Documentation/Splunk/6.3.3/RESTREF/RESTinput#services.2Fcollector

http://docs.splunk.com/Documentation/Splunk/6.3.3/RESTREF/RESTinput#data.2Finputs.2Fhttp

Hope this helps!

Re: How to upload log files to Splunk using REST API?

bizmate — Thu, 03 Mar 2016 15:05:40 GMT

Hi @frobinson, your suggestion gave me some hope. I am trying splunk with a cloud instance until i can provision a local enterprise instance. I have enabled the token as suggested in the documentation.
See - http://dev.splunk.com/view/event-collector/SP-CAAAE7F

I have tried to upload my application logs

$ curl -ki https://prd-p-XXXXXXX.cloud.splunk.com:8088/services/collector -H 'Authorization: Splunk 61EC1DEF-XXXXXXXXXXXXXXXXXXXXX' -d @application201603031354.log 
curl: (7) Failed to connect to prd-p-XXXXXXX.cloud.splunk.com port 8088: Connection timed out

The strange thing is that when i set a token I dont get the same screen visible in the documentation, i.e. I dont see a confirmation of the hostname to send the request to. I have popped in on IRC to ask how i could get a confirmation of the hostname, if that was the problem. Also i confirm the EC is enabled in the global configuration so I am stuck right now. Can EC be enabled on the cloud?

Re: How to upload log files to Splunk using REST API?

frobinson_splun — Thu, 03 Mar 2016 18:08:13 GMT

Hi @bizmate,
I didn't realize until your most recent comment that you are on Splunk Cloud. Let me check with our engineering team to see what differences there are and what you can do. I'll report back!

Re: How to upload log files to Splunk using REST API?

bmacias84 — Thu, 03 Mar 2016 18:47:10 GMT

Yes, you can upload log data via the API. Though I would use a forwarder or HTTPEvent collector.

To upload data you have to use the receivers/simple endpoint using the post method. The post body will contain the your event using an XML or Json format.

https://<host>:<mPort>/services/receivers/simple

Reciever Example
API summary

Re: How to upload log files to Splunk using REST API?

gblock_splunk — Thu, 03 Mar 2016 19:21:13 GMT

Hi dilippanwar

HEC is not the best for uploading files. If you are using our JSON format, you need to parse your data and then turn it into our JSON event protocol. Our new Raw endpoint won't require that as it supports arbitrary text, but it is only available in cloud currently, and it has a default size limit of the payload being 1 meg.

A better option for file upload would be to use our one shot upload API as you can send it a file directly.

In terms of HEC in the cloud being enabled. You can enable it yourself in single instance or trial. For a clustered cloud config, you have to work with support to get the endpoint opened and for token management. You can ask support to open up our REST API (8089) and then use the Splunk CLI / REST API to also manage tokens.

Let me know if you have any questions.

Re: How to upload log files to Splunk using REST API?

frobinson_splun — Thu, 03 Mar 2016 19:25:38 GMT

As a follow-up--please see @gblock 's answer and one shot upload suggestion below 🙂

Re: How to upload log files to Splunk using REST API?

bizmate — Thu, 03 Mar 2016 23:23:08 GMT

Thanks for this suggestion but just to clarify the documentation for the one shot endpoint states 'The path to the file to be indexed. The file must be locally accessible by the server.' but the process/question is to upload a raw file from the client to the splunk server and not reference a file on the splunk server.

Assuming we want to use the oneshot endpoint I guess we need to upload the file first with another endpoint? I ll check the reference.

Going back to 'HEC in the cloud being enabled', pls see my example below. I was getting connection error though I have enabled it and generated a token.

Re: How to upload log files to Splunk using REST API?

bizmate — Fri, 04 Mar 2016 12:37:10 GMT

I downvoted this post because the oneshot endpoint is not for upload of data, as data should be already on the server in the form of a file

Re: How to upload log files to Splunk using REST API?

jrodman — Fri, 04 Mar 2016 15:19:26 GMT

While it's possible to use the UI feature of "upload a file to splunk" and then review the pattern of splunkd_access.log files to see how it accomplishes this, I wouldn't recommend it for production workflow.

Why don't you simply transfer the files to a location that Splunk monitors on another host? If you want the data to go away when Splunk completes it, you can transfer the files into a monitored sinkhole.

Re: How to upload log files to Splunk using REST API?

leosanchezcasad — Thu, 04 Jan 2018 17:00:34 GMT

I downvoted this post because it is about uploading log files, no log data in a specific format.

Re: How to upload log files to Splunk using REST API?

bmacias84 — Thu, 04 Jan 2018 18:45:04 GMT

My post has nothing to do with format. It simply states that you can use the rest endpoint post your date. That endpoint is https://:/services/receivers/simple.