How do I converting a "curl" command to a collection configuration in "Splunk Add-On Builder" console?

[yossefn]

Hi,

I really need help with this issue.
I need to collect logs using REST from a web resource. I'm trying for a lot of time to do it by myself, unfortunately I got stock in the final step.
I have a 'curl' command I'm running against the web resource and I see the logs on my shell screen, all I'm trying to do is to convert this command to a valid REST call using the 'Add-on Builder' but I just can't finish it successfully.

Attached here is a masked version of my 'curl' command, please help me to get it done somehow so I'll be able to collect the logs.

curl -X POST 'https://api.company.webresource.com/v2/logs/audit' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Authorization:Bearer 196e6h17-798a-4e2c-64hr-xxxxxxxxxxxx' -i -d '{"query":{"from_date":1587848400000}}'

Thanks in advance!

[PavelP]

Hello @yossefn,

please try to use this config based on this example: https://answers.splunk.com/answers/736820/rest-ta-inputsconf-vs-curl.html

 [rest://test1]
 endpoint = https://api.company.webresource.com/v2/logs/audit
 activation_key = XXXACTIVATIONKEYHEREXXX
 http_method = POST
 auth_type = none
 http_header_propertys = Authorization=Bearer YYYREMOTEACCESSTMCASTOKENHEREYYY
 response_type = json
 polling_interval = 300
 index = your_index
 index_error_response_codes = 0
 sequential_mode = 0
 sourcetype = your_st
 streaming_request = 0

try to put the uploaded json in request_payload field.

Let me know if it worked.

P.S. you can try also use curl script instead of using this TA because your curl request is simple POST without auth. The only thing is the checkpointing (Typically, you store (check point) the progress of an input source so upon restart, the script knows where to resume reading data. This prevents you from reading and indexing the same data twice.)

[yossefn]

Hi @PavelP
I've created this stanza but still can't see any data coming in. How can I debug it to see the errors?

[PavelP]

Hello @yossefn ,

I've just tested and it works:

[rest://test]
activation_key = 1yourkeyhere1
auth_type = none
endpoint = http://your-server
http_method = POST
index_error_response_codes = 0
request_payload = {"query":{"from_date":1587848400000}}
response_type = text
sequential_mode = 0
sourcetype = rest_ta_1
streaming_request = 0

just fill the UI form (Data Input > REST) with your data, choose Http Method "POST", Request Payload
- {"query":{"from_date":1587848400000}} (without quotes). After you save the configuration, splunk will request the site using API every 60 seconds (can be overwritten in Polling Interval).

[yossefn]

I don't understand why, but it still not working for me.

Where should I put all the headers I have in the URL? Are the should be included as part of the endpoint value?

'https://api.company.webresource.com/v2/logs/audit' -H 'Content-Type: application/json' -H 'Accept: application/json'

In your second example I can't see the Authorization part..

[harsmarvania57]

Have you looked at app https://splunkbase.splunk.com/app/4146/#/details which has curl command and you can fire POST request.

[yossefn]

Hi @harsmarvania57 is this app will allow me to configure a continues REST collection? I want to index events from this web resource.

[harsmarvania57]

I'll suggest to try it in test environment. To index data you need to schedule the query with curl command and store the data in index.

[yossefn]

Sure, I'm working on my test machine. Thanks!