Getting Data In

Help with sending data from api to splunk enterprise

bsheppard8
Loves-to-Learn Lots

I need some help understanding how to send data from an api to splunk enterprise so that I can create a dashboard about the information. The api is open source and located at https://ghibliapi.herokuapp.com/#. I understand that I can get the information using the curl command, but how do I input this information directly into my splunk instance? I don't have the option to use REST API as an data source for 'Add Data'. So far I've tried to print the output to a txt file and monitor that file using the universal forwarder, but I can't split the data into events properly, as the data is ingested line by line and backwards, regardless of the settings to props.conf. These are the current settings in my props file:

[apiver2]
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = "(/{)"
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = ^/{
MUST_BREAK_AFTER = ^/},
MUST_NOT_BREAK_AFTER = ^"id.+
MUST_NOT_BREAK_BEFORE = ^"url.+

And it always displays like this: splunk api ingest.jpgSo I'm thinking that maybe this is because of the data format of the request. What are my options for ingesting this data? A lot of this is new to me, so would HTTP Event Collector work, or is there something else I should do?

Thanks in advance!

Tags (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

You have far more props than is necessary.  Please share some sample events so we can help you set the breaking attributes.  Be sure to indicate where one event ends and the next begins.

Note that the slashes in the props.conf settings need to be backslashes to properly escape the braces. 

Why can you not use the REST API?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

bsheppard8
Loves-to-Learn Lots

I can't use REST API because the option isn't available in my instance. I don't see it under 'Data Inputs'.

I gave an example from my earlier picture. The full event should look something like this:

{
"id": "d868e6ec-c44a-405b-8fa6-f7f0f8cfb500",
"title": "The Red Turtle",
"original_title": "レッドタートル ある島の物語",
"original_title_romanised": "Reddotātoru aru shima no monogatari",
"description": "A man set adrift by a storm wakes up on a beach. He discovers that he is on a deserted island with plenty of fresh water, fruit and a dense bamboo forest. He builds a raft from bamboo and attempts to sail away, but his raft is destroyed by an unseen monster in the sea, forcing him back to the island. He tries again with another, larger raft, but is again foiled by the creature. A third attempt again ends with the raft destroyed, but this time he is confronted by a giant red turtle, which stares at him, and forces him back to the island.",
"director": "Michaël Dudok de Wit",
"producer": "Toshio Suzuki, Isao Takahata, Vincent Maraval, Pascal Caucheteux, Grégoire Sorlat",
"release_date": "2016",
"running_time": "80",
"rt_score": "93",
"people": [
"https://ghibliapi.herokuapp.com/people/"
],
"species": [
"https://ghibliapi.herokuapp.com/species/"
],
"locations": [
"https://ghibliapi.herokuapp.com/locations/"
],
"vehicles": [
"https://ghibliapi.herokuapp.com/vehicles/"
],
"url": "https://ghibliapi.herokuapp.com/films/d868e6ec-c44a-405b-8fa6-f7f0f8cfb500"
}

But you can see in this picture here that the info comes in reverse, and every line is turned into its own event.

splunk screenshot.png

 

I also tried the backslash and restarting splunk, but that didn't change anything.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!