Getting Data In
Highlighted

LINE_BREAKER doesn't seem to work for new add-on

Path Finder

Hi,

I've been trying to create a new add-on to ingest some data into a new sourcetype within splunk via a REST API service. Unfortunately the api returns JSON as a full array so I am unable to use the add-on creators function to automatically separate arrays into separate events.

I have been doing a lot of reading into event breaking and looked at a number of solutions to this using LINEBREAKER and BREAKBEFORE_ONLY however in both instances I haven't succeeded in breaking the event.

The JSON I have been using is [{"idOrg":"abc123","name":"Joe Smith","active":true,"id":"xyz789"},{"idOrg":"efg456","name":"Michael Thomas","active":true,"id":"uvw456"},{"idOrg":"hij789","name":"Craig Lease","active":true,"id":"rst123"}]

I'm trying to get the data into splunk so each nested json is it's own entry. I.e.
{"idOrg":"abc123","name":"Joe Smith","active":true,"id":"xyz789"}
{"idOrg":"abc123","name":"Michael Thomas","active":true,"id":"uvw456"}
{"idOrg":"abc123","name":"Craig Lease","active":true,"id":"rst123"}

I have managed to get this to work as intended when importing this data via the sourcetype editor using the following in props.conf

  • SHOULD_LINEMERGE = false
  • LINE_BREAKER = }(,){

However when running this outside the add-on creator on my machine, I cannot get the same level of success.

It's worthwhile noting that I have the following commands in my props.conf in addition to the ones above:

  • KV_MODE = json
  • SEDCMD-remove_header = s/[//g
  • SEDCMD-remove_footer = s/]//g
  • pulldown_type = 1

So far I have taken the following actions in order to attempt to resolve this. All of which have had no benefit.

  1. Change LINE_BREAKER to \}(,)\{
  2. Add a new SEDCMD to replace }, with }NEWLINE and change LINEBREAKER to NEWLINE. The SEDCMD worked as expected but no luck with LINEBREAKER
  3. Set SHOULDLINEMERGE to true and LINEBREAKER to BREAKBEFOREONLY

Any advice at this stage would be very gratefully received.

Regards

Tom

0 Karma
Highlighted

Re: LINE_BREAKER doesn't seem to work for new add-on

Motivator

Hi there,

Did you try below combination,

BREAK_ONLY_BEFORE = (?m)\{\"idOrg\"\:
SHOULD_LINEMERGE = true
0 Karma
Highlighted

Re: LINE_BREAKER doesn't seem to work for new add-on

Path Finder

Hi,

Thankyou very much for your suggestion, however unfortunately it made no difference.

The answer in the end was as simple as editing the data input within the add-on creator and specify the JSON path as$.[*] within the event extraction settings.

Thanks

Tom

View solution in original post

0 Karma
Highlighted

Re: LINE_BREAKER doesn't seem to work for new add-on

Explorer

Hi @tomawest,

I am facing similar situation where I am trying to extract events from newrelic addon and unable to perform line break for JSON lines from single event to multiple events. Could you please help me explain with stanza configuration and how it is done like which conf file and what stanza i need to add JSON path $.[*] ?

Thanks,
Shashank Soni

0 Karma