topic Re: ingest events with a specific string - remove the rest in Getting Data In

ingest events with a specific string - remove the rest

schose — Wed, 09 Dec 2020 16:16:50 GMT

Hi all,

I'm trying to ingest (multiline) events with the string "public_ip" and remove the rest

props.conf:

[public_ips]
TRANSFORMS-removeallnonsense = remove_unneeded

transforms.conf

[remove_unneeded]
REGEX = (?m)^((?!public_ip).)*$
DEST_KEY = queue
FORMAT = nullQueue

When i now try to ingest Data using HEC:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk 4f40e8ab-99a6-479f-ba13-7352feb11111' \
-d '{"sourcetype": "public_ips", "event":"foobar"}'

is not indexed - fine.

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk 4f40e8ab-99a6-479f-ba13-7352feb11111' \
-d '{"sourcetype": "public_ips", "event":"foobar public_ip: 1.2.3.4 foobar1"}'

this is indexed - fine

when running:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk 4f40e8ab-99a6-479f-ba13-7352feb11111' \
-d '{"sourcetype": "public_ips", "event":"foobar public_ip: 1.2.3.4 foobar\nline2"}'

This is not indexed - not fine. It seems i have a regex multiline issue i do not see..

Thanks for your help in advance,

Andreas

Re: ingest events with a specific string - remove the rest

gcusello — Wed, 09 Dec 2020 16:35:02 GMT

Hi @schose,

could you share a sample of your logs? in this way I can analyze your regex.

Anyway, try a different approach (as described at https://docs.splunk.com/Documentation/Splunk/8.1.0/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues) "take public_ip events and discrd the rest" instead of "discard all not public_ip events"; something like this:

In props.conf:

[public_ips] TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = public_ip DEST_KEY = queue FORMAT = indexQueue

Ciao.

Giuseppe

Re: ingest events with a specific string - remove the rest

schose — Wed, 09 Dec 2020 16:44:07 GMT

Hi @gcusello ,

I tried to share some demo data within the curl commands. 🙂

Anyway... the example you posted worked for me, awesome!

Thanks and best regards,

Andreas

Re: ingest events with a specific string - remove the rest

gcusello — Wed, 09 Dec 2020 16:51:03 GMT

Hi @schose,

good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉