ingest events with a specific string - remove the rest

[schose]

Hi all,

I'm trying to ingest (multiline) events with the string "public_ip" and remove the rest 

props.conf:

[public_ips]
TRANSFORMS-removeallnonsense = remove_unneeded

transforms.conf

[remove_unneeded]
REGEX = (?m)^((?!public_ip).)*$
DEST_KEY = queue
FORMAT = nullQueue

When i now try to ingest Data using HEC:

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk 4f40e8ab-99a6-479f-ba13-7352feb11111' \
-d '{"sourcetype": "public_ips", "event":"foobar"}'

is not indexed - fine.

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk 4f40e8ab-99a6-479f-ba13-7352feb11111' \
-d '{"sourcetype": "public_ips", "event":"foobar public_ip: 1.2.3.4 foobar1"}'

this is indexed - fine

when running: 

curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk 4f40e8ab-99a6-479f-ba13-7352feb11111' \
-d '{"sourcetype": "public_ips", "event":"foobar public_ip: 1.2.3.4 foobar\nline2"}'

This is not indexed - not fine.  It seems i have a regex multiline issue i do not see..

Thanks for your help in advance,

Andreas

[gcusello · Accepted solution]

Hi @schose,

could you share a sample of your logs? in this way I can analyze your regex.

Anyway, try a different approach (as described at https://docs.splunk.com/Documentation/Splunk/8.1.0/Forwarding/Routeandfilterdatad#Filter_event_data_... "take public_ip events and discrd the rest" instead of "discard all not public_ip events"; something like this:

In props.conf:

[public_ips]
TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = public_ip
DEST_KEY = queue
FORMAT = indexQueue

Ciao.

Giuseppe

[schose]

Hi @gcusello ,

I tried to share some demo data within the curl commands. 🙂

Anyway... the example you posted worked for me, awesome!

Thanks and best regards,

Andreas

[gcusello]

Hi @schose,

good for you.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉