topic Re: New Field REX HELP (API) in Splunk Search

New Field REX HELP (API)

havatz — Mon, 28 Sep 2020 08:03:18 GMT

Hi all

I want to create a new AWS monitoring alert.

For the first step I checked the AWS fields and I saw that I need to parse the "principalId" field from the "_raw". (I want to create the rule based on principalID"

Part of the raw:

{\"type\": \"Root\", \"principalId\": \"444444444444\", \"arn\"

Im running this query:

"search index=aws userIdentity.type=Root eventName=ConsoleLogin earliest=-1d | rex field=_raw principalId\W\W:\s\W\W(?P<principalId>\d*)"

and getting results but without the "prinicpialId" new field.

What am I missing in the query?

Thanks!

Re: New Field REX HELP (API)

gcusello — Mon, 28 Sep 2020 08:15:10 GMT

Hi @havatz,

Are you sure that in your logs there's the back slash before quotes?

if not, please, try this regex:

| rex "\"principalId\":\s+\"(?<principalId>[^\"]+)\""

that you can test at https://regex101.com/r/iS9NFl/1

Otherwise, please try this regex:

| rex "principalId\\\":\s+\\\"(?<principalId>[^\"]+)"

that you can test at https://regex101.com/r/iS9NFl/2

Ciao.

Giuseppe

Re: New Field REX HELP (API)

ITWhisperer — Mon, 28 Sep 2020 08:17:03 GMT

Your second screenshot doesn't have any \ before the " although your test data (first screenshot) does. What is actually in your data? You could try

search index=aws userIdentity.type=Root eventName=ConsoleLogin earliest=-1d | rex field=_raw principalId\W+:\s\W+(?P<principalId>\d*)

Re: New Field REX HELP (API)

havatz — Mon, 28 Sep 2020 08:26:46 GMT

Getting this error:

Re: New Field REX HELP (API)

havatz — Mon, 28 Sep 2020 08:29:58 GMT

this it the raw data:

"_raw": "{\"eventVersion\": \"1.05\", \"userIdentity\": {\"type\": \"Root\", \"principalId\": \"11111111111\", \"arn\": \"arn:aws:iam::11111111111:root\", \"accountId\": \"11111111111\", \"accessKeyId\": \"\"}, \"eventTime\": \"2020-09-25T17:49:45Z\", \"eventSource\": \"signin.amazonaws.com\", \"eventName\": \"ConsoleLogin\", \"awsRegion\": \"us-east-1\", \"sourceIPAddress\": \"xxxxxxxxx\", \"userAgent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)

Re: New Field REX HELP (API)

havatz — Mon, 28 Sep 2020 08:33:42 GMT

curl -k -u "user pass" https://localhost:11111/services/search/jobs?output_mode=json -X POST -d search="search index=aws userIdentity.type=Root eventName=ConsoleLogin earliest=-30d | rex field=_raw "principalId\\\":\s+\\\"(?<principalId>[^\"]+)"

{"messages":[{"type":"ERROR","text":"Error in 'SearchParser': Missing a search command before '^'. Error at position '129' of search query 'search index=aws userIdentity.type=Root eventName=...{snipped} {errorcontext = ncipalId>[^\"] )}'.","help":""}]}

Re: New Field REX HELP (API)

ITWhisperer — Mon, 28 Sep 2020 08:34:18 GMT

You are using curl and ReST API? Try escaping the backslashes

"search index=aws userIdentity.type=Root eventName=ConsoleLogin earliest=-1d | rex field=_raw principalId\\W\\W:\\s\\W\\W(?P<principalId>\\d*)"

Re: New Field REX HELP (API)

havatz — Mon, 28 Sep 2020 08:44:28 GMT

Yes.. its API .

This query working great but im still getting the "principalId" empty

Re: New Field REX HELP (API)

havatz — Mon, 28 Sep 2020 08:48:51 GMT

The query should enter retro active on old events also? or should I wait for new ones?

Re: New Field REX HELP (API)

ITWhisperer — Mon, 28 Sep 2020 08:54:28 GMT

Have you looked at the data in the splunk GUI? Are there really backslashes before the double quotes in the data?

Try this:

"search index=aws userIdentity.type=Root eventName=ConsoleLogin earliest=-1d | rex field=_raw principalId\\W+:\\s\\W+(?P<principalId>\\d*)"

Re: New Field REX HELP (API)

havatz — Mon, 28 Sep 2020 09:30:05 GMT

You're right! My mistake
It working!! Thank you!