Splunk Enterprise Security

Help with creating field extractions for map

vikram1583
Explorer

Can you help map creating field extractions Please use the ES CIM model where possible for field names:

There are some variations in the log files so I included these two that we’re looking at:

2019-09-17 **:**:**.**** [Level: INFO][Server: **********][ServerIP: ::1][ThreadId: 141][RequesterIP:**.***.1.1][Verb:POST][RequestUri:https://***svcv3/authenticationgateway/profile/******/login][Headers:[Connection:Keep-Alive|Content-Length:118|Content-Type:application/json|Accept:*/*|Accept-Language:en-us |User-Agent:iOS **Bank (Version 2.18.0 Build 80329; 12.4.1; en-US; iPhone(iPhone11,2); |X-GDC-DeviceID:BA8BB0C7-2FF8-4C37-B17B-A5F01148D38E|X-GDC-Digest:l2RLaisPFvk6libgtBFQb85Sh17kM5moYGp6ipQ2Su0=|X-GDC-SessionToken:fe9bc5d5-259d-402b-aa35-861e0d260068|X-GDC-Method:2|X-GDC-Timestamp:2019-09-17T22:41:10.009|Originator:FlexClient|X-GDC-Version:1.001|X-GDC-ApplicationID:10043|X-GDC-MessageID:BABBFB13-F781-4FF6-B777-894BAF5CBD8A|RequestId:AEABFB13-F781-4FF6-B777-894BAF5CBD8A|X-Forwarded-For:108.**.233.***, 127.**.242.145, 10.126.**.250|X-Original-URL:/***/auth/**/profile/tokens/login| "AuthenticationLevel":"1","WebUserToken":"354643"}"][TimeTaken:][StatusCode:Created(201)]
2019-09-13 23:**:51.3120 [Level: INFO][Server: *****SVC04][ServerIP: ::1][ThreadId: 58][Response:{ ErrorCode = 10003, ErrorDescription = Unable to process the login request, "Code":30116267

Below are the fields need to be extracted:

Accept-Language
User-Agent
X-GDC-DeviceID
X-GDC-SessionToken
X-GDC-Method
X-GDC-ApplicationID
X-Forwarded-For
X-Original-URL
AuthenticationLevel
WebUserToken
StatusCode
ErrorCode
ErrorDescription
Code

For X-Forwarded-For, please only capture the first IP address.

0 Karma

jacobpevans
Motivator

Greetings @vikram1583,

Here's a run-anywhere search to extract Accept-Language as AcceptLanguage. Try playing with that to get the rest of the fields. They're all nearly identical.

| makeresults
| eval _raw = "2019-09-17 ::.** [Level: INFO][Server: **********][ServerIP: ::1][ThreadId: 141][RequesterIP:**.***.1.1][Verb:POST][RequestUri:https://***svcv3/authenticationgateway/profile/******/login][Headers:[Connection:Keep-Alive|Content-Length:118|Content-Type:application/json|Accept:*/*|Accept-Language:en-us |User-Agent:iOS **Bank (Version 2.18.0 Build 80329; 12.4.1; en-US; iPhone(iPhone11,2); |X-GDC-DeviceID:BA8BB0C7-2FF8-4C37-B17B-A5F01148D38E|X-GDC-Digest:l2RLaisPFvk6libgtBFQb85Sh17kM5moYGp6ipQ2Su0=|X-GDC-SessionToken:fe9bc5d5-259d-402b-aa35-861e0d260068|X-GDC-Method:2|X-GDC-Timestamp:2019-09-17T22:41:10.009|Originator:FlexClient|X-GDC-Version:1.001|X-GDC-ApplicationID:10043|X-GDC-MessageID:BABBFB13-F781-4FF6-B777-894BAF5CBD8A|RequestId:AEABFB13-F781-4FF6-B777-894BAF5CBD8A|X-Forwarded-For:108.**.233.***, 127.**.242.145, 10.126.**.250|X-Original-URL:/***/auth/**/profile/tokens/login| \"AuthenticationLevel\":\"1\",\"WebUserToken\":\"354643\"}\"][TimeTaken:][StatusCode:Created(201)]"
| append 
    [ | makeresults
      | eval _raw = "2019-09-13 23:**:51.3120 [Level: INFO][Server: *****SVC04][ServerIP: ::1][ThreadId: 58][Response:{ ErrorCode = 10003, ErrorDescription = Unable to process the login request, \"Code\":30116267" ]

| rex "Accept-Language:(?<AcceptLanguage>[^\|]+)"
Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma

vikram1583
Explorer

hey thanks for the reply its working and i am poor at regex can you send Rex for remaining fields please?

0 Karma

jacobpevans
Motivator

I'll help you out a little more.

rex "Accept-Language:(?<AcceptLanguage>[^\|]+)"

See here for more info regarding the rex command: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rex. By default, we're performing rex on the full _raw value (the two lines in your question) which is what you want. From there, we have "Accept-Language:(?<AcceptLanguage>[^\|]+)"

The basic structure is "(?<FieldName>[field extraction regex])". In your case, the Accept-Language field always starts with Accept-Language: which is why I put that before the parend. (? to signify that.

[^\|]+ - This is the magic extraction. [ and ] defines multiple possible matches. Inside that, we have ^ which just means NOT. After that, we have \| which is just the | character with the escape character \. I did this because your Accept-Language field ends with | in your sample data.

Add that all up, and we're grabbing everything between "Accept-Language:" and "|" in your sample data.

See here for a nice introduction tutorial: https://medium.com/factory-mind/regex-tutorial-a-simple-cheatsheet-by-examples-649dc1c3f285

See here for a fantastic online regex tester where you can practice using regular expressions (you can even use your data): https://regex101.com. You can test your skills here: https://regex101.com/quiz.

Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma

vikram1583
Explorer

will the same Regex work for indexing operations?

0 Karma

jacobpevans
Motivator

I don't understand the question

Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...