Help with creating field extractions for map

[vikram1583]

Can you help map creating field extractions Please use the ES CIM model where possible for field names:

There are some variations in the log files so I included these two that we’re looking at:

2019-09-17 **:**:**.**** [Level: INFO][Server: **********][ServerIP: ::1][ThreadId: 141][RequesterIP:**.***.1.1][Verb:POST][RequestUri:https://***svcv3/authenticationgateway/profile/******/login][Headers:[Connection:Keep-Alive|Content-Length:118|Content-Type:application/json|Accept:*/*|Accept-Language:en-us |User-Agent:iOS **Bank (Version 2.18.0 Build 80329; 12.4.1; en-US; iPhone(iPhone11,2); |X-GDC-DeviceID:BA8BB0C7-2FF8-4C37-B17B-A5F01148D38E|X-GDC-Digest:l2RLaisPFvk6libgtBFQb85Sh17kM5moYGp6ipQ2Su0=|X-GDC-SessionToken:fe9bc5d5-259d-402b-aa35-861e0d260068|X-GDC-Method:2|X-GDC-Timestamp:2019-09-17T22:41:10.009|Originator:FlexClient|X-GDC-Version:1.001|X-GDC-ApplicationID:10043|X-GDC-MessageID:BABBFB13-F781-4FF6-B777-894BAF5CBD8A|RequestId:AEABFB13-F781-4FF6-B777-894BAF5CBD8A|X-Forwarded-For:108.**.233.***, 127.**.242.145, 10.126.**.250|X-Original-URL:/***/auth/**/profile/tokens/login| "AuthenticationLevel":"1","WebUserToken":"354643"}"][TimeTaken:][StatusCode:Created(201)]
2019-09-13 23:**:51.3120 [Level: INFO][Server: *****SVC04][ServerIP: ::1][ThreadId: 58][Response:{ ErrorCode = 10003, ErrorDescription = Unable to process the login request, "Code":30116267

Below are the fields need to be extracted:

Accept-Language
User-Agent
X-GDC-DeviceID
X-GDC-SessionToken
X-GDC-Method
X-GDC-ApplicationID
X-Forwarded-For
X-Original-URL
AuthenticationLevel
WebUserToken
StatusCode
ErrorCode
ErrorDescription
Code

For X-Forwarded-For, please only capture the first IP address.

[jacobpevans]

Greetings @vikram1583,

Here's a run-anywhere search to extract Accept-Language as AcceptLanguage. Try playing with that to get the rest of the fields. They're all nearly identical.

| makeresults
| eval _raw = "2019-09-17 ::.** [Level: INFO][Server: **********][ServerIP: ::1][ThreadId: 141][RequesterIP:**.***.1.1][Verb:POST][RequestUri:https://***svcv3/authenticationgateway/profile/******/login][Headers:[Connection:Keep-Alive|Content-Length:118|Content-Type:application/json|Accept:*/*|Accept-Language:en-us |User-Agent:iOS **Bank (Version 2.18.0 Build 80329; 12.4.1; en-US; iPhone(iPhone11,2); |X-GDC-DeviceID:BA8BB0C7-2FF8-4C37-B17B-A5F01148D38E|X-GDC-Digest:l2RLaisPFvk6libgtBFQb85Sh17kM5moYGp6ipQ2Su0=|X-GDC-SessionToken:fe9bc5d5-259d-402b-aa35-861e0d260068|X-GDC-Method:2|X-GDC-Timestamp:2019-09-17T22:41:10.009|Originator:FlexClient|X-GDC-Version:1.001|X-GDC-ApplicationID:10043|X-GDC-MessageID:BABBFB13-F781-4FF6-B777-894BAF5CBD8A|RequestId:AEABFB13-F781-4FF6-B777-894BAF5CBD8A|X-Forwarded-For:108.**.233.***, 127.**.242.145, 10.126.**.250|X-Original-URL:/***/auth/**/profile/tokens/login| \"AuthenticationLevel\":\"1\",\"WebUserToken\":\"354643\"}\"][TimeTaken:][StatusCode:Created(201)]"
| append 
    [ | makeresults
      | eval _raw = "2019-09-13 23:**:51.3120 [Level: INFO][Server: *****SVC04][ServerIP: ::1][ThreadId: 58][Response:{ ErrorCode = 10003, ErrorDescription = Unable to process the login request, \"Code\":30116267" ]

| rex "Accept-Language:(?<AcceptLanguage>[^\|]+)"

[vikram1583]

hey thanks for the reply its working and i am poor at regex can you send Rex for remaining fields please?

[jacobpevans]

I'll help you out a little more.

rex "Accept-Language:(?<AcceptLanguage>[^\|]+)"

See here for more info regarding the rex command: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rex. By default, we're performing rex on the full _raw value (the two lines in your question) which is what you want. From there, we have "Accept-Language:(?<AcceptLanguage>[^\|]+)"

The basic structure is "(?<FieldName>[field extraction regex])". In your case, the Accept-Language field always starts with Accept-Language: which is why I put that before the parend. (? to signify that.

[^\|]+ - This is the magic extraction. [ and ] defines multiple possible matches. Inside that, we have ^ which just means NOT. After that, we have \| which is just the | character with the escape character \. I did this because your Accept-Language field ends with | in your sample data.

Add that all up, and we're grabbing everything between "Accept-Language:" and "|" in your sample data.

See here for a nice introduction tutorial: https://medium.com/factory-mind/regex-tutorial-a-simple-cheatsheet-by-examples-649dc1c3f285

See here for a fantastic online regex tester where you can practice using regular expressions (you can even use your data): https://regex101.com. You can test your skills here: https://regex101.com/quiz.

[vikram1583]

will the same Regex work for indexing operations?

[jacobpevans]

I don't understand the question