How to extract fields from different events into one field?

[vrmandadi]

Hello Experts,

I am using the interactive field extractor (IFE) to extract URL and status from every event, but the problem is that the URL and status are not in the same position for every event like:

sample event 1

18 Aug 2017 14:45:54 [WARN ] http_srv: Total latency exceeded threshold: 0.180145 seconds (internal 0.180000 s) origin 0.140000 seconds MCHit 0 Status: 200 IP: 69.xxx.xxx.13:2062 URL: http://mmdai-linear-west-01.com/linear-scope010.com/LIVE/1005/hls/ae/DAL_ESPN00/.swn7434a936-f6e3-4e...... (id 50191596)

sample event 2

18 Aug 2017 15:06:30 [WARN ] http_cli: Origin latency exceeded threshold: 0.141100 seconds GET Status: 200[OK] Bytes: 10081 Origin URL: http://linear-scope010.com/LIVE/1007/hls/ae/MSNBC_HD/900.m3u8 refReqId 13542238 reqWait 0 (id 3289754508)

sample event 3

Aug 2017 17:58:07 [WARN ] http_srv: DONE 4199750 0.023084 404[Not Found] UNKNOWN-ID 69.xxx.xxx.13:34038 GET http://mmdai-linear-west-01.com/linear-scope010.com/LIVE/1024/hls/ae/ANE_HD/.swn10562d2c-a676-4764-8...... (id 5

Is it possible to extract and create a new field URL and Response for URL and Status, respectively?

Thanks in advance

[niketn]

Hi vrmandadi, I have answered something similar for your other question on Splunk Answers (https://answers.splunk.com/answers/564999/extract-fields-from-url.html#answer-563812)
In case you need URL field you can use something like the following:

| rex field=_raw "http:\/\/(?<URL>[^\s]+)\s"
| eval URL="http:\\".URL

I have tested the regular expression http:\/\/(?<URL>[^\s]+)\s on regex101.com. Please test and confirm.

[somesoni2]

All these events are from same sourcetype? The 3rd event is throwing off field extraction as there is no good pattern to fix Status value in the 3rd sample.

[vrmandadi]

ya all events are from the same sourcetype, ya third event is a problem.Can it be done through IFX?

[somesoni2]

It's too complex for IFX to do it on it's own. You may have to edit the regex that IFX comes to capture required fields, at least the URL can be done via single regex. Try (URL:\s)*(?<URL>http\S+) to get URL and \s(?<Response>\d{3})(\s|\[) for Response field extracted.

[vrmandadi]

Well I tried the URL one but the problem is ,its also taking the
http_srv
http_cli , which is at the starting of the event

18 Aug 2017 15:06:30 [WARN ] http_cli: Origin latency exceeded threshold: 0.141100 seconds GET Status: 200[OK] Bytes: 10081 Origin URL: http://linear-scope010.com/LIVE/1007/hls/ae/MSNBC_HD/900.m3u8 refReqId 13542238 reqWait 0 (id 3289754508)

and also it is not taking the complete URL as below ,it is taking only till m3u8

http://linear-scope010.com/LIVE/1007/hls/ae/MSNBC_HD/900.m3u8 refReqId 13542238 reqWait 0 (id 3289754508)

http://linear-scope010.com/LIVE/1007/hls/ae/MSNBC_HD/900.m3u8 refReqId 13542238 reqWait 0 (id 3289754508)

The status was is not showing up in the fields section

[niketn]

Hi vrmandadi, I have answered something similar for your other question on Splunk Answers (https://answers.splunk.com/answers/564999/extract-fields-from-url.html#answer-563812)
In case you need URL field you can use something like the following:

| rex field=_raw "http:\/\/(?<URL>[^\s]+)\s"
| eval URL="http:\\".URL

I have tested the regular expression http:\/\/(?<URL>[^\s]+)\s on regex101.com. Please test and confirm.