Splunk Search

How to extract fields from different events into one field?

vrmandadi
Builder

Hello Experts,

I am using the interactive field extractor (IFE) to extract URL and status from every event, but the problem is that the URL and status are not in the same position for every event like:

sample event 1

18 Aug 2017 14:45:54 [WARN ] http_srv: Total latency exceeded threshold: 0.180145 seconds (internal 0.180000 s) origin 0.140000 seconds MCHit 0 Status: 200 IP: 69.xxx.xxx.13:2062 URL: http://mmdai-linear-west-01.com/linear-scope010.com/LIVE/1005/hls/ae/DAL_ESPN00/.swn7434a936-f6e3-4e...... (id 50191596)

sample event 2

18 Aug 2017 15:06:30 [WARN ] http_cli: Origin latency exceeded threshold: 0.141100 seconds GET Status: 200[OK] Bytes: 10081 Origin URL: http://linear-scope010.com/LIVE/1007/hls/ae/MSNBC_HD/900.m3u8 refReqId 13542238 reqWait 0 (id 3289754508)

sample event 3

Aug 2017 17:58:07 [WARN ] http_srv: DONE 4199750 0.023084 404[Not Found] UNKNOWN-ID 69.xxx.xxx.13:34038 GET http://mmdai-linear-west-01.com/linear-scope010.com/LIVE/1024/hls/ae/ANE_HD/.swn10562d2c-a676-4764-8...... (id 5

Is it possible to extract and create a new field URL and Response for URL and Status, respectively?

Thanks in advance

0 Karma

niketn
Legend

Hi vrmandadi, I have answered something similar for your other question on Splunk Answers (https://answers.splunk.com/answers/564999/extract-fields-from-url.html#answer-563812)
In case you need URL field you can use something like the following:

| rex field=_raw "http:\/\/(?<URL>[^\s]+)\s"
| eval URL="http:\\".URL

I have tested the regular expression http:\/\/(?<URL>[^\s]+)\s on regex101.com. Please test and confirm.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

somesoni2
SplunkTrust
SplunkTrust

All these events are from same sourcetype? The 3rd event is throwing off field extraction as there is no good pattern to fix Status value in the 3rd sample.

0 Karma

vrmandadi
Builder

ya all events are from the same sourcetype, ya third event is a problem.Can it be done through IFX?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

It's too complex for IFX to do it on it's own. You may have to edit the regex that IFX comes to capture required fields, at least the URL can be done via single regex. Try (URL:\s)*(?<URL>http\S+) to get URL and \s(?<Response>\d{3})(\s|\[) for Response field extracted.

0 Karma

vrmandadi
Builder

Well I tried the URL one but the problem is ,its also taking the
http_srv
http_cli , which is at the starting of the event

18 Aug 2017 15:06:30 [WARN ] http_cli: Origin latency exceeded threshold: 0.141100 seconds GET Status: 200[OK] Bytes: 10081 Origin URL: http://linear-scope010.com/LIVE/1007/hls/ae/MSNBC_HD/900.m3u8 refReqId 13542238 reqWait 0 (id 3289754508)

and also it is not taking the complete URL as below ,it is taking only till m3u8

http://linear-scope010.com/LIVE/1007/hls/ae/MSNBC_HD/900.m3u8 refReqId 13542238 reqWait 0 (id 3289754508)

http://linear-scope010.com/LIVE/1007/hls/ae/MSNBC_HD/900.m3u8 refReqId 13542238 reqWait 0 (id 3289754508)

The status was is not showing up in the fields section

0 Karma

niketn
Legend

Hi vrmandadi, I have answered something similar for your other question on Splunk Answers (https://answers.splunk.com/answers/564999/extract-fields-from-url.html#answer-563812)
In case you need URL field you can use something like the following:

| rex field=_raw "http:\/\/(?<URL>[^\s]+)\s"
| eval URL="http:\\".URL

I have tested the regular expression http:\/\/(?<URL>[^\s]+)\s on regex101.com. Please test and confirm.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...