Splunk Search

Extract fields from URL

vrmandadi
Builder

Hello,

I have the below URL Types and I am trying to extract 3 fields from them

LIVE as form
hls as rule
TWAMCPH as mode

URL
Example URL1:
http://linear-scope010.abc.com/LIVE/1002/hls/ae/TWAMCPH/98.m3u8

ExampleURL2:
http://mmdai-linear-west-03.abc.com/linear-scope010.abc.com/LIVE/1008/hls/ae/Nat_HD/.swn71c39e69-9b7......


Help with regex to extract a particular field- I want to extract the numbers after as Identity which are in BOLD in the below sample event
Identity: 33752527

Sample event:

19 Aug 2017 01:09:41 [WARN ] http_srv: DONE 5018465 0.010309 404[Not Found] UNKNOWN-ID 69.134.235.12:12113 GET http://mmdai-linear-west-03.abc.com/linear-scope010.abc.com/LIVE/1008/hls/ae/Nat_HD/.swn71c39e69-9b7...... (id 33752527)

Sample event 2:

19 Aug 2017 01:16:22 [WARN ] http_cli: Origin latency exceeded threshold: 0.068990 seconds GET Status: 200[OK] Bytes: 10079 Origin URL: http://linear-scope010.abc.com/LIVE/1008/hls/ae/FX_HD/98.m3u8 refReqId 34040636 reqWait 0 (id 3291791648)

Sample event 3:

19 Aug 2017 01:16:22 [WARN ] http_srv: Total latency exceeded threshold: 0.054962 seconds (internal 0.055000 s) origin 0.000000 seconds MCHit 0 Status: 404 IP: 69.134.235.13:9290 URL: http://mmdai-linear-west-03.abc.com/linear-scope010.abc.com/LIVE/1007/hls/ae/MSNBC_HD/.swn0f1c1094-9...... (id 34040648)

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi vrmandadi,
you don't need to create a regex for each field, also using IFX you can have a single regex to extract more fields.

Use IFX and when you arrive to the extraction, there's a link to show regex, click on it and then click on modify regex, so you can insert my regex with all the fields.

Otherwise, if you already have regex, it's easier to create a new field from web interface but not using IFX, go in [Settings -- Fields -- Fields Extractions -- New], insert:

  • Destination App,
  • a name for the extraction (e.g. form_rule_mode),
  • sourcetype (it's the more important thing!)
  • and finally the full regex; in few minutes you'll have your fields.

Remember that fields will not be immediately available, but after a few minutes (I don't know why!).
Bye.
Giuseppe

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi vrmandadi,
you don't need to create a regex for each field, also using IFX you can have a single regex to extract more fields.

Use IFX and when you arrive to the extraction, there's a link to show regex, click on it and then click on modify regex, so you can insert my regex with all the fields.

Otherwise, if you already have regex, it's easier to create a new field from web interface but not using IFX, go in [Settings -- Fields -- Fields Extractions -- New], insert:

  • Destination App,
  • a name for the extraction (e.g. form_rule_mode),
  • sourcetype (it's the more important thing!)
  • and finally the full regex; in few minutes you'll have your fields.

Remember that fields will not be immediately available, but after a few minutes (I don't know why!).
Bye.
Giuseppe

vrmandadi
Builder

Thank You Giuseppe.Thanks a lot

0 Karma

niketn
Legend

@vrmandadi, your field extraction is a bit complicated only because you are looking at two different log patterns http_srv and http_cli. Are these both coming from same log (or source/sourcetype), or is it your query which has brought them together? You use case would be simpler if these are two separate sources or sourcetypes (so please confirm).

In case both the type of events are in the same source/sourcetype, you can try the following

 <YourBaseSearch>
| rex field=_raw "http:\/\/(?<URL>[^\s]+)\s"
| rex field=_raw "\(id\s(?<id>\d+)\)"
| eval URL=split(URL,"/")
| table id URL
| eval index=if(match(mvindex(URL,1),".com"),"1,2,3","0,1,2")
| eval index=split(index,",")
| eval firstIndex=mvindex(index,0)
| eval secondIndex=mvindex(index,1)
| eval thirdIndex=mvindex(index,2)
| eval form=mvindex(URL,firstIndex)
| eval rule=mvindex(URL,secondIndex)
| eval mode=mvindex(URL,thirdIndex)
| table id form mode rule
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

vrmandadi
Builder

Hello niketnilay,

I am trying to extract those fields and create new fields using the IFX, but its not successful since they dont follow a pattern and they are from same sourcetype.

I want help with the regex to extract
LIVE as form
hls as rule
TWAMCPH as mode ..Can you please help with regex for each field to extract

0 Karma

niketn
Legend

You can create a regex for URL and id since the regular expressions remains the same as that in the rex command.

You can move eval section to macro URL as input. However, give the two different types of events for http_srv and http_cli, I was not able to find same pattern applicable for both.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi vrmandadi,
regex to extract the second one you asked it's no difficoult:

\(id\s(?<identity>\d+)\)

test it at https://regex101.com/r/F1dWey/1

The problem is the first one because you have two different paths with a different number of segments before the form field:
If your sure that after "LIVE" there's a number, you can use this one:

\/(?<form>[^\/]*)\/\d+\/(?<rule>[^\/]*)\/\w+\/(?<mode>[^\/]*)

test it at https://regex101.com/r/HaU7mr/1

Bye.
Giuseppe

0 Karma

vrmandadi
Builder

Hello Giuseppe,

I am using IFX to extract separate fields for each of them,Can you please help me with regex for each of the field like the one you mentioned for ID

Thanks for your time

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...