How do I use Url_parser as a scripted lookup to add a new field in a data model?

[jlagore]

Hi, this is my first posting to this community I believe.

 I am trying to add a new field called uri_path to an existing data model called Web. The only thing in the constraint is index=web. Names have been changed in this posting.

One of the fields in the DM is url which is used to extract the url_path (different name here) via the URLParser scripted lookup.

https://splunkbase.splunk.com/app/3396/#/details

The above link says this below:

"Scripted Lookup

URLParser is also accessible as a scripted lookup. This will be useful for situations where the custom search command cannot be used like if you are building a datamodel. The scripted lookup is slower than the custom search command.

... | eval list="iana|mozilla" | lookup urlparser_lookup url list"

The following SPL works great:

index=web  | head 200

| eval list="iana|mozilla" | lookup urlparser_lookup url list

| table url_path

| eval uri_path=url_path

My question is, how do you make this work when trying to add a new field called (uri_path) to the Web  DM?

If I try to add the below as a calculated field using an Eval Expression, it doesn't work. There are errors related to searching the index when I try search the DM. If I remove the new field, the Dm search works fine again. Should the index line go here before the eval expression? I am thinking it shouldn't.

| eval list="iana|mozilla" | lookup urlparser_lookup url list

| table url_path

| eval uri_path=url_path

I am thinking maybe a lookup definition is needed to try it that way when adding a new field, but when I search for a file called urlparser_lookup, no file is found. It does not show up in the adding new field drop-down list for adding a new field using lookup either. I can't create a lookup definition if I can't find the lookup file.

I am not sure how to implement this. Any help will be much appreciated.

Regards

[scelikok]

Hi @jlagore,

I didn't mean adding these to DM search. I suggested adding search time field extraction for url_path. Then you will able to select and add this field into DM fields.

Every event has a sourcetype assigned at ingestion. . Search time fields extractions works based on sourcetype, source or host. 

1- Find your sourcetype on web logs

2- Go Settings | Source types page, find your sourcetype

3- Click Edit

4- Add these two lines at the bottom of Advanced settings;

    Name --> EVAL-list   Value --> "iana|mozilla"

    Name --> LOOKUP-urlparser  Value --> urlparser_lookup url list OUTPUT url_path

5- Click Save

After that you will be able to see url_path field when you search "index=web  | head 200"

If you see this field you can add to your DM fields.

[jlagore]

Hi scelikok,

I know what the sourcetype is, but I am not seeing anything under Settings for sourcetypes. Under eventtypes I can see the corresponding eventtype and if you drilldown I see a popup that has boxes for Search String, Tags, Color, and Priority and nothing else. I want to try what you stated, but I am not able to find the right place to edit.

[scelikok]

Hi @jlagore,

Sorry, I don't have anything to recommend, maybe you have permission problem.

[jlagore]

Hi scelikok,

I have admin rights and I don't see anything that says sourcetype under Settings. Can you send me a screenshot of what I should see under Settings. I am using Splunk 7.3.3. Thanks for your help.

[scelikok]

Hi @jlagore,

You can see sourcetype settings in below screenshot,

 

[jlagore]

Hi scelikok,

Yes, I have found out that if I log in with admin account on the backend, sourcetype shows up just like your snapshot. However, there is no sourcetype related specifically to the web index. There are a couple of generic web types like access_combined etc, but I will have to get our team admin involved to figure this out since this part is new to me. Thank you for your help.

[scelikok]

Hi @jlagore,

Can you please try adding below EVAL and LOOKUP functions as seperate entries on your sourcetype?

[your_sourcetype]
EVAL-list="iana|mozilla"
LOOKUP-urlparser=urlparser_lookup url list OUTPUT url_path

[jlagore]

Hi scelikok,

I am not sure what you mean by sourcetype here. It is web data and the url field is present in the index and should be already present for an eval expression to interpret. This  code will be in the DM add field using an eval expression. I have learned that the eval word shouldn't be used as it is not needed and works without it in normal eval situations. I realize you are using the EVAL in a different way it appears. I did learn that the url_parser_lookup is an external lookup and is a Python script that gets called.