Is a conditional erex appropriate for my scenario of extracting a folder location?

[DaClyde]

I have a dashboard where the user selects options from a couple of drop downs which are driven by lookups. These drop-downs combine to construct a file path for the search. Menu 1 is a list of location codes ($location$) or a wildcard for All (*), Menu 2 is a list of data types ($dataype$). Unfortunately, two of the data types are at different folder levels, (\server\datatype\location vs \servername\folder\datatype\location), so I haven't been able to work out a consistent way to extract the location folder because it changes depending on data type. I ended up using an erex to extract the location field on the fly like this:

erex jkreceive examples="tx,$location$,al,pa,colorado,ny,stl"

My erex wasn't working 100% of the time, but if I included the $location$ variable in the examples it works great. However, at the end of the search I need to translate the location codes to a descriptive location name using another lookup. This is all fine if the user chooses a location from the menu, but if the user selects "All" from the location menu, which translates to "*", the erex fails and the location to description translations all fail except the locations specified as examples.

I tried doing an eval with an if statement to create a conditional erex, but Splunk keeps terminiating the search saying I'm missing a ). No matter how many )'s I add, it ends the same. Here is the search (in my lookup, 'jkreceive' is the field containing the location codes and 'Site' is the descriptive name):

index="jkr" sourcetype="jkr" file_path="*$datatype$\$location$\\*" 
| eval jkreceive=if(jkreceive="*",(erex jkreceive examples="tx,$location$,al,pa,colorado,ny,stl"),(erex jkreceive examples="tx,al,pa,colorado,ny,stl"))
| eval MB=round(filesize/1024/1024, 2)  
`convertxfr`
| eval "Creation Date"= strptime(timestamp,"%m/%d/%Y %H:%M:%S")  
| convert timeformat="%Y-%m-%d  %T" ctime("Creation Date") AS "Creation Date (Local)" 
| lookup Locations.csv jkreceive OUTPUT Site  
| rename filename as File 
| appendpipe [ stats count 
| eval Site="There are no results to display"
| where count==0] 
| table Site "Transfer Date (CT)" "Creation Date (Local)" File MB      
| sort +"Creation Date (Local)" 

Is a conditional erex possible? Or am I barking up the wrong tree?

[somesoni2 · Accepted solution]

Give this a try

index="jkr" sourcetype="jkr" file_path="*$datatype$\$location$\\*" 
 | rex field=file_path "\/(?<jkreceive>[^\/]+)\/(?<file_name>([A-z0-9\.\s-_:-])*)$"
 | search jkreceive=$location$ 
 | eval MB=round(filesize/1024/1024, 2)  
 `convertxfr`
 | eval "Creation Date"= strptime(timestamp,"%m/%d/%Y %H:%M:%S")  
 | convert timeformat="%Y-%m-%d  %T" ctime("Creation Date") AS "Creation Date (Local)" 
 | lookup Locations.csv jkreceive OUTPUT Site  
 | rename filename as File 
 | appendpipe [ stats count 
 | eval Site="There are no results to display"
 | where count==0] 
 | table Site "Transfer Date (CT)" "Creation Date (Local)" File MB      
 | sort +"Creation Date (Local)" 

[DaClyde]

Not having any luck with this. There are no slashes in the path, only backslashes. If I try to paste it into search, it complains about a missing ), even though there isn't one missing. If I paste it into my dashboard, I just get the same Encountered the following error while trying to update: Error parsing XML on line 303: Premature end of data in tag view line 1 message that makes no sense to me. Looks like something in the regex is escaping the quotes and causing problem with the xml tags in the dashboard.

In an attempt to clarify further, if I had paths like this:

\\server\share\folder\tx12\filename.ext
\\server\share\huntsville\filename.ext
\\server\share\folder\36-99VA\filename.ext

I need to extract tx12, huntsville and 36-99VA as 'jkreceive'.

[somesoni2]

We may be making it very difficult. If 2nd last segment is to be matched with $location$ from the existing field file_path, try this. The split function will split based on backward slash and mvindex will take the 2nd last value.

index="jkr" sourcetype="jkr" file_path="*$datatype$\$location$\\*" 
  | eval jkreceive=mvindex(split(file_path,"\\"),-2)
  | search jkreceive=$location$ 
  | eval MB=round(filesize/1024/1024, 2)  
  `convertxfr`
  | eval "Creation Date"= strptime(timestamp,"%m/%d/%Y %H:%M:%S")  
  | convert timeformat="%Y-%m-%d  %T" ctime("Creation Date") AS "Creation Date (Local)" 
  | lookup Locations.csv jkreceive OUTPUT Site  
  | rename filename as File 
  | appendpipe [ stats count 
  | eval Site="There are no results to display"
  | where count==0] 
  | table Site "Transfer Date (CT)" "Creation Date (Local)" File MB      
  | sort +"Creation Date (Local)" 

[DaClyde]

Ah, that works beautifully! I was so hung up on a regex solution I wasn't considering other options. Thank you sir, answer accepted and points awarded.

[somesoni2]

Do you mean the paths can be ="/This/is/multi/location/file_name.ext or /This/is/multi/location/file- 1.ext and 2nd last work location is what you want to extract, then you can do something like this. (Everything before the rex is just to generate sample data)

| gentimes start=-1 | eval myField="/This/is/multi/location/file_name.ext#/This/is/multi/location/file- 1.ext" | table myField | makemv myField delim="#" | mvexpand myField 
| rex field=myField "\/(?<location>[^\/]+)\/(?<file_name>([A-z0-9\.\s-_:-])*)$"

[DaClyde]

The two paths would be more like \\server\share\location\filename.ext or \\server\share\folder\location\filename.ext.

[somesoni2]

What are different values that location dropdown can have? You're basically trying to extract jkreceive field from the raw data, so what are different formats it may appear (sample entries for all formats)? You could just use rex to get the field extracted and we can help build that if we can see some logs.

[DaClyde]

That is part of my problem, when the location folders were created, we had a "standard", but no thought was given to something like parsing the log data with Splunk, so the "standard" isn't very regex friendly. Basically that "location" folder name can be any combination of letters, numbers and a dash (-). And because the folder level is not always the same, we can't just key off the slashes in the path.

In theory, if we could start at the end of the path, work back from the file, and say the value before the final \ in the path was the location...that would work. I did something similar to break out the actual filename at the end of the path, but couldn't figure out how to get just the one folder preceeding the slash.