topic Re: Searching a URL for file name that may contain spaces in Splunk Search

Searching a URL for file name that may contain spaces

Mkaz — Mon, 13 Feb 2017 15:58:42 GMT

Have a record in a log that looks like the following:

Wed Oct 26 10:41:14 2016 0 10.40.112.27 437434 /dirlevel1/dirlevel2/dirlevel3/dirlevel4/chr 2610 4109.txt b s o r aaa_aaaaaaa ssh 0 *

The record is delimited by spaces. I'm trying to pull the filename from the directory provided: /dirlevel1/dirlevel2/dirlevel3/dirlevel4/chr 2610 4109.txt

The issue I'm running into is that the file name may have a space or multiple spaces in it. The following code works, but provides the next set of filed values when it runs into a space within the file name. If the search can be performed from right to left starting at the "b" in the 8th field from the left and take everything from that point to the right up till the first "/" that would be fine. Not sure how to do that though? Any suggestions?

Code used is:

index="ti_is_st" sourcetype="xfer_log" URI=* Status=* IP_Address=* File_Size=* Service_Account=*| rex field=URI "\/(([^\s\/]+\/)*)(?<fileName>[\S]+)" |search fileName="*" Service_Account="*"|table _time IP_Address Service_Account fileName File_Size Status  |replace o with "Download Successful" i with "Upload Successful" j with "Upload Errored" k with "Upload Aborted"  p with "Download Errored" q with "Download Aborted" in Status

Thanks

Re: Searching a URL for file name that may contain spaces

somesoni2 — Mon, 13 Feb 2017 17:04:07 GMT

Give this a try

index="ti_is_st" sourcetype="xfer_log" URI=* Status=* IP_Address=* File_Size=* Service_Account=*| rex field=_raw "^(\S+\s+){8}\/(([^\s\/]+\/)+)(?<fileName>.+)(\s+\S+){8}$" |search fileName="*" Service_Account="*"|table _time IP_Address Service_Account fileName File_Size Status  |replace o with "Download Successful" i with "Upload Successful" j with "Upload Errored" k with "Upload Aborted"  p with "Download Errored" q with "Download Aborted" in Status

Re: Searching a URL for file name that may contain spaces

jrballesteros05 — Mon, 13 Feb 2017 17:07:49 GMT

Hello, I don't understand the question very well (Maybe is my English :D) but I think you want to extract this:

/dirlevel1/dirlevel2/dirlevel3/dirlevel4/chr 2610 4109.txt

Am I right?

I think your problem is regex, if you help me with more information I might help you.

Re: Searching a URL for file name that may contain spaces

Mkaz — Mon, 13 Feb 2017 17:38:50 GMT

Thanks for your response...

Well... Yes, basically just trying to pull the complete filename including the spaces only. The "/dirlevel1/dirlevel2/dirlevel3/dirlevel4/" is not required.

Re: Searching a URL for file name that may contain spaces

Mkaz — Mon, 13 Feb 2017 17:41:41 GMT

Thanks for your response...

This sits in a query statement and is throwing an error: Encountered the following error while trying to update: In handler 'views': Error parsing XML on line 37: Premature end of data in tag form line 1

Re: Searching a URL for file name that may contain spaces

somesoni2 — Mon, 13 Feb 2017 17:50:00 GMT

Are you updating the query in dashboard from Edit -> Source xml option?
If yes,then use this

index="ti_is_st" sourcetype="xfer_log" URI=* Status=* IP_Address=* File_Size=* Service_Account=*| rex field=_raw "^(\S+\s+){8}\/(([^\s\/]+\/)+)(?&lt;fileName&gt;.+)(\s+\S+){8}$" |search fileName="*" Service_Account="*"|table _time IP_Address Service_Account fileName File_Size Status  |replace o with "Download Successful" i with "Upload Successful" j with "Upload Errored" k with "Upload Aborted"  p with "Download Errored" q with "Download Aborted" in Status

Re: Searching a URL for file name that may contain spaces

Mkaz — Mon, 13 Feb 2017 18:04:22 GMT

Great thanks... Tried it and its close. Its pulling the file name correctly but not the status filed. In the record below, its pulling a status of "of" as opposed to the "o" which would be converted to Download Successful. Its happening on all records that have a space in the file name. This one happened to have several spaces.

Record:
NNN NNNNN aaaaa-Aaaa of Aaaaaa Aaaaaa NNNN.xls b s o r AAAAAA ssh 0 *

Re: Searching a URL for file name that may contain spaces

jrballesteros05 — Tue, 29 Sep 2020 12:50:44 GMT

Hello, if you only need the filename. I would do it in two ways.

If the filename comes with the metadata "source" you can extract in the props.conf and create a new field:

EXTRACT-filename=\S+\/(?P.*?).txt in source
If the filename does not come with the metadata "source", you can use the

index="ti_is_st" sourcetype="xfer_log" URI=* Status=* IP_Address=* File_Size=* Service_Account=| rex field=_raw "\S+\/(?P.?).txt" |search fileName="" Service_Account=""|table _time IP_Address Service_Account fileName File_Size Status |replace o with "Download Successful" i with "Upload Successful" j with "Upload Errored" k with "Upload Aborted" p with "Download Errored" q with "Download Aborted" in Status
In the two ways, the key is the regex you are using. I tried "\S+\/(.*?).txt" in regex101.com and it worked for me.

I hope this help you.

Re: Searching a URL for file name that may contain spaces

Mkaz — Mon, 13 Feb 2017 18:40:39 GMT

Thanks Jr... My apologies for not stating this earlier, but the file names can end in multiple file formats such as .txt, .xls, .xfr, etc... There would also be mainframes file that may be named aaaaa.aaaaaa.aaaaaaa.aaaaa.

Re: Searching a URL for file name that may contain spaces

jrballesteros05 — Mon, 13 Feb 2017 18:45:54 GMT

Ok, you can use this regex.

   \S+\/(?P<filename>.*?)\..*?\s+

Re: Searching a URL for file name that may contain spaces

somesoni2 — Mon, 13 Feb 2017 19:32:09 GMT

How are you extracting Status field? I don't see a Status field being extracted in the query itself, so it's probably extracted using saved field extractions and you should check the regular expression their on why Status field is wrong for your sample event.

Re: Searching a URL for file name that may contain spaces

Mkaz — Tue, 29 Sep 2020 12:52:06 GMT

This is the full query we're using. I tried the basic \s+\S+){5,6}$ in Regex101 and it seemed to pull properly, but what I have isn't assigning the correct code. Its pulling part of the NNNN in the filename. Also, we're pulling the file size from the record also which seems to be out of alignment now.

index="ti_is_st" sourcetype="xfer_log" | rex field=_raw "^(\S+\s+){8}\/(([^\s\/]+\/)+)(?<fileName>.+)(\s+\S+){8}$" |rex field=_raw "(\s+\S+){5,6}$(?<status>.+(i|j|k|o|p|q))\s"|search "$field2$" "$field3$" |table _time ip_address Service_Account fileName file_size status |replace o with "Download Successful" i with "Upload Successful" j with "Upload Errored" k with "Upload Aborted" p with "Download Errored" q with "Download Aborted" in status

Record:

Wed Oct 26 10:41:14 2016 0 10.40.112.27 437434 /dirlevel1/dirlevel2/dirlevel3/dirlevel4/chr 2610 4109.txt b s o r aaa_aaaaaaa ssh 0 *

Re: Searching a URL for file name that may contain spaces

Mkaz — Wed, 15 Feb 2017 14:16:31 GMT

Thanks... I was able to get it to work via the query below. I was trying to pull the status code out of the record also, which I am still having issues with.

I tried the basic \s+\S+){5,6}$ in Regex101 and it seemed to pull properly, but what I have isn't assigning the correct code. Its pulling part of the NNNN in the filename. Also, we're pulling the file size from the record also which seems to be out of alignment now.

index="ti_is_st" sourcetype="xfer_log" | rex field=_raw "^(\S+\s+){8}\/(([^\s\/]+\/)+)(?.+)(\s+\S+){8}$" |rex field=_raw "(\s+\S+){5,6}$(?.+(i|j|k|o|p|q))\s"|search "$field2$" "$field3$" |table _time ip_address Service_Account fileName file_size status |replace o with "Download Successful" i with "Upload Successful" j with "Upload Errored" k with "Upload Aborted" p with "Download Errored" q with "Download Aborted" in status

Record:

Wed Oct 26 10:41:14 2016 0 10.40.112.27 437434 /dirlevel1/dirlevel2/dirlevel3/dirlevel4/chr 2610 4109.txt b s o r aaa_aaaaaaa ssh 0 *