topic Re: How to get match to work with two sources in Splunk Search

How to get match to work with two sources

rossparfect — Mon, 26 Aug 2019 20:17:18 GMT

Okay so here's the problem, 2 sources and I have to rex out a file name and match it to a field within another file if it matches show success it if fails show failed, all sounds achievable in Splunk, but I keep hitting a problem that it will show a success match but won't show a failed match,

Heres the code ( I've changed the fields etc but see below):

Any help on this would be great Im new to splunk 
index="sample_log" sourcetype="STAR:OUT:ALL" | rename Trans_Name as REF |lookup REF APPL-FILENAME as Stream_Name |join 
REF [search index="sample_log" sourcetype="STAR:OUT:ACK" | rex field=source "ACK
_(?<REF>\w+)_" ]
|rex field=source "_(?<STP_Process_Time>\d{14})."
| eval results = if(match('REF', "\w{10}\d{6}\w+"), "Success", "Failed")
|table results

Re: How to get match to work with two sources

MuS — Mon, 26 Aug 2019 20:27:40 GMT

Hi rossparfect,

give this a try:

(index="sample_log" sourcetype="STAR:OUT:ALL") OR (index="sample_log" sourcetype="STAR:OUT:ACK" )
| rex field=source "ACK _(?<REF>\w+)_"
| eval REF=case(isnotnull(Trans_Name),  Trans_Name, isnotnull(REF), REF, 1=1, "unknown") 
| lookup REF APPL-FILENAME as Stream_Name 
| rex field=source "_(?<STP_Process_Time>\d{14})."
| eval results = if(match(REF, "\w{10}\d{6}\w+"), "Success", "Failed")
| table results

If this still does not work, start removing | SPL from the back until you get the expected results.

Otherwise provide some sanitised sample events and the expect result and it will be easier to help.

Hope this helps ...

cheers, MuS

Re: How to get match to work with two sources

Sukisen1981 — Mon, 26 Aug 2019 20:28:26 GMT

hmm you are performing a join by REF and you extract a rex also as REF.
So, what happens? by default the join command takes common field (A intersection B in set language), are you sure you , in fact have REF fields which should return failed?
The intersection join be default will return events (with REF field) common to both searches. Do the common REF fields have the same pattern as your match regex?

Re: How to get match to work with two sources

rossparfect — Mon, 26 Aug 2019 20:29:59 GMT

Ill give that a shot and get back to you thanks Mus ( should add ive been reading all your other answers and your splunk conf presentations) all good stuff ( that said I think Ive ended up down a dark hole with this one,

Ill update shortly,

Thanks

Ross

Re: How to get match to work with two sources

MuS — Mon, 26 Aug 2019 20:37:23 GMT

HaHA, nice played Sir 😉

Re: How to get match to work with two sources

rossparfect — Mon, 26 Aug 2019 20:55:52 GMT

Hi Mus cant attach files do you have an email address I can send to ?

Re: How to get match to work with two sources

rossparfect — Mon, 26 Aug 2019 20:58:33 GMT

or example would be as follows

CSV files and outbound file and then an inbound acknowledgement file,

The outbound would have the field transmission inside the file

transmission
123456

the ack file would only have success in the file itself but in the file name would have

123456

so I have to extract the file name from the ack file and then match it to a field name in another file, all with different sources.

I can get it to show a success but it wont show any failed files (which is what we need as well), tried the above and it came back with 5 out of 5 but should only have 2 out of five,

Re: How to get match to work with two sources

MuS — Mon, 26 Aug 2019 21:14:32 GMT

It sounds/look to me the regex does not match correct, can you add a list of some REF values here?

Re: How to get match to work with two sources

rossparfect — Mon, 26 Aug 2019 22:41:02 GMT

Hi Mus , Ill add some tomorrow, its 23:36 in the Uk and Im knackered, however if I do a stats values and an MV expand i do get a result that the fields match, however cant do much after that with that search, Ill post that up tomorrow as well,

Thanks for your help so far,

Re: How to get match to work with two sources

rossparfect — Wed, 30 Sep 2020 01:57:03 GMT

Evening Mus

Ok another day at it and still struggling ( tried a case statement and it works when I put in the txt for the ref and compare) but not when I try and compare the 2 fields.

So heres some example sources and sourcetypes

Source type one - obi:one:ken:out
Source type two - obi:one:ken:ack

source - ack_123456aaabbb_13052019.csv - contents Success
souce starwars_outbound.csv - contents Transmission / 123456aaabbb

What needs to happen

I need to extract the tranmission from the source file and then match it to the contents in the outbound.csv file.

I then need to put all the details in a table with additional fields from a lookup and contents from the file, but and this is where it starts to fall down,

If there isnt a match or the source file doesnt exist ( it may come in later) then I need to show that as failed.

The best I can get is either having it working using stats(values) and an MVexpand which doesnt allow me much else that I can display, or just having a success being displayed, for example for 2 out of the 5 files,

When I should see the following

Transmission Outtype Filetype Acknowledgement Time
123456aaabbb D Light Success
123456aaaccc E sabre failed

Re: How to get match to work with two sources

MuS — Wed, 28 Aug 2019 23:21:38 GMT

You're provided samples here that will never match the initial posted search

For example you're doing a regex on source for ACK but provide source as ack_ above, the other thing is that REF should match another regex of 10 any word characters followed by 6 digits followed by one or more any word character but the provided sample contains 6 digits followed by 6 any word characters ¯\_(ツ)_/¯

You really should provide some real sample events, otherwise there is no way I can help you.

cheers, MuS

Re: How to get match to work with two sources

rossparfect — Wed, 30 Sep 2020 02:00:23 GMT

thanks Mus,

Alas cant post any real events due to the company I work for,

However apologies yes it should be ACK,

So for example ( and these are as close as I can get to real events)

File one is

Outside_SHM300TW_20190730.csv

Contents in it

XAHBAABCXX21081AA,SHM300TW

Which has to then be matched to a different source that has the long ref number in the title of the CSV

ACK_XAHBAABCXX21081AA_20190730.csv

So what I need to do is regex out the number after the ACK and then match it to contents of the other file,

however I also need to show the events that dont have a match yet,

Hopefully that helps as thats the closest I can get to sharing events ( as Ive had to make these up based on the actual events)

thanks

Ross

Re: How to get match to work with two sources

rossparfect — Thu, 29 Aug 2019 09:16:39 GMT

Hi Mus,

Feel free to swear on this one ( Im scottish so do it alot) however Ive sorted my issue,

I spoke to the data owners again and I have used and outer to do the join then a eval on if the actual process time is present for the ack file ( as if it isnt there isnt an ack file), this allows me to show a result or not sent,

So all in all, now can have a Acknowledgement of either success or Not sent,

Thanks for your help though and Ill continue watching the videos, ppts and following you on Splunk answers.

onwards and upwards

Thanks

Ross

Re: How to get match to work with two sources

MuS — Thu, 29 Aug 2019 20:03:45 GMT

HaHA, no reason to swear 😉

Glad you made it work for you in the end!

Slàinte, MuS