dealing with duplicate variables after tranaction command

[jalfrey]

I am working with IPFix data from a firewall. The first template returns the flow information. That is stuff like Source IP, Destination IP, Flow ID. The second template returns URL related data. Using the trasaction command I can correlate the events based on the Flow ID. In both templates are the dest_mac. The trouble is one of the templates always returns the MAC of 00:00:00:00:00:00 which is bad data. I would like to filter that data out any way possible.

How do I filter it?

[kristian_kolb]

If you want to filter it out before it is even indexed, you could use the anonymization techniques discussed in the docs (see below) to remove the faulty MAC before it even reaches the index;

http://docs.splunk.com/Documentation/Splunk/6.0.1/Data/Anonymizedatausingconfigurationfiles

If you want to replace it during search, i.e. after the transaction, you can use the following (assuming the multivalued field is called MAC;

... | transaction FlowID | eval MAC=mvfilter(MAC != 00:00:00:00:00:00) |

or, if they always come in the same order (in this case the good MAC always come before the bad)

... | transaction FlowID | eval MAC=mvindex(MAC,0) 

if the bad always come before the good, use mvindex(MAC,1)

See more here;
http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/CommonEvalFunctions

/k

[kristian_kolb]

Yes, well. These operations will not change the event that is presented on screen (which is stored in the _raw field). However, they will change the extracted field values, so what happens if you run the following;

index=sonicwall TemplateID=257 OR TemplateID=262 | transaction session_id startswith=TemplateID=257 | head 2 |eval dest_mac=mvfilter(dest_mac !="00:00:00:00:00:00") | table session_id dest_mac

/k

[jalfrey]

search:
index=sonicwall TemplateID=257 OR TemplateID=262 | transaction session_id startswith=TemplateID=257 | eval dest_mac=mvfilter(dest_mac !="00:00:00:00:00:00")
returns:
TemplateID=257 session_id=2149159280 src_mac=00:50:56:a1:75:51 dest_mac=00:00:00:00:00:00 src_ip=192.168.1.1 dest_ip=192.168.1.254 initiator_GW-IP_Addr=0.0.0.0 responder_GW-IP_Addr=0.0.0.0 src_int=19 src_port=63843 dest_port=443 init_to_resp_pkts=8 init_to_resp_octets=3260 init_to_resp_delta_pkts=8 init_to_resp_delta_octets=3260 start_time=2014-01-28 09:29:12 end_time=1969-12-31 16:00:00 tcp_flag=1 protocol=6 app_id=49178