Someone please help me translating Regular expression for Field extractions.

[superhm]

Splunk collecting security solusion logs that is DefensPro.

I want to extract fields.

Log samples are..

Dec 26 15:59:00 10.18.18.2 DefensePro: 26-12-2017 15:59:00 WARNING 105 Anomalies "TTL Less Than or Equal to 1" UDP 59.150.19.252 1985 224.0.0.2 1985 13 Regular "Packet Anomalies" sampled 1 0 N/A 0 N/A medium forward

Dec 26 15:38:38 10.18.18.1 DefensePro: 26-12-2017 15:38:38 WARNING 113 Anomalies "Invalid TCP Flags" TCP 220.77.181.118 1497 220.64.16.210 7795 13 Regular "Packet Anomalies" sampled 1 11 N/A 0 N/A low drop

Dec 26 14:37:21 172.21.160.236 DefensePro: 26-12-2017 14:37:09 WARNING 125 Anomalies "L4 Source or Dest Port Zero" TCP 84.15.56.252 0 203.239.57.127 23 13 Regular "Packet Anomalies" sampled 1 0 N/A 0 N/A low drop
Dec 26 14:36:10 10.18.18.2 DefensePro: 26-12-2017 14:36:10 WARNING 104 Anomalies "Invalid IP Header or Total Length" TCP 180.135.189.234 0 220.64.16.250 0 13 Regular "Packet Anomalies" sampled 1 0 N/A 0 N/A low drop

And my select fields that

In regular order
1. Date : 26-12-2017 16:20:58
2. Severity : Warning
3. Category : Anomalies
4. AttackName : "TTL Less Than or Equal to 1"
5. Protocol : UDP
6. SrcIP : 59.150.19.252
7. DstIP : 224.0.0.2
8. DstPort : 1985
9. PolicyName : "Packet Anomalies"
10. AttackStatus : sampled
11. Risk : medium
12. Action : forward

Automatic log extraction by Splunk is invalid.
Can you create a regular expression to extract fields from the log above?

[harsmarvania57 · Accepted solution]

HI @superhm,

Please use below regex

DefensePro:\s(?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]*)\s\d+\s(?<Category>[^\s]*)\s(?<AttackName>[^"]*"[^"]*")\s(?<Protocol>[^\s]*)\s(?<SrcIP>[^\s]*)\s\d+\s(?<DstIP>[^\s]*)\s(?<DstPort>[^\s]*)\s\d+\s\w+\s(?<PolicyName>[^"]*"[^"]*")\s(?<AttackStatus>[^\s]*)\s\d+\s\d+\s[^\s]*\s\d+\s[^\s]*\s(?<Risk>[^\s]*)\s(?<Action>[^\s]*)

Based on above regex I have created sample query with dummy data you have provided (First 4 lines are used to generate dummy data)

|makeresults | eval _raw="Dec 26 15:59:00 10.18.18.2 DefensePro: 26-12-2017 15:59:00 WARNING 105 Anomalies \"TTL Less Than or Equal to 1\" UDP 59.150.19.252 1985 224.0.0.2 1985 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A medium forward"
|append [makeresults | eval _raw="Dec 26 15:38:38 10.18.18.1 DefensePro: 26-12-2017 15:38:38 WARNING 113 Anomalies \"Invalid TCP Flags\" TCP 220.77.181.118 1497 220.64.16.210 7795 13 Regular \"Packet Anomalies\" sampled 1 11 N/A 0 N/A low drop"]
|append [makeresults | eval _raw="Dec 26 14:37:21 172.21.160.236 DefensePro: 26-12-2017 14:37:09 WARNING 125 Anomalies \"L4 Source or Dest Port Zero\" TCP 84.15.56.252 0 203.239.57.127 23 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A low drop"]
|append [makeresults | eval _raw="Dec 26 14:36:10 10.18.18.2 DefensePro: 26-12-2017 14:36:10 WARNING 104 Anomalies \"Invalid IP Header or Total Length\" TCP 180.135.189.234 0 220.64.16.250 0 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A low drop"]
| rex "DefensePro:\s(?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]*)\s\d+\s(?<Category>[^\s]*)\s(?<AttackName>[^\"]*\"[^\"]*\")\s(?<Protocol>[^\s]*)\s(?<SrcIP>[^\s]*)\s\d+\s(?<DstIP>[^\s]*)\s(?<DstPort>[^\s]*)\s\d+\s\w+\s(?<PolicyName>[^\"]*\"[^\"]*\")\s(?<AttackStatus>[^\s]*)\s\d+\s\d+\s[^\s]*\s\d+\s[^\s]*\s(?<Risk>[^\s]*)\s(?<Action>[^\s]*)"

So your query will be

<yourBasesearch> | rex "DefensePro:\s(?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]*)\s\d+\s(?<Category>[^\s]*)\s(?<AttackName>[^\"]*\"[^\"]*\")\s(?<Protocol>[^\s]*)\s(?<SrcIP>[^\s]*)\s\d+\s(?<DstIP>[^\s]*)\s(?<DstPort>[^\s]*)\s\d+\s\w+\s(?<PolicyName>[^\"]*\"[^\"]*\")\s(?<AttackStatus>[^\s]*)\s\d+\s\d+\s[^\s]*\s\d+\s[^\s]*\s(?<Risk>[^\s]*)\s(?<Action>[^\s]*)"

I hope this helps.

Thanks,
Harshil

[superhm]

Thank you so much harsmarvania57!
You were a great help to me.
I will study more by consulting carefully. : )

[niketn]

@harsmarvania57, you beat me to it. Almost same regex. I have extracted src port and dest port both and kept provision for some other capturing groups to be extracted. Following is the run anywhere search:

| makeresults
| eval data="Dec 26 15:59:00 10.18.18.2 DefensePro: 26-12-2017 15:59:00 WARNING 105 Anomalies \"TTL Less Than or Equal to 1\" UDP 59.150.19.252 1985 224.0.0.2 1985 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A medium forward;Dec 26 15:38:38 10.18.18.1 DefensePro: 26-12-2017 15:38:38 WARNING 113 Anomalies \"Invalid TCP Flags\" TCP 220.77.181.118 1497 220.64.16.210 7795 13 Regular \"Packet Anomalies\" sampled 1 11 N/A 0 N/A low drop;Dec 26 14:37:21 172.21.160.236 DefensePro: 26-12-2017 14:37:09 WARNING 125 Anomalies \"L4 Source or Dest Port Zero\" TCP 84.15.56.252 0 203.239.57.127 23 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A low drop;Dec 26 14:36:10 10.18.18.2 DefensePro: 26-12-2017 14:36:10 WARNING 104 Anomalies \"Invalid IP Header or Total Length\" TCP 180.135.189.234 0 220.64.16.250 0 13 Regular \"Packet Anomalies\" sampled 1 0 N/A 0 N/A low drop"
| makemv data delim=";" 
| mvexpand data
| rename data as _raw
| rex "DefensePro: (?<Date>\d{2}-\d{2}-\d{4}\s\d{2}:\d{2}:\d{2})\s(?<Severity>[^\s]+)\s\d+\s(?<Category>[^\s]+)\s\"(?<AttackName>[^\"]+)\"\s(?<Protocol>[^\s]+)\s(?<SrcIP>[^\s]+)\s(?<SrcPort>[^\s]+)\s(?<DstIP>[^\s]+)\s(?<DstPort>[^\s]+)\s\d+\s([^\s]+)\s\"(?<PolicyName>[^\"]+)\"\s(?<AttackStatus>[^\s]+)\s\d+\s\d+\s([^\s]+)\s\d+\s([^\s]+)\s(?<Risk>[^\s]+)\s(?<Action>.*)"

[superhm]

Thank you for your opinion. : )
And it helped me a lot.

[harsmarvania57]

I thought to extract SrcPort but in original requirement it was not given so I skipped it. 😛