SEDCMD Help

[ldnail_at_TI]

I am attempting to use SEDCMD on ingest to eliminate extra "data" from my logs (and license). This will be running on Heavy Forwarder. Turns out SEDCMD only works on _raw during ingest which is complicated with the Palo TA as it separates CONFIG, THREAT, TRAFFIC, etc.. into their own sourcetypes, so I have to operate off of sourcetype=pan:log which looks like this:

Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,GLOBALPROTECT,...
Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,HIPMATCH,...
Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,CONFIG,...
Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,THREAT,...
Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,TRAFFIC,...

I need to perform the SEDCMD only on lines with a TRAFFIC in the 4th field, which I can identify just fine with:     

^(?:[^,]*,){3}TRAFFIC

The challenge begins here. I need to capture the first field, which is from start of the line up to the first comma. So for this line:

     Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,TRAFFIC,...

I only want to capture

     Jan 7 10:19:47 palohost 1

Any advise?

[kamlesh_vaghela]

@ldnail_at_TI 

Can you please try this?

[YOUR_SOURCE_TYPE]
SEDCMD-a = s/,(.*)TRAFFIC,(.*)$//g
.
.
.

Thanks
KV
▄︻̷̿┻̿═━一

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated. 

[ldnail_at_TI]

that captures everything afterwards

thanks

[kamlesh_vaghela]

ok @ldnail_at_TI 

What is your expected OP from above screen?

KV

[ldnail_at_TI]

---

@kamlesh_vaghela wrote:

ok @ldnail_at_TI 

What is your expected OP from above screen?

KV

---

"   I only want to capture

     Jan 7 10:19:47 palohost 1
"
which would then be deleted in sedcmd

[kamlesh_vaghela]

@ldnail_at_TI 

SEDCMD-a = s/,(.*)TRAFFIC,(.*)$//g

will give you below results.

Note.  This configuration will work with new coming event only.

I have used this sample event.

Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,GLOBALPROTECT,...
Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,HIPMATCH,...
Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,CONFIG,...
Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,THREAT,...
Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,TRAFFIC,...

Thanks
KV
▄︻̷̿┻̿═━一

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.

[ldnail_at_TI]

Thats pretty much the same result as the last and its trips everything but the first field. Results with your SEDCMD

5/28/21 3:32:29.100 PM

May 28 15:32:30 palohost 1 host = pdq source = palohost-0528-15.log sourcetype = pan:log

Source log line:

May 28 15:32:30 palohost 1,2021/05/28: 20:32:29,011901036309,TRAFFIC,end,2305,2021/05/28 20:32:29,<someip>,<someip>,0.0.0.0,0.0.0.0,<rule>,<user>,,<protocol>,<virtualsystem>,EXTTUNNEL,EXTINSIDE,tunnel,ethernet0/0,,,150000,,50000,443,0,0,,tcp,allow,143734,112711,31023,340,2021/05/28 20:30:59,75,<classification>,,,,,,,211,129,<action>,123,456,0,0,,palohost,from-policy,,,0,,0,,N/A,0,0,0,0,<some guid>,0,0,,,,,,,

Here is what I want it to look like without the first field.

5/28/21 3:32:29.100 PM

,2021/05/28: 20:32:29,011901036309,TRAFFIC,end,2305,2021/05/28 20:32:29,<someip>,<someip>,0.0.0.0,0.0.0.0,<rule>,<user>,,<protocol>,<virtualsystem>,EXTTUNNEL,EXTINSIDE,tunnel,ethernet0/0,,,150000,,50000,443,0,0,,tcp,allow,143734,112711,31023,340,2021/05/28 20:30:59,75,<classification>,,,,,,,211,129,<action>,123,456,0,0,,palohost,from-policy,,,0,,0,,N/A,0,0,0,0,<some guid>,0,0,,,,,,, host = xxx source = palohost-0528-15.log sourcetype = pan:traffic

[kamlesh_vaghela · Accepted solution]

@ldnail_at_TI 

Can you please try this? 

SEDCMD-a = s/[^,]+(.*TRAFFIC,)/\1 /g

Screen:

Thanks
KV
▄︻̷̿┻̿═━一

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.