topic Re: SEDCMD Help in Getting Data In

SEDCMD Help

ldnail_at_TI — Mon, 07 Jun 2021 18:18:08 GMT

I am attempting to use SEDCMD on ingest to eliminate extra "data" from my logs (and license). This will be running on Heavy Forwarder. Turns out SEDCMD only works on _raw during ingest which is complicated with the Palo TA as it separates CONFIG, THREAT, TRAFFIC, etc.. into their own sourcetypes, so I have to operate off of sourcetype=pan:log which looks like this:

Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,GLOBALPROTECT,... Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,HIPMATCH,... Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,CONFIG,... Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,THREAT,... Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,TRAFFIC,...

I need to perform the SEDCMD only on lines with a TRAFFIC in the 4th field, which I can identify just fine with:

^(?:[^,]*,){3}TRAFFIC

The challenge begins here. I need to capture the first field, which is from start of the line up to the first comma. So for this line:

Jan 7 10:19:47 palohost 1,2021/06/07: 15:19:46,011901036309,TRAFFIC,...

I only want to capture

Jan 7 10:19:47 palohost 1

Any advise?

Re: SEDCMD Help

kamlesh_vaghela — Tue, 08 Jun 2021 03:58:56 GMT

@ldnail_at_TI

Can you please try this?

[YOUR_SOURCE_TYPE] SEDCMD-a = s/,(.*)TRAFFIC,(.*)$//g . . .

Thanks
KV
▄︻̷̿┻̿═━一

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.

Re: SEDCMD Help

ldnail_at_TI — Tue, 08 Jun 2021 13:18:13 GMT

that captures everything afterwards

thanks

Re: SEDCMD Help

kamlesh_vaghela — Tue, 08 Jun 2021 13:44:19 GMT

ok @ldnail_at_TI

What is your expected OP from above screen?

Re: SEDCMD Help

ldnail_at_TI — Tue, 08 Jun 2021 13:48:00 GMT

@kamlesh_vaghela wrote:
ok @ldnail_at_TI
What is your expected OP from above screen?
KV

" I only want to capture

Jan 7 10:19:47 palohost 1
"
which would then be deleted in sedcmd

Re: SEDCMD Help

kamlesh_vaghela — Tue, 08 Jun 2021 14:08:28 GMT

@ldnail_at_TI

SEDCMD-a = s/,(.*)TRAFFIC,(.*)$//g

will give you below results.

Note. This configuration will work with new coming event only.

I have used this sample event.

Thanks
KV
▄︻̷̿┻̿═━一

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.

Re: SEDCMD Help

ldnail_at_TI — Tue, 08 Jun 2021 17:57:58 GMT

Thats pretty much the same result as the last and its trips everything but the first field. Results with your SEDCMD

5/28/21
3:32:29.100 PM

May 28 15:32:30 palohost 1

host = pdq
source = palohost-0528-15.log
sourcetype = pan:log

Source log line:

May 28 15:32:30 palohost 1,2021/05/28: 20:32:29,011901036309,TRAFFIC,end,2305,2021/05/28 20:32:29,<someip>,<someip>,0.0.0.0,0.0.0.0,<rule>,<user>,,<protocol>,<virtualsystem>,EXTTUNNEL,EXTINSIDE,tunnel,ethernet0/0,,,150000,,50000,443,0,0,,tcp,allow,143734,112711,31023,340,2021/05/28 20:30:59,75,<classification>,,,,,,,211,129,<action>,123,456,0,0,,palohost,from-policy,,,0,,0,,N/A,0,0,0,0,<some guid>,0,0,,,,,,,

Here is what I want it to look like without the first field.

5/28/21
3:32:29.100 PM

,2021/05/28: 20:32:29,011901036309,TRAFFIC,end,2305,2021/05/28 20:32:29,<someip>,<someip>,0.0.0.0,0.0.0.0,<rule>,<user>,,<protocol>,<virtualsystem>,EXTTUNNEL,EXTINSIDE,tunnel,ethernet0/0,,,150000,,50000,443,0,0,,tcp,allow,143734,112711,31023,340,2021/05/28 20:30:59,75,<classification>,,,,,,,211,129,<action>,123,456,0,0,,palohost,from-policy,,,0,,0,,N/A,0,0,0,0,<some guid>,0,0,,,,,,,

host = xxx
source = palohost-0528-15.log
sourcetype = pan:traffic

Re: SEDCMD Help

kamlesh_vaghela — Wed, 09 Jun 2021 05:49:01 GMT

@ldnail_at_TI

Can you please try this?

SEDCMD-a = s/[^,]+(.*TRAFFIC,)/\1 /g

Screen:

Thanks
KV
▄︻̷̿┻̿═━一

If any of my reply helps you to solve the problem Or gain knowledge, an upvote would be appreciated.