Deployment Architecture

forwarded events and field extraction

KhalidAlharthi
Explorer

Hello Members,

 

i have data coming from HF indexed in indexer and i can search it the problem at the details of event 

 

for example : event sample cs4=FIREEYE test

when i see the details of this event i see cs4=FIREEYE only first string other is truncated why?

 

 

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

We don't know your data, we don't know your config.

But my shot would be that your data is not properly onboarded - you don't have a proper configuration for this type of source so Splunks tries by default to extract key-value pairs and does it with its own built-in mechanics which ends up as you can see.

FireEyes can be painful to set up. Try to avoid CEF altogether - it's not very nice to parse.

0 Karma

KhalidAlharthi
Explorer

this is a sample of event 

<149>Jul 23 18:54:24 fireeye.mps.test cef[5159]: CEF:0|fireeye|HX|4.8.0|IOC Hit Found|IOC Hit Found|10|rt=Jul 23 2019 16:54:24 UTC dvchost=fireeye.mps.test categoryDeviceGroup=/IDS categoryDeviceType=Forensic Investigation categoryObject=/Host cs1Label=Host Agent Cert Hash cs1=fwvqcmXUHVcbm4AFK01cim dst=192.168.1.172 dmac=00-00-5e-00-53-00 dhost=test-host1 dntdom=test deviceCustomDate1Label=Agent Last Audit deviceCustomDate1=Jul 23 2019 16:54:22 UTC cs2Label=FireEye Agent Version cs2=29.7.0 cs5Label=Target GMT Offset cs5=+PT2H cs6Label=Target OS cs6=Windows 10 Pro 17134 externalId=17688554 start=Jul 23 2019 16:53:18 UTC categoryOutcome=/Success categorySignificance=/Compromise categoryBehavior=/Found cs7Label=Resolution cs7=ALERT cs8Label=Alert Types cs8=exc act=Detection IOC Hit msg=Host test-host1 IOC compromise alert categoryTupleDescription=A Detection IOC found a compromise indication. cs4Label=IOC Name cs4=SVCHOST SUSPICIOUS PARENT PROCESS

 

i need to do field extractions and make the event display all the data without truncating inside the details of event

0 Karma

PickleRick
SplunkTrust
SplunkTrust

It's not truncating as such. It's just that by default Splunk's key-value pairs extraction works up to a delimiter - in this case, space unless the string is quoted IIRC. Since you don't have any custom extractions defined and use default settings, Splunk simply extracts from key=value pairs.

As I said - there is at least one (I think there were more of them but some might be archived) app for ingesting CEF data. But since it's ugly because the format is not very well-specified, unless you have a very good reason for sticking with CEF, I'd suggest you go to the console and change the notification format.

To make things even more interesting, as I see on "mine" HX, the default (and actually the only available) format for notifications straight from the box is JSON. Is this a notification from CM about an alert from HX?

0 Karma

KhalidAlharthi
Explorer

i fixed the issue by using regex With SEDCMD command on HF to fix the parsing and now everything is good

 

thanks for help @PickleRick 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Interesting approach. Out of sheer curiosity - what SEDCMD did you use?

0 Karma

KhalidAlharthi
Explorer

I did this regex using SEDCMD on HF before sending data to indexers 

s/(\w+)=([^\s"][^"\r\n=]*\s[^\r\n=]*)(?=\s|$)/\1="\2"/g

 

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. That's one way to do it. Be aware thought that it probably will break if you get quotes in your field values.

0 Karma

KhalidAlharthi
Explorer

if the key=value the value has space it will quote it so splunk can parse it without any issue if there is no space splunk already knows key=value so he will parse the information without any issue .

 

@PickleRick thanks for your helping now i'm facing big problem regarding batchadding 😧

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Yes. I'm not talking about space, I'm talking about quotes.

For example, if part of your event was

cs4=SVCHOST SUSPICIOUS "PARENT" PROCESS 

(I don't care if that makes sense as such, it's just about syntax)

Your regex will turn it into

cs4="SVCHOST SUSPICIOUS "PARENT" PROCESS"

And Splunk will extract only the part up to the second or third quote.

That's why I don't like CEF - it's troublesome to manipulate. Because if you try to manually extract fields using regex anchoring on the equal sign, you end up trying to make sure it doesn't break if your equal sign is in the value of the field. (I'm not even sure CEF properly handles such situation; don't remember).

0 Karma

KhalidAlharthi
Explorer

tbh, i tried more than 2 day's to fix this situation i end up with SEDCMD i look at the event field extractions and it's appears good enough i know what are you referring if there is quotes already persent it will miss with key=value fields 

 

any ways thanks for everything now i'm facing such headche problem batch adding issue with indexers and i think it's because the bandwith of the end-points . @PickleRick 

0 Karma

KhalidAlharthi
Explorer

it's like this 

<149>Jul 23 18:54:24 fireeye.mps.test cef[5159]: CEF:0|fireeye|HX|4.8.0|IOC Hit Found|IOC Hit Found|10|rt=Jul 23 2019 16:54:24 UTC dvchost=fireeye.mps.test categoryDeviceGroup=/IDS categoryDeviceType=Forensic Investigation categoryObject=/Host cs1Label=Host Agent Cert Hash cs1=fwvqcmXUHVcbm4AFK01cim dst=192.168.1.172 dmac=00-00-5e-00-53-00 dhost=test-host1 dntdom=test deviceCustomDate1Label=Agent Last Audit deviceCustomDate1=Jul 23 2019 16:54:22 UTC cs2Label=FireEye Agent Version cs2=29.7.0 cs5Label=Target GMT Offset cs5=+PT2H cs6Label=Target OS cs6=Windows 10 Pro 17134 externalId=17688554 start=Jul 23 2019 16:53:18 UTC categoryOutcome=/Success categorySignificance=/Compromise categoryBehavior=/Found cs7Label=Resolution cs7=ALERT cs8Label=Alert Types cs8=exc act=Detection IOC Hit msg=Host test-host1 IOC compromise alert categoryTupleDescription=A Detection IOC found a compromise indication. cs4Label=IOC Name cs4=SVCHOST SUSPICIOUS PARENT PROCESS

 

this as a raw data but when i try to expand the details of event i see it's truncated . i will provide you with the config i did inside HF this morning.

 

Thx

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Yup, classic FireEye CEF. There was an add-on for FireEye on Splunkbase but it's archived already (last version was released 7 years ago so no wonder) - https://splunkbase.splunk.com/app/1904 As far as I remember it also had some issues with proper parsing.

If you want to use CEF, you might try this add-on https://splunkbase.splunk.com/app/487 but I wouldn't count on it being CIM-compliant.

0 Karma

KhalidAlharthi
Explorer

is there another options for parsing like editing props.conf since i don't want to add new app 

 

is there any possibility for this type of events to just edit props.conf?

 

my props.conf

[trellix]

category = Custom

pulldown_type = 1

TIME_FORMAT = ^<\d+>

EVAL-_time = strftime(_time, "%Y %b %d %H:%M:%S")

TIME_PREFIX = %b %d %H:%M:%S

0 Karma

PickleRick
SplunkTrust
SplunkTrust

An app is just a bunch of files. For field extractions they just contain a bunch of props/transforms settings. I'd still consider switching to a more sane reporting format first. For example - json.

0 Karma
Get Updates on the Splunk Community!

Message Parsing in SOCK

Introduction This blog post is part of an ongoing series on SOCK enablement. In this blog post, I will write ...

Exploring the OpenTelemetry Collector’s Kubernetes annotation-based discovery

We’ve already explored a few topics around observability in a Kubernetes environment -- Common Failures in a ...

Use ‘em or lose ‘em | Splunk training units do expire

Whether it’s hummus, a ham sandwich, or a human, almost everything in this world has an expiration date. And, ...