Splunk Search

Is it possible to use dedup or "|" commands in the base search of a data model?

bwindham
Path Finder

I have an instance using ServiceNow data where I want to dedup the data based on sys_updated_on to get the last update and status of the incident.

sourcetype="snow:pm_project" | dedup number sortby -sys_updated_on

However, I keep getting "|" pipes are not allowed.

I started looking at modifying the data model json file, but still got the message.

I might note that I am using "Root Event" to get acceleration to work with this.
I know I can do searches to use dedup. Should I use "Root Search" and "Root Event" together? Not sure how I would do that.

This is the search I ultimately want:

sourcetype="snow:pm_project" | dedup number sortby -sys_updated_on | search state=-5   

where state would be child objects beneath the base search.

Any help would be appreciated.....I have spent a lot of time banging my head on this and want to use data models for acceleration.

0 Karma
1 Solution

tskinnerivsec
Contributor

A base search is usually just that, very basic "sourcetype = . . . " I've used a number of Data Models that have duplicate values in their fields, take for instance the Email data model, depending on how many sourcetypes are feeding it and from where, you can easily have duplicate messageid vailues. I have never needed to solve this problem within the data model, I've always been able to dedup after the data model search. For example:

| tstats summariesonly=t allow_old_summaries=t sum(All_Email.size) as "EmailMessageSize", from datamodel=Email where All_Email.user = $user$ groupby All_Email.user _time All_Email.subject All_Email.message_id All_Email.recipient span=1s | search All_Email.recipient != *foo* | dedup All_Email.message_id | . . .

View solution in original post

tskinnerivsec
Contributor

A base search is usually just that, very basic "sourcetype = . . . " I've used a number of Data Models that have duplicate values in their fields, take for instance the Email data model, depending on how many sourcetypes are feeding it and from where, you can easily have duplicate messageid vailues. I have never needed to solve this problem within the data model, I've always been able to dedup after the data model search. For example:

| tstats summariesonly=t allow_old_summaries=t sum(All_Email.size) as "EmailMessageSize", from datamodel=Email where All_Email.user = $user$ groupby All_Email.user _time All_Email.subject All_Email.message_id All_Email.recipient span=1s | search All_Email.recipient != *foo* | dedup All_Email.message_id | . . .

bwindham
Path Finder

Thanks tskinnerivesec....I'll go that route.

0 Karma
Get Updates on the Splunk Community!

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...

Enterprise Security Content Update (ESCU) | New Releases

In October, the Splunk Threat Research Team had one release of new security content via the Enterprise ...