Splunk Search

Is it possible to use dedup or "|" commands in the base search of a data model?

bwindham
Path Finder

I have an instance using ServiceNow data where I want to dedup the data based on sys_updated_on to get the last update and status of the incident.

sourcetype="snow:pm_project" | dedup number sortby -sys_updated_on

However, I keep getting "|" pipes are not allowed.

I started looking at modifying the data model json file, but still got the message.

I might note that I am using "Root Event" to get acceleration to work with this.
I know I can do searches to use dedup. Should I use "Root Search" and "Root Event" together? Not sure how I would do that.

This is the search I ultimately want:

sourcetype="snow:pm_project" | dedup number sortby -sys_updated_on | search state=-5   

where state would be child objects beneath the base search.

Any help would be appreciated.....I have spent a lot of time banging my head on this and want to use data models for acceleration.

0 Karma
1 Solution

tskinnerivsec
Contributor

A base search is usually just that, very basic "sourcetype = . . . " I've used a number of Data Models that have duplicate values in their fields, take for instance the Email data model, depending on how many sourcetypes are feeding it and from where, you can easily have duplicate messageid vailues. I have never needed to solve this problem within the data model, I've always been able to dedup after the data model search. For example:

| tstats summariesonly=t allow_old_summaries=t sum(All_Email.size) as "EmailMessageSize", from datamodel=Email where All_Email.user = $user$ groupby All_Email.user _time All_Email.subject All_Email.message_id All_Email.recipient span=1s | search All_Email.recipient != *foo* | dedup All_Email.message_id | . . .

View solution in original post

tskinnerivsec
Contributor

A base search is usually just that, very basic "sourcetype = . . . " I've used a number of Data Models that have duplicate values in their fields, take for instance the Email data model, depending on how many sourcetypes are feeding it and from where, you can easily have duplicate messageid vailues. I have never needed to solve this problem within the data model, I've always been able to dedup after the data model search. For example:

| tstats summariesonly=t allow_old_summaries=t sum(All_Email.size) as "EmailMessageSize", from datamodel=Email where All_Email.user = $user$ groupby All_Email.user _time All_Email.subject All_Email.message_id All_Email.recipient span=1s | search All_Email.recipient != *foo* | dedup All_Email.message_id | . . .

bwindham
Path Finder

Thanks tskinnerivesec....I'll go that route.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...