Splunk Search

Is it possible to use dedup or "|" commands in the base search of a data model?

bwindham
Path Finder

I have an instance using ServiceNow data where I want to dedup the data based on sys_updated_on to get the last update and status of the incident.

sourcetype="snow:pm_project" | dedup number sortby -sys_updated_on

However, I keep getting "|" pipes are not allowed.

I started looking at modifying the data model json file, but still got the message.

I might note that I am using "Root Event" to get acceleration to work with this.
I know I can do searches to use dedup. Should I use "Root Search" and "Root Event" together? Not sure how I would do that.

This is the search I ultimately want:

sourcetype="snow:pm_project" | dedup number sortby -sys_updated_on | search state=-5   

where state would be child objects beneath the base search.

Any help would be appreciated.....I have spent a lot of time banging my head on this and want to use data models for acceleration.

0 Karma
1 Solution

tskinnerivsec
Contributor

A base search is usually just that, very basic "sourcetype = . . . " I've used a number of Data Models that have duplicate values in their fields, take for instance the Email data model, depending on how many sourcetypes are feeding it and from where, you can easily have duplicate messageid vailues. I have never needed to solve this problem within the data model, I've always been able to dedup after the data model search. For example:

| tstats summariesonly=t allow_old_summaries=t sum(All_Email.size) as "EmailMessageSize", from datamodel=Email where All_Email.user = $user$ groupby All_Email.user _time All_Email.subject All_Email.message_id All_Email.recipient span=1s | search All_Email.recipient != *foo* | dedup All_Email.message_id | . . .

View solution in original post

tskinnerivsec
Contributor

A base search is usually just that, very basic "sourcetype = . . . " I've used a number of Data Models that have duplicate values in their fields, take for instance the Email data model, depending on how many sourcetypes are feeding it and from where, you can easily have duplicate messageid vailues. I have never needed to solve this problem within the data model, I've always been able to dedup after the data model search. For example:

| tstats summariesonly=t allow_old_summaries=t sum(All_Email.size) as "EmailMessageSize", from datamodel=Email where All_Email.user = $user$ groupby All_Email.user _time All_Email.subject All_Email.message_id All_Email.recipient span=1s | search All_Email.recipient != *foo* | dedup All_Email.message_id | . . .

bwindham
Path Finder

Thanks tskinnerivesec....I'll go that route.

0 Karma
Get Updates on the Splunk Community!

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...