topic Post Processing Search as default in Splunk Enterprise Security

Post Processing Search as default

shayhibah — Thu, 24 May 2018 08:37:39 GMT

I have multiple logs with the same unique field.
for instance:

Time: 10:00:00 Log-id: 0x1212 Message: ABCD
Time: 10:00:11 Log-id: 0x5565 Message: EFG
Time: 12:00:00 Log-id: 0x1212 Message: HIJK
Time: 10:00:00 Log-id: 0x5565 Message: LMNO
Time: 16:05:00 Log-id: 0x1212 Message: PQR

I would like that for every query user does, before showing query results, to filter out the results and to return only the latest log (Time field) for each unique log (Log-id field).

I have many dashboards and search bars and I would like that this will happen automatically for each query.
Is it possible? If so, Can anyone help me with this?

Re: Post Processing Search as default

niketn — Thu, 24 May 2018 10:27:10 GMT

@shayhibah, if your raw events are already sorted in reverse chronological with Time order and fields are extracted including Log_id, you just need dedup based on Log_id field.

<yourBaseSearch>
| dedup Log_id

If they are not you would need to create field extraction and sort results in reverse chonological order by Time. Refer to the following example with run anywhere search based on your sample data.

|  makeresults
|  eval data="Time: 10:00:00 Log-id: 0x1212 Message: ABCD;Time: 10:00:11 Log-id: 0x5565 Message: EFG;Time: 12:00:00 Log-id: 0x1212 Message: HIJK;Time: 10:00:00 Log-id: 0x5565 Message: LMNO;Time: 16:05:00 Log-id: 0x1212 Message: PQR"
|  makemv data delim=";"
|  mvexpand data
|  rename data as _raw
|  rex "Time: (?<Time>[^\s]+)\sLog-id: (?<Log_id>[^\s]+)\sMessage: (?<Message>.*)"
|  sort - Time
|  dedup Time

Re: Post Processing Search as default

shayhibah — Thu, 24 May 2018 11:54:44 GMT

Hi niketnilay,

First - thank you for your response.

I do know the dedup command but - I want that it will happen automatically for each query or out-of-the-box dashboard.
I dont wanna change every dashboard query or tell the user to add it whenever he send query using search bar.

Is there any configuration file to define that each query, before return the results to the user, do "dedup" on the result?

Re: Post Processing Search as default

niketn — Thu, 24 May 2018 11:58:31 GMT

Instead of calling a base search you can create a macro and within the macro you can also have dedup command.

Re: Post Processing Search as default

shayhibah — Thu, 24 May 2018 12:56:41 GMT

May I ask for example? how to connect the macro to some sourcetype I've created?