Getting Data In
Highlighted

How to combine different types of events from different data sources into one event?

Path Finder

Hello

I have three different data sources (so 3 different types of events)

DataSource1: EventNumber Ticket
DataSource2: EventNumber CreateEventDate
DataSource3: Ticket CreateTicket_Date

I would like to get one event which has: Ticket CreateTicketdate CreateEventDate

Is it possible to build a search which shows this as one event, taking also into account possible new completely different data sources (e.g. DataSource_4)?

I was trying transaction (not efficient), lookups (which I created from two different sources and use inputlookup populated to the 3rd one. In the search, there was some specification to those 3 data sources. Doing a lot of reports I would always need to take into account this part.

Thank you

0 Karma
Highlighted

Re: How to combine different types of events from different data sources into one event?

Esteemed Legend

What correlation fields do you have to link events between sources?

0 Karma
Highlighted

Re: How to combine different types of events from different data sources into one event?

Path Finder

Correlation is by the same column name

So DataSouce1 EventNumber = DataSource2 EventNumber
DataSource1 Ticket= DataSource3 Ticket

0 Karma
Highlighted

Re: How to combine different types of events from different data sources into one event?

SplunkTrust
SplunkTrust

You didn't mention how you tried to use transaction. I admit it's not the most efficient thing, but I've used it on fairly large datasets well if you can limit the time and events it's operating on.

... | transaction Event_Number Ticket maxspan=15m maxpause=15m maxevents=3

If speed is still a problem, you could create a Data Model containing that information and accelerate it.

That does assume Event_Number and Ticket aren't equal to one another within a 15 minute period, and obviously assuming they get generated within 15 minutes of each other. Adjust as necessary.

0 Karma
Highlighted

Re: How to combine different types of events from different data sources into one event?

SplunkTrust
SplunkTrust

just a funny thought; who about this:

 base search here | eval corr_field= coalesce(Event_Number, Ticket) | stats values(*) AS * by corr_field
0 Karma
Highlighted

Re: How to combine different types of events from different data sources into one event?

Explorer

I would like to use the below transaction in Data Model
| transaction maxevents=2 keeporphans=true
what is the best way to do this?

0 Karma
Highlighted

Re: How to combine different types of events from different data sources into one event?

Esteemed Legend

Avoid transaction like the plague that it is. It should only be used for transitive key mapping (e.g. some events have EmployeeID, others have Address, others have loginID and each of this is fully unique to a single individual. Then the best way to link is to use | transaction EmployeeID Address loginID. Otherwise do yourself a favor and do not ever use it; it does not scale.

0 Karma
Highlighted

Re: How to combine different types of events from different data sources into one event?

Explorer

Sure, thanks Woodcock.

0 Karma
Highlighted

Re: How to combine different types of events from different data sources into one event?

Esteemed Legend

Try this:

... | eventstats values(Ticket) AS Ticket BY Event_Number
| eventstats values(Event_Number) AS Event_Number BY Ticket
| stats values(*) AS * by Ticket

View solution in original post

0 Karma