How to merge two very high volume index into one?

[usernamejpblais]

Hello,

I have 2 index, one that received about 40 millions records per day and the other one about 80% of the first index.  I have 2 uniques fields in each index that allows me to merge the 2 index. 

Is it possible to merge the 2 index before ingestion?  Because if I do mentionne index A OR index B with an eval after it, it's working but I have to limit the period to not more than  a couple of hour or else it take a lottttt of time before getting the result. 

Plus if I want to limit the result by having a selection on other criteria, I need to merge them before aplying those criteria because the information is devided into the 2 index.

I have been trying to figure out this one for months now with a lot of trial and error that is why i'm giving a shot here.

Thanks!   

[fredclown]

It sounds like what you want to do is join two datasets based on a couple common field. Is that correct? There are a few commands in Splunk that let you do this and it depends upon your use case which you would use. Could you post example data (not real data) with the appropriate field names and an example of what you would like to see as the result?

[isoutamo]

Hi

what you are meaning with work merge? Do you want just put events to one index or do you want to merge events together based on those common fields?

Anyhow it's almost impossible to us said anything what you can immediately do, as we don't know what those indexes contains and how those are used. Also we should know what kind of access, retention period and cardinality you data have. Also we should know your splunk environment's architecture and size.

I suggest you to contact Splunk Support and use you AOD (admin on demand) credits to discuss with they and look data and other details to find best option. Another option is try to find some local splunk partner or other staff who can help you, but all those needs to see and understand your data to get good solution.

r. Ismo

[usernamejpblais]

Hello!

I want to merge events together based on those common fields.

We use Splunk Entreprise version 8.2.6.

Were a bank, our Splunk plateforme is enormous.  We want to keep data for the last 7 years.

In index A I have information about our transactions with metrics about the CPU time. elapse time, from were it was called etc... in index B I have information concerning the type and sub type of the transaction that is not present in index A.  I need to merged the 2 index to be able to produce some statistic about let say the sum of transaction A type B subtype C.  But merging 30 millions records with another 25 millions records before being able to sum the number of records is very long.    I was hopping to be able to merge them together before ingestion in the HF.

Thanks! 

[isoutamo]

If you want to merge those events before ingesting those into splunk, you need some other tools to do it!

I think that the biggest issue is to join those events on stream when they are coming? Also how you can handle the situation when event for index A is coming at time X and event for index B is coming at X+Y where Y > Z min? Basically you should spool those events somewhere before you could be sure that you could merge those. Probably the easiest way will be that your bank system will write those log events only on one log where those are already merged into one event.

I'm not sure, but maybe you could look Splunk DSP or Cribil to solve this? On Splunk Slack there is separate channel for Cribil where you could ask if this is doable with it.