Splunk Search

Merge the data on index

simo
Explorer

Hi All,

I'm in this situation

index a index b
id neme idneme
1simone 1simone
3francesco 2marco
4luca   

 

I have a scheduled search that extracts data from index a and writes it to index b. as you can see in both index a and b there is id = 1. my search currently duplicates the result this way

index b
idneme
1simone
2marco
1simone
3francesco
4luca

 

is there a function merge type sql? the expected result is the following

index b
idneme
1simone
2marco
3francesco
4luca

 

can you help me? thanks for any answer

Best Regards,

Simone

Labels (1)
Tags (2)
0 Karma

aasabatini
Motivator

Ciao Simo,

 

intendi unire i due indici?

 

in splunk puoi utilizzare la funzione OR

es: index a OR index b

oppure utilizzare le funzioni join come in SQL ma perdono di performance

Saluti

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

simo
Explorer

ciao,

no io vorrei scrivere all interno dell indice B solo le righe dell indicie A che non sono preseti in B.

avevo pensato di escludere gli id presenti in  B ma su miliardi di righe ci mettre troppo per farlo 

Simone

0 Karma

aasabatini
Motivator

Ciao @simo 

la search per collezionare i log dentro l'indice B dovrebbe essere così:

the search for collecting logs inside index B should look like this:

 

index=a OR idex=b | stats dc(index) AS dc_index values(index) AS index BY name, ID
| eval Status=if(dc_index=2,"Both",index)
| search NOT status="a"
|table name ID
| collect index=b

 

 

 

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

gcusello
Legend

Hi guys, Ciao a tutti

please write in english for the other people of Community!

To have a complete situation, you could run something like this.

index=indexA OR index=indexB
| stats dc(index) AS dc_index values(index) AS index BY name
| eval Status=if(dc_index=2,"Both Indexes",index)
| table name Status

then you can choose the name to take.

It's not clear for me if you want the id from IndexA or if you want to generate a new id.

Ciao.

Giuseppe

P.S.: there's an Italian Splunk User Group that you can find at https://community.splunk.com/t5/Roma-User-Group/gh-p/ug-roma

0 Karma

simo
Explorer

hi @gcusello ,

I would like to write on index B only the new ids present on index A. it's possible?

Simone

0 Karma

gcusello
Legend

Hi @simo,

let me understand:

  • the object you're calling index B is a summary index or a lookup?
  • you spoke about IDs, where they come from: index A (but they could be replicated) or it's generated at report level?

If it's a summary index, you should run something like this:

index=indexB NOT [ search index=indexB | dedup neme | fields neme ]
| table id neme
| collect index=indexB

If indexB it's a lookup, try something like this:

index=indexB NOT [ | inputlookup indexB | dedup neme | fields neme ]
| table id neme
| outputlookup indexB append=true

in both the solutions there's the problem of ID that you could manage at report level.

Ciao.

Giuseppe

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!