Deployment Architecture

How to forward from stand-alone server to an index cluster

mykol_j
Path Finder

Sounds easy, eh? I've been using Splunk since v3 -- and I've setup forwarding for servers dozens of times, and migrated countless indexes, but this one is kicking my butt.

I have a stand-alone Splunk server (Enterprise) that's been ingesting data for years in the form of CSV files and providing a front end for analysts. I need to decommission that box and get the data into our main cluster. I setup forwarding from the stand-alone server to feed into a heavy forwarder (that has a thousand other hosts feeding into it) and then into the cluster. It's working insomuch as it forwarded data but only from the last CSV file (back to March 17th, FWIW). I can't simply copy the files into a new index because of the cluster, and I no longer have the previous CSV files to re-ingest (going back to 2009). I've tried clearing the fishbucket hoping to force it to resend everything it knows. It's feeding into an index of the same name. No errors in splunkd.log...

Thoughts?

Thanks!

Michael

Labels (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @mykol_j,

forwarding isn't retroactive.

the only way is extract all data in row format and indexing them again.

It isn't an easy job because you have to take host, index and sourcetype informations from the logs, you can do this manually or with a not immediate override process using regexes or a script.

The only alternative is to copy these indexes in one Indexer, modify eventtypes taking both ond ald new indexes and waiting that the data from the old Indexer will be out of the retention period.

You cannot add the old Indexer to the cluster because also cluster replication isn't retroactive.

Ciao.

Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @mykol_j,

forwarding isn't retroactive.

the only way is extract all data in row format and indexing them again.

It isn't an easy job because you have to take host, index and sourcetype informations from the logs, you can do this manually or with a not immediate override process using regexes or a script.

The only alternative is to copy these indexes in one Indexer, modify eventtypes taking both ond ald new indexes and waiting that the data from the old Indexer will be out of the retention period.

You cannot add the old Indexer to the cluster because also cluster replication isn't retroactive.

Ciao.

Giuseppe

0 Karma

mykol_j
Path Finder

Grazie Giuseppe,

Not exactly what I wanted to hear, but at least it's an answer.

I guess I'll have to do a giant export, then an import -- ugh, not looking forward to that.

Cheers,

0 Karma

gcusello
SplunkTrust
SplunkTrust

hi @mykol_j.

I'm sorry!

In Italy we say "messenger brings no pain"!

Anyway, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...