topic Re: Can Index on Satellite stay In sync with Index on Planet Earth? in Getting Data In

Can Index on Satellite stay In sync with Index on Planet Earth?

mzorzi — Fri, 16 Jul 2010 23:05:47 GMT

I would like to deploy Splunk in a non reliable network.
I have an Index on a Satellite which is indexing events locally and also sending them to the main index on Planet Earth. Once in a while, at irregular times, the communication goes down. Is there any way to keep the two indexes synchronized?

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

Lowell — Sat, 17 Jul 2010 00:26:56 GMT

That is a truly awesome scenario. (Not that your communication goes down, but that your using splunk in space!) Any chance that your talking about that at the splunk user conference in August?

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

ftk — Sat, 17 Jul 2010 00:45:31 GMT

If you are using a forwarder (with local indexing or without), if the link to the receiving server goes down it will resume forwarding where it left off once the link comes back up. I do not know if Splunk accounts for bad bits due to solar storms though...

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

Chris_R_ — Sat, 17 Jul 2010 00:47:25 GMT

TZ=Earth/Orbit
:D

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

mzorzi — Sat, 17 Jul 2010 01:39:58 GMT

but during the time it lose visibility, the forwarders queues get blocked, which means the satellite will stop indexing and missing precious data?

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

ftk — Sat, 17 Jul 2010 02:13:42 GMT

It is my understanding that even if the forwarder queue is blocked, the local indexing continues.

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

jrodman — Sat, 17 Jul 2010 03:49:00 GMT

ftk: it depends upon the forwarding configuration. You can decouple them, but by default they are coupled. Meaning a forwarding blockage will eventually halt local indexing, so a complete record will get to both places eventually.

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

ftk — Sun, 18 Jul 2010 02:22:53 GMT

I'm wondering - why are you indexing locally on the satellite rather than just using the forwarder to the main Indexer? Also interesting to know would be what kind of inputs you are monitoring, with this info it would be easier to suggest a solid configuration for you.

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

adamw — Sun, 09 Sep 2012 22:57:20 GMT

I hate to be that guy who revives old threads, but there is a talk at Splunk .Conf 2012 regarding this issue being potentially addressed in Splunk 5.

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

DamianS — Fri, 30 Nov 2012 12:56:35 GMT

Be aware that due to relativistic effects there are sync issues between Satellites and the surface of the Earth if you care about small time internals. This is a common issue with writing software for Satellites though, so well researched and documented.

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

MHibbin — Fri, 30 Nov 2012 13:40:52 GMT

@DamianS is referring to:

http://en.wikipedia.org/wiki/Gravitational_time_dilation

and

http://en.wikipedia.org/wiki/Global_Positioning_System#Timekeeping

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

lukejadamec — Thu, 29 Aug 2013 01:47:08 GMT

Notwithstanding all other potential problems with space travel, if you have an indexer in orbit, on Mars, or across town, and you want to keep that indexer in-sync with an indexer in your office then you have two problems.

1) Adjusting for the difference in relativity (e=mc^2) between objects in motion.

2) Protecting against downtime of the network connection.

The solution to problem 1) Wait until NTP moves to a 128 bit time representation, because the current 64 bit representation cannot account for the difference in velocity between your local office and that of a satellite or Mars (the across town delta is ~0).

The solution to problem 2) Splunk has a config for that - Use useACK=true with a maxQueueSize=500GB. If you can afford a satellite or base on Mars, then you can afford some extra storage.

By the way, if you need some off-world on-site support let me know.

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

stephane_cyrill — Tue, 07 Apr 2015 10:25:44 GMT

Taking in account the previous answers, I think that summary indexing can be helpful to solve the problem of late arriving data when the the communication goes down.

Re: Can Index on Satellite stay In sync with Index on Planet Earth?

morethanyell — Mon, 02 Mar 2020 04:30:54 GMT

If your logs from outerspace use atomic clock that is programmed to represent the current UTC, taking into account the effects of Einstein's generalrelativity (curvature of spacetime), then there shouldn't be any problem.

The problem you will have to deal with is the delay in _indextime. See, your log's _time is the information your search timepicker will look after.

Say your spaceship has a log at 2020-03-01T00:00:00 about a malfunction of one of its thingamagjiggies. Your search head that is on Earth has an alert that looks after "malfunctions on one of the thingamajiggies" every 5 minutes. It will cause an issue if the log arrives on your Earth-bound indexer 6 minutes late. Because at 2020-03-01T00:00:00 and at 2020-03-01T00:00:05, the alert wouldn't be able to detect the log (because it's not ingested yet). And at 2020-03-01T00:00:10, the alert would've already missed it. There are however tweaks that can solve these types of issues.

In short, having a Splunk forwarder from outerspace wouldn't cause much headache as long as the network is alright (heck, I've seen NASA livestreams from low-orbit space, forwarding logs wouldn't be that hard).