Can Index on Satellite stay In sync with Index on Planet Earth?

[mzorzi]

I would like to deploy Splunk in a non reliable network.
I have an Index on a Satellite which is indexing events locally and also sending them to the main index on Planet Earth. Once in a while, at irregular times, the communication goes down. Is there any way to keep the two indexes synchronized?

[morethanyell]

If your logs from outerspace use atomic clock that is programmed to represent the current UTC, taking into account the effects of Einstein's generalrelativity (curvature of spacetime), then there shouldn't be any problem.

The problem you will have to deal with is the delay in _indextime. See, your log's _time is the information your search timepicker will look after.

Say your spaceship has a log at 2020-03-01T00:00:00 about a malfunction of one of its thingamagjiggies. Your search head that is on Earth has an alert that looks after "malfunctions on one of the thingamajiggies" every 5 minutes. It will cause an issue if the log arrives on your Earth-bound indexer 6 minutes late. Because at 2020-03-01T00:00:00 and at 2020-03-01T00:00:05, the alert wouldn't be able to detect the log (because it's not ingested yet). And at 2020-03-01T00:00:10, the alert would've already missed it. There are however tweaks that can solve these types of issues.

In short, having a Splunk forwarder from outerspace wouldn't cause much headache as long as the network is alright (heck, I've seen NASA livestreams from low-orbit space, forwarding logs wouldn't be that hard).

[stephane_cyrill]

Taking in account the previous answers, I think that summary indexing can be helpful to solve the problem of late arriving data when the the communication goes down.

[lukejadamec]

Notwithstanding all other potential problems with space travel, if you have an indexer in orbit, on Mars, or across town, and you want to keep that indexer in-sync with an indexer in your office then you have two problems.

1) Adjusting for the difference in relativity (e=mc^2) between objects in motion.

2) Protecting against downtime of the network connection.

The solution to problem 1) Wait until NTP moves to a 128 bit time representation, because the current 64 bit representation cannot account for the difference in velocity between your local office and that of a satellite or Mars (the across town delta is ~0).

The solution to problem 2) Splunk has a config for that - Use useACK=true with a maxQueueSize=500GB. If you can afford a satellite or base on Mars, then you can afford some extra storage.

By the way, if you need some off-world on-site support let me know.

[MHibbin]

@DamianS is referring to:

http://en.wikipedia.org/wiki/Gravitational_time_dilation

and

http://en.wikipedia.org/wiki/Global_Positioning_System#Timekeeping

[DamianS]

Be aware that due to relativistic effects there are sync issues between Satellites and the surface of the Earth if you care about small time internals. This is a common issue with writing software for Satellites though, so well researched and documented.

[adamw]

I hate to be that guy who revives old threads, but there is a talk at Splunk .Conf 2012 regarding this issue being potentially addressed in Splunk 5.

[ftk]

I'm wondering - why are you indexing locally on the satellite rather than just using the forwarder to the main Indexer? Also interesting to know would be what kind of inputs you are monitoring, with this info it would be easier to suggest a solid configuration for you.

[Chris_R_]

TZ=Earth/Orbit
:D

[ftk]

If you are using a forwarder (with local indexing or without), if the link to the receiving server goes down it will resume forwarding where it left off once the link comes back up. I do not know if Splunk accounts for bad bits due to solar storms though...

[jrodman]

ftk: it depends upon the forwarding configuration. You can decouple them, but by default they are coupled. Meaning a forwarding blockage will eventually halt local indexing, so a complete record will get to both places eventually.

[ftk]

It is my understanding that even if the forwarder queue is blocked, the local indexing continues.

[mzorzi]

but during the time it lose visibility, the forwarders queues get blocked, which means the satellite will stop indexing and missing precious data?

[Lowell]

That is a truly awesome scenario. (Not that your communication goes down, but that your using splunk in space!) Any chance that your talking about that at the splunk user conference in August?