Deployment Architecture
Highlighted

failover cluster - splunk re-index when cluster has switched node

Explorer

I use Splunk 4.2 and have 2 node A and B on Cluster and logs has compression in zip files

  1. A node Active
  2. SplunkForwarder on A forward all logs to indexer
  3. switch node to B
  4. SplunkForwarder on B forward duplicate logs again.

i don't want duplicate logs from B,
how can I do for this problem? or it is bug of 4.2

thank you.

Tags (2)
0 Karma
Highlighted

Re: failover cluster - splunk re-index when cluster has switched node

Motivator

I'd need more details on what's involved in your Cluster switch over. Is the log directory moved from being available only on node A, to only being available on node B.

If this is the case, it's not a bug, as after the switch to node B all these files would be new to SplunkForwarder on B. All of the data that keeps track of what logs have been indexed would be stuck over on node A.

Would it be possible to have the log rotated as part of the cluster switch? Then have Splunk only look at the active file? Again, difficult to give specifics without more knowledge of your cluster configuration.

0 Karma
Highlighted

Re: failover cluster - splunk re-index when cluster has switched node

Explorer

Thank you for your answer,
I keep splunk_db and all logs on san box.
In my test case, I try to use this cluster to forward .log(not compress) files and the result is work well.

More info.

Splunk forwarder has been install on each node.
San box will move to available on B after switched.
in the same case, no problem with normal log (file isnot compression)

0 Karma
Highlighted

Re: failover cluster - splunk re-index when cluster has switched node

Explorer

I still have this issue.
Thank you.

0 Karma
Highlighted

Re: failover cluster - splunk re-index when cluster has switched node

Path Finder

As long as Splunk starts before the application(s) that generate the logs to be indexed you could add:

followTail = true

to the source definitions in inputs.conf. This would get Splunk to only look at data added after Splunk has started to monitor the logs. The danger is that if the application starts before Splunk has started to monitor the logs you may miss log entries.

Highlighted

Re: failover cluster - splunk re-index when cluster has switched node

Explorer

that is a good suggestion for me, i will try and update on this post in soon.
but, I have some question about this.

why it only have problem with compression file on cluster?

Thank you again, for this answer.

0 Karma
Highlighted

Re: failover cluster - splunk re-index when cluster has switched node

Explorer

i think, followTail = [0|1] is a good tip.
in this case. i can try to use ignoreOlderThan param on B for help to read log file by ignore the old files.

but, i still want to know the answer of my main question.
i wish to see SplunkForwarder working completely with compression files same as normal log file.

so, if you have any comments for my main question please let me know.

Thank you.

0 Karma
Highlighted

Re: failover cluster - splunk re-index when cluster has switched node

Explorer

ignoreOlderThan=2d is a good solution for short term workaround. We can set the parameter on both node and apply on input.conf. Anyway, we are waiting for long term solution from Splunk Support team.