Knowledge Management

Forwarding summary index from search head to Indexer

Path Finder

I have a search head and two indexers name Indx_01 and Indx_02. I have creaetd a summary index in the search head and wanted to forward that index to both the indexers. So I have changed the outputs.conf file in the srch head and restarted splunk in the search head.
Now for Indx_02 the summary inedex is working fine but Indx_01 it is not working.
Have I missed something?

Tags (1)
1 Solution

Path Finder

You outputs.conf does not look right to me.

There are 3 levels of config you can apply in outputs.conf.

Global, Target Group and Single Server.

You only have a global stanza. The setting defaultGroup defines the "name" of the default "Target Group", which you have not yet defined.

Try this:

# GLOBAL CONFIG
[tcpout]
# Name of the default Target Group
defaultGroup = loadBalancedGroup
disabled = 0

# TARGET GROUP CONFIG
# Define a target group called "loadBalancedGroup"
[tcpout:loadBalancedGroup]
# Specify members of new target group
server=10.28.204.212:9997, 10.28.204.213:9997

# SINGLE SERVER CONFIG
[tcpout-server://10.28.204.212:9997]

# SINGLE SERVER CONFIG
[tcpout-server://10.28.204.213:9997]

View solution in original post

Path Finder

The Delete command in search does not actually delete any data it marks it as deleted so that it no longer shows up in searches. That is one option if you do not care about recovering space. You need to explicitly grant the can_delete permission for your role.

If you want to completely clean the index, then why not bring one indexer down at a time and use the clean command, i.e. splunk clean eventdata -index nameOfSummaryIndex

0 Karma

Path Finder

I have one more question. When I am trying to push the indexer data from srch head to 2 indexers. None of the time the data got pushed in the same time. There is always a delay of 2-3 hours. Like srch hed started pushing data to Indx_01 at 10:00 and for Indx_02 it is 11:30.. is there any cause behind that?

0 Karma

Path Finder

Now the data is getting pushed from the search head to Indexers. But why the event count is still 0 for the summary indexer in the search head?

0 Karma

Path Finder

I have deleted the summary Index from srch hd, Indx1 and Indx2 executing the command
$SPLUNK_HOME/bin/splunk clean eventdata -index summary

Then recareted the summary in the srch head with different search condition. Now the Event count is not increased from 0 where as I can see many events under that search.
Also the data is not getting pushed to the Indexers.
Need HELP!!!

0 Karma

Path Finder

Great I'm glad to hear it is working now.

Those two lines are actually doing nothing right now but I included them for completeness. They would be used if you wished to have different config per individual connection to each server. At the moment, using the config I posted above, you are using a Target Group to define your settings for the whole target group.

And yes it will be load-balancing data across both servers in the Target Group.

0 Karma

Path Finder

Thanks for you answer. I have a query.
Suppose I want to delete the summary Index from the search head and dont want to stop data flowing to the Indexers what should I do?
I beleive deleting Summary Index from Search will not help...rght??
Can I delete the data from the Indexers that is alreay pushed?

0 Karma

Path Finder

You have checked that there are ESTABLISHED TCP connections from the SH to both indexers on port 9997.
You have checked the splunkd.log on the SH for clues.
You have changed outputs.conf to be more specific.

  1. Is the summary index defined on both indexers?
  2. Are there clues in the splunkd.log on indexer1 to suggest problems?
  3. If you configure outputs.conf on the SH to only send to indexer1, does this now work?
  4. Have you ever got any data to go to indexer1 on port 9997?
  5. Is there frequent summary data being written and have you left it long enough to be sure nothing is going to Indexer1?
0 Karma

Path Finder

Now it worked fine..
The problem is with the search string there I have made it search for last 4 hours and there was no data from the last four hours...so it was not indexing...
But can you tell me one more thing?

SINGLE SERVER CONFIG

[tcpout-server://10.28.204.212:9997]

SINGLE SERVER CONFIG

[tcpout-server://10.28.204.213:9997]

is it duplicating the data to both the IPs or sending some of the data to IP1 and other to IP2?

0 Karma

Path Finder

No, It does not work. Even though I have gvn the port for 10.28.204.212..the data is still going to 10.28.204.213.

Now to check it?????

I have created and search and saved the same with a schedule to run in every 15 mins. SO I think there should be some data to be wriiten under summart. I have Tried the same from last friday...so it should be a sufficient time..

0 Karma

Path Finder

Fine my answers below:
Yes. It is defined in both the indexers

I have the below mentioned error in the splunkd.log file under Indx1
12-17-2012 10:50:28.892 -0500 ERROR HTTPClient - Cannot find host "splunkbase.splunk.com": Name or service not known
12-17-2012 10:50:28.892 -0500 ERROR ApplicationUpdater - Error checking for update via https://splunkbase.splunk.com/api/apps:resolve/checkforupgrade: Invalid URI

0 Karma

Path Finder

You outputs.conf does not look right to me.

There are 3 levels of config you can apply in outputs.conf.

Global, Target Group and Single Server.

You only have a global stanza. The setting defaultGroup defines the "name" of the default "Target Group", which you have not yet defined.

Try this:

# GLOBAL CONFIG
[tcpout]
# Name of the default Target Group
defaultGroup = loadBalancedGroup
disabled = 0

# TARGET GROUP CONFIG
# Define a target group called "loadBalancedGroup"
[tcpout:loadBalancedGroup]
# Specify members of new target group
server=10.28.204.212:9997, 10.28.204.213:9997

# SINGLE SERVER CONFIG
[tcpout-server://10.28.204.212:9997]

# SINGLE SERVER CONFIG
[tcpout-server://10.28.204.213:9997]

View solution in original post

Path Finder

I have changed the outputs.conf file as per you.
But then also it is not working for 10.28.204.212.

But 10.28.204.213 it is working.

Not able to find the problem

0 Karma

Path Finder

Could you paste in your outputs.conf from the SH?

Are there errors in "\SPLUNK_HOME/var/log/splunk/splunkd.log" on the SH that give a clue as to why there is no forwarding to Indx_01?

Assuming you are forwarding to port 9997 on both indexers, does the following command,

Linux:

netstat -na | grep 9997

Windows:

netstat -na | findstr "9997"

, show ESTABLISHED connections to both indexers?

0 Karma

Path Finder

Find below the outputs.conf file content for the search head:

[tcpout]
defaultGroup = 10.28.204.212:9997, 10.28.204.213:9997
disabled = 0

netstat -na | grep 9997 out put is also OK for both the indexers.

i did not get any clue in the splunkd.log file.
Plesae help me out!!!

0 Karma