Deployment client is not indexing data to the Deployment server? (50 credit will be rewarder for the person who found the solution)

[pavanae]

Hi the following were the splunkd.log messages in the deployment client. I don't know why it isn't showing any warnings or errors and also it didn't indexing anything. But you can see that it took too long to write the second and third log file. Never experienced this before.

11-06-2015 20:08:12.187 -0500 INFO  HttpPubSubConnection - SSL connection with id: connection_10.200.160.21_8089_svcldprdsea01.aeo.ae.com_svcldprdsea01.ae.com_3B3FD84B-BB72-460F-81D9-41DC7F97EA09

11-07-2015 04:26:19.118 -0500 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.

11-07-2015 04:26:19.155 -0500 INFO  WatchedFile - Will begin reading at offset=24999200 for file='/opt/splunkforwarder/var/log/splunk/metrics.log.1'.file='/opt/splunkforwarder/var/log/splunk/metrics.log.1'.

Following is my inputs.conf

[monitor:///opt/endeca/apps/ab/logs/dgraphs/DgraphA1.log]
index=search
crcSalt = <SOURCE>
sourcetype=log4j

[monitor:///opt/endeca/apps/ab/logs/dgraphs/DgraphA1.reqlog]
index=search
crcSalt = <SOURCE>
sourcetype=log4j

[monitor:///opt/endeca/apps/ab/logs/dgraphs/DgraphA1.start.log]
index=search
crcSalt = <SOURCE>
sourcetype=log4j

[monitor:///opt/endeca/apps/ab/logs/dgraphs/DgraphA1.updatelog]
index=search
crcSalt = <SOURCE>
sourcetype=log4j

[monitor:///opt/endeca/apps/ab/logs/provisioned_scripts/AEBaselineUpdate.log]
index=search
crcSalt = <SOURCE>
sourcetype=log4j

Not sure why I haven't seen any logs flowing into Splunk. Please suggest why it's not happening.

[wrangler2x]

I'm assuming you are using deployment apps and serving them with the deployment server to the deployment client (forwarder). If so, take a look on the forwarder for the deployment app(s), and verify they are what you expect.

Say, for example, that the deployment app that you are storing the input.conf files (listed in your question) is called MyDeploymentApp (and it is in $SPLUNK_HOME/etc/deployment-apps on the deployment server/indexer). Take a look in $SPLUNK_HOME/etc/apps on the forwarder and you should see a MyDeploymentApp directory (or whatever your deployment app is actually called). If so, look in the default subdirectory at inputs.conf and make sure what is there is what you expect to be there (that is, the input.conf file on the deployment server). If it is there, make sure that as user splunk you can access the files on that forwarder you are trying to monitor. If it is not there, then grep for the IP address of the forwarder in the splunkd.log file on the deployment server and look for any trouble there. Also, make sure that the deployment app name is associated with a serverClass in the deployment server's $SPLUNK_HOME/etc/system/local/serverclass.conf, and that the forwarder's host name is associated with the serverClass as well.

You should be making changes to the serverclass.conf file using forwarder management so it will be also in the live configuration. If you are doing them manually then you'll need to restart splunkd if you make new changes.

You should also be able to see the deployment app bundle on the deployment server in $SPLUNK_HOME/var/run/tmp/MyserverClass (or whatever your serverClass is called) directory as a .bundle file. This is a tar file created by the deployment server when these conditions exist on the deployment server:

1. The serverClass is declared in ~/etc/system/local/serverclass.conf
2. The deploymentApp is associated with the serverClass in ~/etc/system/local/serverclass.conf
3. The deploymentApp directory exists in ~/etc/deployment-apps and is properly populated
4. The serverclass.conf file is in the live configuration

Whenever you make changes to the deploymentApp, they should be sent to the forwarder in due time, but you can push it up a bit using splunk reload deploy-server -class MyserverClass (or whatever your serverClass is called).

[benafo]

Did you setup server class? If yes, you need to add your clients to the server class and deploy the app ( output.conf.) to all the clients in that server class. Also check the "sentoindexer" configuration.

[MuS · Accepted solution]

Hi pavanae,

there are plenty of things to check:

• On the forwarder, check with this command /opt/splunkforwarder/bin/splunk cmd btool --debug inputs list | grep -vi default as Splunk user and check if your inputs.conf is applied.
• On the forwarder, can the user running Splunk read the files?
• On the forwarder, check /opt/splunkforwarder/var/log/splunk/splunkd.log for any entries related to the files being monitored.
• On the forwarder, check outputs.conf for any typos.
• On the forwarder, check with this command /opt/splunkforwarder/bin/splunk show forward-server if the indexer is listed as Active forwards:
• On the indexer, check if the configured index search does exists.
• On the indexer, run this search index=search earliest=0.
• On the indexer, search index=_internal for connection from the forwarder.

There are other tips and hints here http://docs.splunk.com/Documentation/Splunk/6.3.0/Troubleshooting/Cantfinddata#Are_you_using_forward...

Hope this helps ...

cheers, MuS

[MuS]

So, did my answer solve your problem?
In this case I will give you a gentle reminder on the topic of this question 😉

[gwobben]

Not sure what you mean by "deployment client" and "deployment server". Usually you try to send data from "Forwarders" to "Indexers", not to deployment servers. If you want to send data from one Splunk instance to another you might want to try the outputs.conf file:
http://docs.splunk.com/Documentation/Splunk/6.3.1/Forwarding/Configureforwarderswithoutputs.confd

[pavanae]

I am using a stand alone environment in which deployment server and indexer and search are the same server. I mean Deployment client means Forwarder server and deployment server means indexer. And I was successfully configured and forwarded and data for tha other indexes but i am not able to done with this forwarder servers. I'm not sure what went wrong. Paths are correct, Ports are working no network issue.

[esix_splunk]

Provide your outputs.conf and deploymentclient.conf from the client.