Getting Data In

How do I edit my inputs.conf to monitor logs from two servers (webserver) with the same location?

pranav_agile
Explorer

Hi,

I am trying to get logs from two different servers running Tomcat application, but have the same location. The following is the inputs.conf

[default]
host = $decideOnStartup

[monitor://C:\Tomcat\logs\]
disabled=false
sourcetype=access_combined_wcookie
index=globalscreening
whitelist=localhost_access_log.*txt$


[monitor://C:\Tomcat\logs\app-errors.log]
sourcetype=catalina
index=globalscreening

[monitor://C:\Tomcat\logs\app-events.log]
sourcetype=catalina
index=globalscreening

[monitor://C:\Tomcat\logs\ws28.log]
sourcetype=catalina
index=globalscreening

The issue is I am getting logs into Splunk from server A, but not from server B. I have tried to include host information into the above stanza, but the logs stopped ingesting into Splunk.

Could you please let me know where I am going wrong and how to correct it?

thank you.

Tags (2)
0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

[Edit: fixed btool command]

From your responses above it seems your inputs probably work correctly. FritzWittwer's advice is good, a quick check of /opt/splunkuniversalforwarder/var/log/splunk/splunkd.log may tell you something. Obviously, confirm the paths I gave there.

If the log file doesn't quickly pinpoint the issue, check that outputs are correctly specified. The two systems should both have pretty much the same outputs.conf, I'd guess. (In my case, pretty much ALL my UFs have the same outputs.conf file).

If outputs.conf appears right as well, you might want to run btools in debug mode to see if you can spot the problem. On each UF, you'll run something like

/opt/splunkuniversalforwarder/bin/splunk cmd btool --debug outputs list > myoutputs.txt

and possibly

/opt/splunkuniversalforwarder/bin/splunk cmd btool --debug inputs list >myinputs.txt

(More btool info here) Then review those two files. It'll take a little searching around (at least in the inputs, the outputs is probably pretty small), but compare the two and I suspect you'll find your problem.

View solution in original post

Richfez
SplunkTrust
SplunkTrust

[Edit: fixed btool command]

From your responses above it seems your inputs probably work correctly. FritzWittwer's advice is good, a quick check of /opt/splunkuniversalforwarder/var/log/splunk/splunkd.log may tell you something. Obviously, confirm the paths I gave there.

If the log file doesn't quickly pinpoint the issue, check that outputs are correctly specified. The two systems should both have pretty much the same outputs.conf, I'd guess. (In my case, pretty much ALL my UFs have the same outputs.conf file).

If outputs.conf appears right as well, you might want to run btools in debug mode to see if you can spot the problem. On each UF, you'll run something like

/opt/splunkuniversalforwarder/bin/splunk cmd btool --debug outputs list > myoutputs.txt

and possibly

/opt/splunkuniversalforwarder/bin/splunk cmd btool --debug inputs list >myinputs.txt

(More btool info here) Then review those two files. It'll take a little searching around (at least in the inputs, the outputs is probably pretty small), but compare the two and I suspect you'll find your problem.

pranav_agile
Explorer

thank you rich7177 and everyone here. It has truly helped me to nail down the issue and logs have started ingesting into Splunk.

Yes I did compare the inputs/outputs debug logs from the two servers and found that wb030 wasn't added to one of the App that forwards the indexers config to it.

I added the host and could see the logs started ingesting into splunk.

FritzWittwer
Contributor

did you check the log file in $SPLUNK_HOME/var/log/splunk/splunkd.log ?

0 Karma

meenal901
Communicator

Is $decideOnStartup being populated on both hosts properly. Can you try the following entry as default:
host-1
[default]
host = wb030

host-2
[default]
host = wb031 --> OTHER HOST NAME

These input.conf are present in $SPLUNK_HOME/etc/system/local or app level?

Check in splunkd.log in case there is any error while reading the file? May be path is not exactly same or there is read permission issue?

pranav_agile
Explorer

I have updated inputs.conf as, but still its not getting data from wb030:
[default]
host = $decideOnStartup

[monitor://C:\Tomcat\logs]
disabled=false
sourcetype=access_combined_wcookie
index=globalscreening
whitelist=localhost_access_log.*txt$

[monitor://C:\Tomcat\logs\app-errors.log]
disabled=false
sourcetype=catalina
host=wb030
index=globalscreening

[monitor://C:\Tomcat\logs\app-events.log]
disabled=false
sourcetype=catalina
host=wb030
index=globalscreening

[monitor://C:\Tomcat\logs\ws28.log]
disabled=false
sourcetype=catalina
host=wb030
index=globalscreening

0 Karma

pranav_agile
Explorer

wb030 is the second web server. I would like to confirm that wb030 does have the data files in the C:\Tomcat\logs.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You mentioned you've two tomcat servers. Do you have Splunk forwarder installed on both the Tomcat servers and have deployed this inputs.conf on both the forwarders?

0 Karma

pranav_agile
Explorer

yes, splunk forwarders is installed on both the servers. inputs.conf is also deployed on each of the servers.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...