topic Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others? in Security

As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

withrythm — Wed, 12 Aug 2015 06:09:54 GMT

From a particular Server, we are able to see logs on Splunk from certain locations, but for others, there are no logs. If there is a permission or other such issues while accessing these logs, where would such errors be logged?

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

diogofgm — Wed, 12 Aug 2015 11:13:00 GMT

have you checked your user roles in both servers? you might have different permission for index access and (depending on the search you are doing) you might have different default indexes for search.

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

richgalloway — Wed, 12 Aug 2015 12:26:39 GMT

To be clear, do you want to see Splunk log (splunkd.log, etc.) or logs forwarded from monitored systems?

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

woodcock — Wed, 12 Aug 2015 16:45:01 GMT

It depends. Be as specific as you possibly can and maybe somebody here can help.

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

withrythm — Thu, 13 Aug 2015 00:37:49 GMT

Below is what is required to be available on SPLUNK:

Server 1- logs from Path 1
Server 1- logs from Path 2

The permissions for Path 1 and Path 2 are same. However, I am able to see logs from Path 1 but not from Path 2.

My question is, supposedly if there is some issue in Path 2 while Server 1 is trying to access the logs/files on those path; where are such error logs captured. Are these available on SPLUNK GUI anywhere? or on Server 1? or on SPLUNK Servers? and how to access such error logs.

Thanks in advance!

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

maciep — Thu, 13 Aug 2015 00:57:37 GMT

The Splunk logs from that sever should be in Splunk:

index=_internal host=Server1 sourcetype=splunkd

You can obviously sift through the results, maybe searching for "error" or "monitor" or "tailing" (don't recall off hand the best search to find these kinds of problems).

Did you ever get logs from Path 2 into Splunk? If not, it might be worth posting your inputs on here to see if something may be off there.

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

withrythm — Tue, 29 Sep 2020 06:58:03 GMT

Thanks Maciep for your response. No these are new paths added to the inputs.conf. below are the details from the input file. All of these are directories:

[monitor:///opt/app/weblogic/oracle/middleware/user_projects/domains/endeca_server_domain/servers/Managed-2/logs]
index = ftts
sourcetype = application_logs
disabled = 0
crcSalt =
followTail = 0

[monitor:///opt/app/weblogic/oracle/middleware/logs]
index = ftts
sourcetype = application_logs
disabled = 0
crcSalt =
followTail = 0

[monitor:///var/opt/app/endecaserver/logs]
index = ftts
sourcetype = application_logs
disabled = 0
crcSalt =
followTail = 0

[monitor:///var/opt/app/.snapshot/daily*]
index = ftts
sourcetype = snapshot_logs
disabled = 0
crcSalt =
followTail = 0

[monitor:///home/endeca/oraInventory/logs]
index = ftts
sourcetype = application_logs
disabled = 0
crcSalt =
followTail = 0

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

grijhwani — Thu, 13 Aug 2015 01:17:22 GMT

There are so many different reasons why logs might not be appearing, that no, a simple search access would not be the full route to diagnosis. It might supply the answers, if they are the sort of issues logged in the audit logs.

But it could simply be that the data is not being logged at all, and that could be down to any or all of misconfigured endpoint forwarder or indexer, SSL faults, broken network, local file permissions (all of which could be absent from the indexer because of the problem itself).

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

withrythm — Thu, 13 Aug 2015 01:31:49 GMT

Thanks Grijhwani

But since we are receiving logs from the same server from another path shouldnt it rule out SSL, broken network, miconfigured endpoint forwarder/indexer?

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

richgalloway — Thu, 13 Aug 2015 12:51:14 GMT

The forwarder's logs should be on your indexers. You can also find them on the forwarder itself at SPLUNK_HOME/var/log/splunk.

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

richgalloway — Thu, 13 Aug 2015 12:54:40 GMT

Which of these is not being forwarded?

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

woodcock — Thu, 13 Aug 2015 13:11:29 GMT

The main reason that a log is ignored is because of the problem described (and solved) here:

http://answers.splunk.com/answers/231959/why-are-files-in-a-monitored-directory-being-skipp.html

http://docs.splunk.com/Documentation/Splunk/6.2.5/Data/Howlogfilerotationishandled

The easiest thing to try is to add this to your inputs.conf stanzas for each input:

crcSalt = <SOURCE>

Make sure that you copy this exactly (do not change capitalization).

Re: As a Splunk front end user, would I be able to see why logs from certain locations on a server are available on Splunk but not from others?

maciep — Sat, 15 Aug 2015 20:48:36 GMT

Yep, it would be helpful to know which stanzas you're not getting logs from. Or if you're getting logs from all of the stanzas, but not every log you'd expect with a certain stanza if that makes sense.

As mentioned by woodcock, if any of those files have large headers that are the same, then that might explain it as well.