Master server is not figuring in splunk_server list when searching in _internal

[mvagionakis]

Hello Splunkers,

I have a problem when I'm searching in _internal index from my master server.

My architecture consists a master server et four indexers.

When I search index=_internal on my master server I have results only from my indexers but not from the local server.
If I specify in my search the name of my local server index=_internal splunk_server=master then it works and I have all the results.

I discovered that when I tried to check my license usage for 30 days that was empty.
However, the "today" license usage works perfectly.

I have this problem since last week but I didn't change anything on my servers during this period.

I verified privileges and ownership in /opt/splunk but everything is ok.
My user is admin and can access all indexes, full access.
All my config files are consistent and no error found when I run the debug command splunk cmd btool check

Do you have any ideas ?

Thank you in advance
Michael

[harsmarvania57]

Are you forwarding Cluster Master logs to Indexers ? If not then it is recommended to send data from Search Head and CM to Indexers, please refer https://docs.splunk.com/Documentation/Splunk/7.0.1/DistSearch/Forwardsearchheaddata

[mvagionakis]

Hello Harsmarvania57 ,

Thank you for replying.
My CM already forwarding to the indexers.

thx

[harsmarvania57]

Is it ? I don't think so because when you use splunk_server in your query this means you are searching data from Indexers (Means from those servers on which data is actually stored). So in your case when you run index=_internal splunk_server=master it is displaying data means you are trying to search data from your Cluster Master only.

When you changed outputs.conf to send data from CM to IDX, after that have you restarted splunk?

[mvagionakis]

Hello Harsmarvania57 ,

thanks for your prompt reply.

Yes I did restart splunk service, as I told at the beginning, it was working until last week but non modification made between last good known configuration and today.

The good news are that I think that the problem is solved.
I inserted in my inputs.conf (in system/local) the following in order to force parsing the license_usage.log:

 [monitor://$SPLUNK_HOME/var/log/splunk/license_usage.log]
    index = _internal
    disabled = 0

That's I don't understand is why did it stop overnight?
And also, in the default inputs.conf it already parse all log files in

$SPLUNK_HOME/var/log/splunk

thanks once again
Michael

[harsmarvania57]

That's strange, if it stops again then run this command $SPLUNK_HOME/bin/splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus on CM , it will display all the files which splunk is reading with status and percent so that you can identify whether splunk is reading log files or not.

[mvagionakis]

Hello Harsmarvania57 ,

ok, I'll let you know if the problem comes back again.

Thank you very much.
Michael

[mvagionakis]

awesome 🙂

Thank you very much.