I have a problem when I'm searching in _internal index from my master server.
My architecture consists a master server et four indexers.
When I search index=_internal on my master server I have results only from my indexers but not from the local server.
If I specify in my search the name of my local server index=_internal splunk_server=master then it works and I have all the results.
I discovered that when I tried to check my license usage for 30 days that was empty.
However, the "today" license usage works perfectly.
I have this problem since last week but I didn't change anything on my servers during this period.
I verified privileges and ownership in /opt/splunk but everything is ok.
My user is admin and can access all indexes, full access.
All my config files are consistent and no error found when I run the debug command splunk cmd btool check
Do you have any ideas ?
Thank you in advance
Are you forwarding Cluster Master logs to Indexers ? If not then it is recommended to send data from Search Head and CM to Indexers, please refer https://docs.splunk.com/Documentation/Splunk/7.0.1/DistSearch/Forwardsearchheaddata
Is it ? I don't think so because when you use
splunk_server in your query this means you are searching data from Indexers (Means from those servers on which data is actually stored). So in your case when you run
index=_internal splunk_server=master it is displaying data means you are trying to search data from your Cluster Master only.
When you changed outputs.conf to send data from CM to IDX, after that have you restarted splunk?
Hello Harsmarvania57 ,
thanks for your prompt reply.
Yes I did restart splunk service, as I told at the beginning, it was working until last week but non modification made between last good known configuration and today.
The good news are that I think that the problem is solved.
I inserted in my
inputs.conf (in system/local) the following in order to force parsing the
[monitor://$SPLUNK_HOME/var/log/splunk/license_usage.log] index = _internal disabled = 0
That's I don't understand is why did it stop overnight?
And also, in the default
inputs.conf it already parse all log files in
thanks once again
That's strange, if it stops again then run this command
$SPLUNK_HOME/bin/splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus on CM , it will display all the files which splunk is reading with status and percent so that you can identify whether splunk is reading log files or not.