Hello Splunkers,
I have a problem when I'm searching in _internal index from my master server.
My architecture consists a master server et four indexers.
When I search index=_internal on my master server I have results only from my indexers but not from the local server.
If I specify in my search the name of my local server index=_internal splunk_server=master then it works and I have all the results.
I discovered that when I tried to check my license usage for 30 days that was empty.
However, the "today" license usage works perfectly.
I have this problem since last week but I didn't change anything on my servers during this period.
I verified privileges and ownership in /opt/splunk but everything is ok.
My user is admin and can access all indexes, full access.
All my config files are consistent and no error found when I run the debug command splunk cmd btool check
Do you have any ideas ?
Thank you in advance
Michael
Are you forwarding Cluster Master logs to Indexers ? If not then it is recommended to send data from Search Head and CM to Indexers, please refer https://docs.splunk.com/Documentation/Splunk/7.0.1/DistSearch/Forwardsearchheaddata
Hello Harsmarvania57 ,
Thank you for replying.
My CM already forwarding to the indexers.
thx
Is it ? I don't think so because when you use splunk_server
in your query this means you are searching data from Indexers (Means from those servers on which data is actually stored). So in your case when you run index=_internal splunk_server=master
it is displaying data means you are trying to search data from your Cluster Master only.
When you changed outputs.conf to send data from CM to IDX, after that have you restarted splunk?
Hello Harsmarvania57 ,
thanks for your prompt reply.
Yes I did restart splunk service, as I told at the beginning, it was working until last week but non modification made between last good known configuration and today.
The good news are that I think that the problem is solved.
I inserted in my inputs.conf
(in system/local) the following in order to force parsing the license_usage.log
:
[monitor://$SPLUNK_HOME/var/log/splunk/license_usage.log]
index = _internal
disabled = 0
That's I don't understand is why did it stop overnight?
And also, in the default inputs.conf
it already parse all log files in
$SPLUNK_HOME/var/log/splunk
thanks once again
Michael
That's strange, if it stops again then run this command $SPLUNK_HOME/bin/splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus
on CM , it will display all the files which splunk is reading with status and percent so that you can identify whether splunk is reading log files or not.
Hello Harsmarvania57 ,
ok, I'll let you know if the problem comes back again.
Thank you very much.
Michael
awesome 🙂
Thank you very much.