Our sendemail function seems to have stopped working or is only working sporadically as it might send email the next day for an issue that occurred the day before or it may not send anything at all.  Has anyone seen this before?  We did in the past and edited the sendemail.py for that one, but this issue is brand new and we're seeing it in our 7.2.6 environment and in our 8.1.2 environment.  We are a clustered environment. ... View more

We recently upgraded to version 8.1.2 Splunk and now our email alerts don't appear to be working.  I had this issue in version 6.6.3, but had not seen it since then (we did upgrade to 7.2.6 before moving to 8.1.2--did not see the issue then either).  I have looked at older logs that said to remove a few lines in the sendemail.py file and I found similar lines in the new sendemail and marked them out with a # sign, but it did not fix the issue.  We are delaying our upgrade of the Production system until we solve this email issue.  Any ass ... View more

When I go to the monitoring console and take a look at the forwarders, the console shows them as all missing yet our environment is receiving logs from many of them.  I tried to rebuild it in the monitoring console, but it fails.  Not sure how to get this fixed. ... View more

I am trying to upload documents from a user who's log files has multiple dots in the naming convention: The logs have a *.0.1.log in its file extension, would it be possible to use *.*.*.log instead? The logs have a *-20210218.000023-5644-5716.0.log in its file extension, would it be possible to use *.*.*.*.log instead? The logs have a *-20201218.105324-11260-11240.0.log in its file extension, would it be possible to use *.*.*.log instead? The logs have a *-20210209.145105-16220-11864.0.log in its file extension, would it be possible to use *.*.*.log instead? The logs have a *-20201218.105324-25876-14744.0.log in its file extension, would it be possible to use *.*.*.log instead?   Is there a way to get all of these from .log to .*.*.*.log into Splunk using one monitoring stanza? ... View more

We just loaded an AIX server with a forwarder and we're getting can't write file "/.splunk/authToken_#####_8089: Permission denied We have loaded a number of servers with the splunkforwarder, but just started getting this message.  We have a particular user to run the process and it has worked on other servers, but not these new ones.  This is using Universal Forwarder 8.0.5.  The process actually starts, but when you do ./splunk list monitor is when you get the can't write file.  We have several more servers to load and would really like an assist here. ... View more

I have just acquired the Event Timeline Visualization application and loaded it on my Deployer and pushed it out to my search heads.  Unfortunately not all the function is showing.  Do I need to load this individually on all the search heads?  I'm at a loss on how to get this out to them via the normal push from the Deployer to the search head cluster.   ... View more

index=xxxx source="/esbplogsdir/prod/Enable/LOG_Maximo_LSI_Work/Maximo/LSI_IN_msg_prod.log" OR source="/esbplogsdir/prod/WS/LOG_Maximo_SmallWorld_IPICustomerInfo/Maximo/SmallWorld_IN_msg_prod.log" | rex "(?i).*? \\- (?P<FIELDNAME>[a-f0-9]+\\-[a-f0-9]+\\-[a-f0-9]+\\-[a-f0-9]+\\-[a-f0-9]+)(?= )" | rex "(?i):.*? \\- (?P<FIELDNAME>\\d+\\.\\d+)(?= )" | search "[ERROR]" OR "failed" |stats dc(FIELDNAME) as ERROR_TRANSACTION_COUNT by source | rename source as SOURCE   This is the search, but if one or both sources have a 0 return, I want a line that lists the log file and a 0 to show in my table.  How can I do this? ... View more

I am getting an error when trying to push my application from the Deployment server to the Cluster Master.  This is the same for all the different apps that are listed in the deployment-apps directory.  This has worked many times, but just started this issue.  It was once fixed with the _internal being put in the _cluster/local/indexes.conf, but that is still there an I still get the issue. Failed to load app. Application=duke_nix_forwarder_inputs cannot be loaded, as path=/web/splunk/etc/deployment-apps/duke_nix_forwarder_inputs does not exist ... View more

I have been able to find searches for roles mapped to AD Groups, but I need to get the indexes those roles are allowed to use.  I'm creating a search to assist in upgrading to a new system.  This mapping will give me the relationships.  I have used:  | rest /services/admin/roles | table title, srchIndexesAllowed | rename title as role, but it only gives me the role and searchIndexes.  I need to map it to AD groups too.  Also, when I went to download that searches content, it did not download all the data. I also used: | rest /services/admin/roles | table title, srchIndexesAllowed | rename title as role, but what I really need is all three values: Role mapped to AD Group and role mapped to indexes in the same search.       ... View more

This is a note my client sent: I can’t really use the logs from Splunk, because they are spotty.. Think I figured out why it’s missing data though.  From our last meeting, they bumped up the transfer rate from 1024KBps to 2048KBps because the CE was generating too much data.  However, the CE raw logs generate roughly about 75MB/Minute or 1.25MBps raw.  Since Splunk does use compression at ~17%, the 2KBps gets you about 11KBps raw.  The 1.25MBps is roughly 217.6KBps compressed.  Hence, it gives you roughly 5% of the total logs.  To support Openway and prevent dropping any logs, we need to set a conservative rate to handle 110MB/minute raw, which would be ~319KBps. We did set in the limits.conf to the 2048 mentioned above, but we're not certain how to set to a level the client needs. ... View more

I have been trying to put together a dashboard for my clients that shows their usage.  The search I am still having issues with is for Average Daily ingest by index over 30 days.  I want it to show just the average number.  I have found searches that get me a distance, but none are specifically what I need. ... View more

I have tried putting the following in the props.conf file: NO_BINARY_CHECK = true and NO_BINARY_CHECK = 1 and CHARSET=AUTO   But none of these worked.  I did them in the props.conf file.  But when I look at the splunkd.log file, the Splunk still thinks of the file as binary.  How can I fix this?     ... View more

The edit dashboard has the Schedule PDF Delivery grayed out. I am root and have all the permissions I believe I need, but it remains grayed out. I'm on version 7.2.6. I verified that at least admin had the schedule_search and admin_all_objects and it does and even restarted splunk, but the feature remains grayed out. Can anyone assist. ... View more

I have the following from a client: I was about to make is for a new AD group “Splunk_CAPS_CAS_Payments” so that they are restricted to data that come from the source logging files //LOG_CAPS_CAS_PaymentEDMS//CAPS//CAS_IN_msg_prod.log and //LOG_CAS_CAPS_InvoiceGL//CAS//CAPS_IN_msg_prod.log Is this possible? These users would only be able to pull the logs from the one CAPS_IN_msg_prod.log. How can this be done? ... View more

I have a new client that has files named as follows: xxxx.xxxx.log Splunk is not ingesting them. How can I ingest logs that have that type of naming convention. I believe splunk is only looking at the xxx.xxx and can't match it to the /*.log stanza I have in the inputs.conf. ... View more