Glad it was a simple fix this time!     ... View more

I don't have any experience in AS400. But, if you are able to export the logs from the mainframe with a third party app. You could use a forwarder to send the data to Splunk. If the data you get the data CIM compliant you should be able to use it in ES. In search answers there seems to be a varied amount of ways that people are using to get this data in. Good luck, Derrek ... View more

In my experience having it owned by splunk:splunk has given me the best results. However you should be good if that file has permissions of 750. -Derrek ... View more

Check that the permissions/ownership of your Splunk directory are correct. I ran into a similar issue where Splunk was started as root user and when the new app was installed it was installed as the root user. Any changes that I made to that app while logged into the Splunk instance did not persist as it didn't have rights to write to the directory. Thank you, Derrek ... View more

First let me correct myself. only the raw data is replicated. So in the above example you can swap the terms raw and index files. Here is what happens when you lose a replicated index in at 2:1 scenario. When the searchable index goes down. the replicated index will then start to reindex from the raw data. The time it takes to make it fully searchable again is unknown as it depends on multitude of factors. But to reach that "fully searchable" threshold it will have to finish reindexing the all of the buckets in that index. Regards, Derrek ... View more

This depends on the amount of disk you have available and are willing to give up. When you do a Rep_factor of 2 and search_factor of 2 it is keeping both the raw data and the index files on the both indexers it is replicating to. When you have a search_factor of 1 it is keeping raw and index files on one indexer and then just replicating the index files to the other indexer. This saves on space and provides enough data to rebuild the index if needed. On a multi-site cluster I would do at least origin:2,total:3. replication This will provide you site1: indexer with raw&index files, indexer with index files and site2: indexer with index files If you do a search factor of origin1:total2 this will make the site2 indexer maintain both raw and index files. Take a few moments to review this Splunk Doc as it goes in more detail on setting up search and replication factors on a mulitsite cluster. https://docs.splunk.com/Documentation/Splunk/7.3.2/Indexer/Sitereplicationfactor Thank you, Derrek ... View more

I think once you have Site replication confirmed you should be good with a single site backup. But please keep this in mind. Any data not sync'd across the lines or in transit will be lost if the non backup site died or the connectivity was cut. If you feel this answer was sufficient please take moment and accept it. Take care, Derrek ... View more

Are you backing up for resiliency or for data retention purposes? What is your site replication factor? to give you a real quick rundown on site replication. the site specific settings in the [clustering] stanza are: site_replication_factor = origin:2,total:3 site_search_factor = origin:1,total:2 Origin means how many copies to keep on the originating site. the difference in the total specifies how many copies to keep on other sites. Only focusing on the replication factor here is an example from above would be the following Site 1: 2 copies Site 2: 1 copy For additional settings and information on replication strategies take a look here. https://docs.splunk.com/Documentation/Splunk/7.3.2/Indexer/Sitereplicationfactor If your repfactor doesn't' have the origin and total specified you are likely not replicating across sites. -Derrek ... View more

To resolve an already installed directory you'll have to ask the system admin to chown -R splunkuser:splunk For the future I would do one of two things. 1) Ask that a sudoers rule be put in place for the splunk user to "sudo yum install splunk" 2) ask that the RPM file run a chown on the $RPM_BUILD_ROOT to splunkuser:splunk Separately I would use a sudo command to start splunk as the splunk user for the first run and seed the password. You can find how to do that towards the bottom of this doc. https://docs.splunk.com/Documentation/Splunk/7.3.1/Installation/StartSplunkforthefirsttime Good Luck! -Derrek ... View more

Here are the things I would check. 1) Does this search have IPTables or Firewalld running? If so, Ensure the proper ports are open (replication port, web port, splunkd, etc. ) 2) Is this captain able to talk to the other search heads? 3) Search index=_internal host= For errors that are reporting in. Usually when I run into this problem it is either the software firewall on the OS or a firewall rule on the network that is blocking the traffic. -Derrek ... View more

You can use the infrastructure app https://splunkbase.splunk.com/app/3975/ OR there is this Nix app that come with some collection scripts https://splunkbase.splunk.com/app/3975/ Depending on what information you are trying to gather you may be able to use a mixture of pieces from both apps or just set up your own monitoring stanza's. ... View more

If you know the full file path of the logs you are trying to monitor. Can you search index=_internal for that file path and see if the system is attempting to monitor the files and maybe receiving a permissions error? ... View more

Where are you trying to set the hostname at? The host that has the UF installed on it. Or the inputs.conf from the deployed app on the UF. (this will set the hostname for the logs ingested by the input.conf stanza) ... View more

See if this helps you at alll. host_regex = \/app\/splunkforwarder\/logs\/(\w+)\/\w+.log$ Unless there are special characters in your path this should capture the containername. I believe your regex was erroring because of the *.log. (I could be wrong. Hopefully this helps) ... View more

I found a previous answers link that may help you with this. https://answers.splunk.com/answers/525220/how-to-rename-column-values-when-making-a-chart.html ... View more

What errors are you seeing for the expected file in $SPUNK_HOME/var/log/splunk/splunkd.log? ... View more

You will likely need to configure your TIME_FORMAT in props.conf. However, without seeing the data it's hard to say. You should take a read through https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/Configuretimestamprecognition this should help you in isolating down timestamp in your data. ... View more