Hello, I have a test environment and the SHC members aren't allocated the recommended resources (because it's test) however i haven't had any issues with the environment until recent. For whatever reason in my test environment the 3 node SHC members keep getting shut down because of signal 9 (the server itself is killing the splunk process) Signal 9 is a KILL signal from an external process. The server is running out of memory, and thats the cause for the kill If i restart the SHC members the resources are freed but the spiral starts over once again. screenshot that shows the decline, something is eating away at it.     When i run the top command on the searcheads and press e to change the unit i can see it's splunk mongod that's taking up most of the mem so far. I also will have replication issue every now and again, where i have to resyc. ... View more

Hello All, I'm attempting to convert a splunk instance from windows to Linux but am having a hard time understanding the process fully. These are the steps https://docs.splunk.com/Documentation/Splunk/latest/Installation/MigrateaSplunkinstance Windows has splunk 7.2.5 with ITSI installed. I've used 7zip to tar the splunk home dir to a file called splunk.tar.gz Am I supposed to have splunk installed on Linux at 7.2.5? The same way you would for any instillation and then copy over the splunk.tar.gz from the windows server... Then extract it's content over the Linux out of the box 7.2.5 Linux instance? Or am I supposed to copy extract the windows splunk.tar.gz file to opt/splunk location and then Install splunk 7.2.5? Trying to extract the splunk.tar.gz file in Linux works to put the bits in the /opt/splunk location so at first glance everything looks fine...but if you go to try and run "splunk start" from the bin directory it tells you "splunk start" isn't a know command. So somewhere down the line I'm getting this fairly easy process confused. Can someone please spin me in the right direction, what am I doing wrong? What's out of sequence?     Edit: sooo I see I missed step 3, that details installing splunk on the Linux node.   What's really confusing here is it doesn't tell you that you have to grab pieces of the windows install to copy over top of the Linux dirs..  My assumption was you copy the entire windows home directory overtop or to /opt/splunk in Linux... And that's doesn't appear to be the case. The instructions are not clear here and that's where there's confusion ... View more

I'm having a hard time getting my stanza setup correctly. I basically want to monitor the maillog directories (maillog + maillog-date) and choose the best appropriate sourcetype However the archive maillog directories aren't coming in Can someone spin me In the right direction on how to better write this stanza? Please resist the urge to send me a splunk doc link as I've been rummaging through those for a while.. it's not clicking   Can someone please help me in rewriting a better stanza [monitor:///var/log] Whitelist =(maillog$) disabled = false sourcetype = maillog Index = linux Currently it's not working where it's pulling in the archive logs. So anything with a date after maillog isn't getting pulled I think I tried [monitor:///var/log/maillog*] without the whitelist but it isn't working ... View more

I'm trying to follow guides on how to create a new indexed field. Basically creating a field that gives us the name of the hf the data came from: "Splunk_HF" Im having a hard time understanding how to actually grab the heavy forwarders name. If this was a raw log i would attempt to do regex on the host based on where that name in we can within the log but here in drawing a blank. It's like I'm pulling the info from the air and I simply don't know the right syntax to make this happen I'm sure plenty have done it his before and it should be similar for each of us. Can someone please stir me in the right direction with my configurations   This is my idea so far, can someone please correct my mistakes Transforms.conf [getting_splunk_forwarder] DEST_KEY = MetaData:Host REGEX = I have no idea Format = host::$1 Props.conf TRANSFORMS-extract = getting_splunk_forwarder Fields.conf [getting_splunk_forwarder] INDEXED = true   I really do appreciate splunk docs and generally people will post links for answers by themselves, but if someone could please show me the proper syntax in the stanza it will help me understand. ... View more

Hello, We are trying to trim the fat In our log ingestion and need to determine what's actually in use vs what's ingested. Is this a possibility in any sense? Weather it be at the source types level or even More granular at the source or specific events  I think this would be super useful as we currently don't know what's valuable information In splunk and what's not.   Thanks! ... View more

Hello All I'm trying to use eval if like command with json type data (kv_mode = json) but it seems as though it's not respecting the command when used on this type of data. I'm searching Nessus data and we are using Splunk_TA_nessus I'm trying to do something like: index=nessusdata sourcetype="tenable:sc:vuln" scan_result_info.name="my scan*" | eval newfield=if(like(scan_result_info.name, "my scan%"), "it's working", "it's not working") All results return as not working meaning the if like eval isn't working. I've tried it eval a=if(scan_result_info.name like "my scan%", "working", "not working") Neither works with Nessus type data but everything works when I use the same commands on iis type data. I know that I'm typing the commands correctly. Could someone explain to me how to get this to work with data where kv_mode = json Is there another way to go about this or am I out of luck with eval if like against Json type data? ... View more

I'm utilizing tokens on my dashboard to dynamicly generate date ranges for the header. This involved adding a panel that uses html. It works fine to run the tokens in the header of the dashboard but the tokens don't run when the dashboard is scheduled to delivery pdf <div id="custom_header"> <div id="custom_header_title"><h1>weekly report for analyst for week $earliest_time$ through $latest_time$</h1></div> </div> when the report is emailed via PDF it actually shows $earliest_time$ through $latest_time$ (ignores running the tokens)... Vs the dashboard that actually runs the tokens and showing the relative dates I believe this maybe a limitation in splunk looking at the doc but ask for someone confirm? And if there work around? https://docs.splunk.com/Documentation/Splunk/7.3.3/Alert/EmailNotificationTokens ... View more