Hello, I have a test environment and the SHC members aren't allocated the recommended resources (because it's test) however i haven't had any issues with the environment until recent. For whatever reason in my test environment the 3 node SHC members keep getting shut down because of signal 9 (the server itself is killing the splunk process) Signal 9 is a KILL signal from an external process. The server is running out of memory, and thats the cause for the kill If i restart the SHC members the resources are freed but the spiral starts over once again. screenshot that shows the decline, something is eating away at it.     When i run the top command on the searcheads and press e to change the unit i can see it's splunk mongod that's taking up most of the mem so far. I also will have replication issue every now and again, where i have to resyc. ... View more

Hello All, I'm attempting to convert a splunk instance from windows to Linux but am having a hard time understanding the process fully. These are the steps https://docs.splunk.com/Documentation/Splunk/latest/Installation/MigrateaSplunkinstance Windows has splunk 7.2.5 with ITSI installed. I've used 7zip to tar the splunk home dir to a file called splunk.tar.gz Am I supposed to have splunk installed on Linux at 7.2.5? The same way you would for any instillation and then copy over the splunk.tar.gz from the windows server... Then extract it's content over the Linux out of the box 7.2.5 Linux instance? Or am I supposed to copy extract the windows splunk.tar.gz file to opt/splunk location and then Install splunk 7.2.5? Trying to extract the splunk.tar.gz file in Linux works to put the bits in the /opt/splunk location so at first glance everything looks fine...but if you go to try and run "splunk start" from the bin directory it tells you "splunk start" isn't a know command. So somewhere down the line I'm getting this fairly easy process confused. Can someone please spin me in the right direction, what am I doing wrong? What's out of sequence?     Edit: sooo I see I missed step 3, that details installing splunk on the Linux node.   What's really confusing here is it doesn't tell you that you have to grab pieces of the windows install to copy over top of the Linux dirs..  My assumption was you copy the entire windows home directory overtop or to /opt/splunk in Linux... And that's doesn't appear to be the case. The instructions are not clear here and that's where there's confusion ... View more

I'm having a hard time getting my stanza setup correctly. I basically want to monitor the maillog directories (maillog + maillog-date) and choose the best appropriate sourcetype However the archive maillog directories aren't coming in Can someone spin me In the right direction on how to better write this stanza? Please resist the urge to send me a splunk doc link as I've been rummaging through those for a while.. it's not clicking   Can someone please help me in rewriting a better stanza [monitor:///var/log] Whitelist =(maillog$) disabled = false sourcetype = maillog Index = linux Currently it's not working where it's pulling in the archive logs. So anything with a date after maillog isn't getting pulled I think I tried [monitor:///var/log/maillog*] without the whitelist but it isn't working ... View more

I'm trying to follow guides on how to create a new indexed field. Basically creating a field that gives us the name of the hf the data came from: "Splunk_HF" Im having a hard time understanding how to actually grab the heavy forwarders name. If this was a raw log i would attempt to do regex on the host based on where that name in we can within the log but here in drawing a blank. It's like I'm pulling the info from the air and I simply don't know the right syntax to make this happen I'm sure plenty have done it his before and it should be similar for each of us. Can someone please stir me in the right direction with my configurations   This is my idea so far, can someone please correct my mistakes Transforms.conf [getting_splunk_forwarder] DEST_KEY = MetaData:Host REGEX = I have no idea Format = host::$1 Props.conf TRANSFORMS-extract = getting_splunk_forwarder Fields.conf [getting_splunk_forwarder] INDEXED = true   I really do appreciate splunk docs and generally people will post links for answers by themselves, but if someone could please show me the proper syntax in the stanza it will help me understand. ... View more

Hello, We are trying to trim the fat In our log ingestion and need to determine what's actually in use vs what's ingested. Is this a possibility in any sense? Weather it be at the source types level or even More granular at the source or specific events  I think this would be super useful as we currently don't know what's valuable information In splunk and what's not.   Thanks! ... View more

Hello All I'm trying to use eval if like command with json type data (kv_mode = json) but it seems as though it's not respecting the command when used on this type of data. I'm searching Nessus data and we are using Splunk_TA_nessus I'm trying to do something like: index=nessusdata sourcetype="tenable:sc:vuln" scan_result_info.name="my scan*" | eval newfield=if(like(scan_result_info.name, "my scan%"), "it's working", "it's not working") All results return as not working meaning the if like eval isn't working. I've tried it eval a=if(scan_result_info.name like "my scan%", "working", "not working") Neither works with Nessus type data but everything works when I use the same commands on iis type data. I know that I'm typing the commands correctly. Could someone explain to me how to get this to work with data where kv_mode = json Is there another way to go about this or am I out of luck with eval if like against Json type data? ... View more

I'm utilizing tokens on my dashboard to dynamicly generate date ranges for the header. This involved adding a panel that uses html. It works fine to run the tokens in the header of the dashboard but the tokens don't run when the dashboard is scheduled to delivery pdf <div id="custom_header"> <div id="custom_header_title"><h1>weekly report for analyst for week $earliest_time$ through $latest_time$</h1></div> </div> when the report is emailed via PDF it actually shows $earliest_time$ through $latest_time$ (ignores running the tokens)... Vs the dashboard that actually runs the tokens and showing the relative dates I believe this maybe a limitation in splunk looking at the doc but ask for someone confirm? And if there work around? https://docs.splunk.com/Documentation/Splunk/7.3.3/Alert/EmailNotificationTokens ... View more

Hello, I need help fixing an issue with search time field extractions in juniper fw logs (very chatty). The issue isn't actually the props or transforms (I don't believe). They extract the events perfectly....."most of the time". But every now and again you click on an event and notice that it's fields haven't been extracted. The events with the missing field extractions are similar in context as the ones that work so no obvious reasons as to why it's not following the rule book of the props and transforms set before them. Doesn't make sense that it works all the time for majority firewalls and only works sometimes for 3. It is noted that within 1 second there's about 113 events (all sharing the same second time stamp) however there's other firewalls that are sending the same amount of events and never have the issue. All firewalls (host) are using the exact same props and transforms.conf files via deployment server sent For example the exact same server that was having the issue before around 1pm isn't having it right now. So I guess it's somewhat sporadic. It's only effecting 3 host out of 22 (now these 3 host are on the top side of events, though there's a 4th on the topside with the same # of events and never has issues). Juniper fw logs > hf/syslog server > index cluster Any thoughts of what to look into as to why for only a few firewalls are having this issue. Thanks ... View more

Hello All, I have a situation where I need to figure out a creative solution before sending out a specific alert but having a hard time. Problem: Security needs to know when Macfee mail logs "status" shows as "Emailed Deferred". Currently they get the alert based on the cron schedule set for the alert and it works, However it can appear as a false positive as sometimes they'll get this alert, login to the email system just to see that the status is no longer deferred (it sent out). How do I bake logic into this alert where splunk won't send out an alert unless the status has been that way over x amount of time? It's like the logic would need to look at each email, check the time stamp of each event, if the status switches to Email deferred monitor that email/host for x amount of time incase there's an email sent event within your set time period. So the idea is, if that if an email is Deferred perhaps monitor that email (by subject) for 10 minutes. If an "email sent" event is found for that same subject within 10 minutes do not send alert. But if no "email sent" event is found then send the alert as normal after that 10 minutes have expired. Index=mcafee sourcetype=Meg status="Emailed Deferred"| stats values(_time) as Time values(sender) as Sender values(dest) as Destination values(status) as Status by host|sort-Time ... View more