About jackjack

jackjack · ‎12-10-2021

I found a solution so I'll post it. Unfortunately it doesn't solve the filtering issue. But it's good enough for me. I ended up splitting out my index into dev and prod groups so I was able to solve this by simply specifying the index (the index performs all the filtering I need). This search was helpful for me to determine which index to use in the command |tstats dc(host) as val max(_time) as _time where index=* host=* by host, index Then I simply modified the two index lines to use the index that I saw referenced in the above search.

jackjack · ‎12-07-2021

Hello, I am planning to setup some custom metrics indexes using this guide: https://docs.splunk.com/Documentation/ITSI/4.10.2/Entity/CustomIndexes First question: There are two linux metrics macros. I am planning to modify both to be safe but I assume I only need to modify the ta one since that is what I am using to collect data. What is the other one for? 1. itsi_entity_type_nix_metrics_indexes 2. itsi_entity_type_ta_nix_metrics_indexes Second question: I did a ctrl+f on itsi_im_metrics and noticed that the itsi_im_metrics_indexes macro uses a similar definition to the itsi_entity_type macros that I listed above. I cannot find any information about this macro. Do I need to modify it? What is it used for? Thank you!

jackjack · ‎12-02-2021

I always perform a search before asking a question. I did not find my specific question answered. I am able to search across ALL hosts however my issue specifically is that I cannot figure out how to split them by host name. I looked through the post you linked and this is the only link that may be helpful: https://www.duanewaddle.com/proving-a-negative/ However that is still a fundamentally different process using lookup tables and it's performing a count instead of looking at the last time sent. Are you suggesting that my method, which is presented by splunk itself through the IT Essentials Learn App, will not work? Can I not filter using something similar to one of the methods I attempted in my original post?

jackjack · ‎12-01-2021

I am attempting to use a search from IT Essentials Learn named "Alert when host stops reporting data - Linux - IT Essentials Work" Is it possible to filter this alert by host type? I've performed a number of tests now and it seems my only option is to search against all hosts. Here is the search from IT Essentials Learn |tstats dc(host) as val max(_time) as _time where index="<INDEXES-TO-CHECK>" host="<HOSTS-TO-CHECK>" by host |append [|metadata type=hosts index="<INDEXES-TO-CHECK>" | table host lastTime | rename lastTime as _time | where _time>now()-(60*60*12) | eval val=0] |stats max(val) as val max(_time) as _time by host | where val=0 | rename val as "Has Data" | eval Missing Duration= tostring(now()-_time, "duration") | table host "Has Data" "Missing Duration" I modified the 2 index lines and the host line. If I use * for all 3 it kind of works but checks against every host. If I use host=*dev* it displays all hosts without the name *dev* as evaluating to 0 whereas all the *dev* hosts get evaluated to 1. To counteract this I tried adding a where host=*dev* elsewhere (I tried it in the metadata portion, as a where clause at the end of all the metadata piping, as a where clause next to where val=0, etc.) but this has the effect of just completely removing a host that isn't sending data from the list (or removing all hosts), so that also does not work. Is it possible to split this up based on hosts or am I stuck with an all or nothing? Edit: I tried adding a where all the way at the end. It does not work with host="*dev*" However, I can use host!="some host name" to filter those out. I'm not sure why I can use negation but not wildcards? Edit2: I am searching on the prior 5mins if that matters at all

jackjack · ‎11-29-2021

Thanks for your suggestions! Turns out the field never exists in a parsed state. This query solved the issue for me. index="*" host="*dev*" source="/var/log/auth.log" process="sudo" ufw | rex field=_raw "COMMAND=(?<ufw_path>\/usr\/sbin\/ufw )(?<ufw_command>.*)" | where ufw_command!="status" | table _time host user _raw ufw_path ufw_command

jackjack · ‎11-23-2021

Hello all, I am trying to setup a search that logs ufw commands, while ignoring any ufw status commands. I have tried a number of methods so far but cannot get the COMMAND field to filter appropriately. Here is a version of the search: ``` index="*" host="*dev*" source="/var/log/auth.log" process="sudo" COMMAND="/usr/sbin/ufw" | table _time host user _raw | where COMMAND!="*/usr/sbin/ufw status*" ``` I've tried a number of things including trying NOT instead of !, searching for various strings (status, *status*, etc.), filtering on the _raw field instead of COMMAND, using search instead of where, putting the table after the where, etc. I cannot get the events to filter out. It seems like I either get all the events or none of the events depending on the filter I choose. Any help here? Thank you!

jackjack · ‎10-27-2021

Based on Alert Scheduling Best Practices it is recommended to use a time window that matches up with the cron window. "Both the search time range and the alert schedule span one hour, so there are no event data overlaps or gaps."

jackjack · ‎10-25-2021

This question is based on a comment from @woodcock on this post: https://community.splunk.com/t5/Splunk-Search/Why-are-real-time-searches-not-running-and-getting-error-quot/m-p/281407 in which the alert equation provided is as follows: "Schedule it to cover a span of X and run it every X/2. This covers the case where events at the end of span t an the beginning of t+1 would just miss triggering in those windows but will hit in the next alert run. Then make X as large as you can stomach." I do not fully understand this so I am hoping someone can help me out here. Let's say I have an alert running every 5 mins. By that equation I should search -10m to now. But isn't that going to also significantly overlap with the prior run? Why not search -6m to now, for example? How do span sizes affect things? Here is an alert I have running every 5 mins. I did notice the search itself picks up the current span and the prior span so I have been wondering how to optimize this properly. | mstats avg(cpu_metric.pctIdle) as Idle WHERE index="itsi_im_metrics" AND host="*" span=5m by host | eval cpu_utilization=round(100 - Idle,2) | where cpu_utilization > 90 | stats list(host) as host_list list(cpu_utilization) as avg_cpu_utilization

jackjack · ‎10-25-2021

Thank you for linking that. Looks like I'll need to spend a bit of time understanding how to optimize everything. Unfortunately I won't have time to dive deep on the optimization work for a bit. Do you happen to know how to change the search default for ad-hoc searches (like when I'm trying a new query out) with that system?

jackjack · ‎10-22-2021

Is it possible to change the default search performance to "high_perf" in splunkcloud? On splunkcloud in the search bar you have the option of setting search performance to: standard_perf (search default), limited_perf, high_perf or Policy-Based Pool. I have begun using high_perf for my queries since otherwise things are WAY too slow. However it constantly changes me back to the default standard_perf. I cannot find the setting for this anywhere and have had no luck searching for documentation either.

jackjack · ‎10-21-2021

IT Essentials Learn App led me down the wrong path. This is the search I am using which appears to work just fine. I think linux_secure has been deprecated seeing as it does not show up anywhere in the default config files (and these list the default sourcetype for each source so clearly it's no longer being used as a default sourcetype with the unix add on). index="*" host="*" source="/var/log/auth.log" tag="authentication" process="sudo" tag="success" | table _time host user source process tag _raw

jackjack · ‎10-21-2021

No worries, it's hard to keep everything straight in splunk. So it wasn't a problem with the config. It was just IT Essentials Learn leading me down the wrong path. It seems to me that linux_secure has been deprecated. I was able to get the results I need using a different search which I will post shortly.

jackjack · ‎10-20-2021

Unfortunately I am responsible for both splunk setup and splunk searching. I think you're right in that the data is there, I just need to find it another way. I think IT Essentials Learn just led me down the wrong path - perhaps linux_secure was the method in the past and things have changed now.

jackjack · ‎10-20-2021

Do you know of a source that tells me how to configure it? I have looked through the existing add-on setup documentation and cannot find what I need. Are you sure the TA_nix add-on is deprecated? I thought it was just the App itself? I don't see any deprecation callouts on its page. Those other apps aren't supported by splunk so I cannot use them. I've had issues with unsupported apps in the past so all apps must be splunk supported now. Additionally, it looks like they'll cause conflict with the add-on for unix, which is working (just not with the linux_secure type).

jackjack · ‎10-19-2021

Hi all, I am trying to setup some sort of dashboard to view a list of sudo commands by server. I started with the IT Essentials Learn App which recommends this command: index=* sourcetype=linux_secure process=sudo COMMAND=* host=* | rex "COMMAND=(?<raw_command>.*)" | eval COMMAND=coalesce(raw_command, COMMAND) | table _time host USER PWD COMMAND This command did not work for me so I started playing with it a bit. I realized that the sourcetype=linux_secure does not exist. My understanding is that the splunk add-on for unix and linux is supposed to apply this sourcetype. I verified my configuration and didn't see anything to modify so I went ahead and looked at the $SPLUNK/etc/apps/Splunk_TA_nix/default/inputs.conf file. I cannot find a single instance of sourcetype=linux_secure in that config file so I don't think that sourcetype is being applied to any sources. Has linux_secure been deprecated or do I simply need to modify my local/inputs.conf file with something? Does anyone have a recommended way to perform this search? I have tried a number of methods but am struggling to get what I need.

jackjack · ‎10-18-2021

This answer was the most helpful for me. I am adding a few things I found helpful for anyone using Certbot/LetsEncrypt 1. Generate the pem key using the letsencrypt certs cd /etc/letsencrypt/live/your-server-hostname/ cat cert.pem privkey.pem chain.pem > splunk.pem chmod 777 splunk.pem 2. Use the following for inputs.conf [http] disabled = 0 index = your-hec-index-name enableSSL = 1 serverCert = /etc/letsencrypt/live/your-server-hostname/splunk.pem sslPassword = crossOriginSharingPolicy = * 3. Troubleshoot the connection This comes from this forum post curl -k https://your-heavy-forwarder.com:8088/services/collector/event -H "Authorization: Splunk splunk-hec-token" -d '{"event":"hello world"}' -v

jackjack · ‎10-13-2021

Nvm I searched for it and was able to find it. Modified the permissions to show it for all apps. Saved the URLs for direct access to ITSI alerts - I don't think there is a tab for this in the IT Essentials apps though so unless someone knows the URL they won't be able to access alerts that are saved only for those apps? The URLs I saved: https://mycompany.splunkcloud.com/en-US/app/it_essentials_learn/alerts https://mycompany.splunkcloud.com/en-US/app/itsi/alerts

jackjack · ‎10-13-2021

Pretty much the title. I have created alerts using IT Essentials Learn app. The alert is running because I receive alerts to slack. However, I cannot figure out where the alert is housed so I am unable to return to the edit screen and modify the alert. I've looked through both IT apps as well as the Search and Reporting app alerts panel. I cannot find the alerts anywhere. Where are they housed?

jackjack · ‎09-14-2021

Hello All, I have set up the Splunk Add-On and Splunk App for Unix and Linux. Data is flowing properly however I am having an issue with alerts. I am trying to set up alerts for various things to slack. I have the first alert on memory working. I set it to 1 min real-time and it seems to work just fine. This is the working query: `os_index` source=vmstat | where max(memUsedPct) > 90 | stats max(memUsedPct) by host However, when I try to do the same for disk, it does not work. I have tried expanding to 5min and 30min real-time windows but the only way I get data to show up in this query is by removing the where clause. I also tried using something like latest() instead of max() but that didn't help. What am I doing wrong here? `os_index` source=df | where max(UsePct) > 10 | stats max(UsePct) by host Thank you, jackjack

jackjack · ‎09-14-2021

How long did it take for alerts to show up in the dashboard for you? I see them firing if I go to Activity > Triggered Alerts but I still do not see them in my Splunk App for Unix Alerts Dashboard.

jackjack · ‎09-14-2021

This was the solution for me. CLI commands in the other answer didn't work because my license is expired and I need to switch groups. Once it expires you lose ability to change group from both the GUI and CLI.

jackjack · ‎09-09-2021

Thank you both for your assistance! I am working to setup the app now. This seems like a much better solution.

jackjack · ‎09-08-2021

We meet again PickleRick, Thank you for your response. So here is my current inputs.conf file. I've gone through everything in my /var/log folder and attempted to classify it. As you can see I have a number of logs with an unknown source type (or a guess that I'm unsure if it's correct or not) - how can I figure out what to put there? ## inputs.conf for splunk universal forwarders ## /var/log # update-alternatives, symbolic links # Ok to ignore? [monitor:///var/log/alternatives.log] disabled = false index = main # sourcetype = ??? # auth log (sudo, ssh, etc.) [monitor:///var/log/auth.log] disabled = false index = main sourcetype = linux_secure # bootstrap (this may not actually get updated on boot) # Ok to ignore? [monitor:///var/log/bootstrap.log] disabled = false index = main # sourcetype = linux_bootlog # btmp log (failed login attempts) # Splunk cannot index this data type # TODO figure out how to get this in splunk # [monitor:///var/log/btmp] # disabled = false # index = main # sourcetype = ?? # dpkg log (dpkg and apt installs) [monitor:///var/log/dpkg.log] disabled = false index = main # sourcetype = ?? # faillog (failed user logins) # Splunk cannot index this data type # TODO figure out how to get this in splunk # [monitor:///var/log/faillog] # disabled = false # index = main # sourcetype = ?? # kern log (kernel logs) [monitor:///var/log/kern.log] disabled = false index = main # sourcetype = linux_messages_syslog # lastlog (last login by user) # Splunk cannot index this data type # TODO figure out how to get this in splunk # [monitor:///var/log/lastlog] # disabled = false # index = main # sourcetype = ?? # syslog (system logs) [monitor:///var/log/syslog] disabled = false index = main sourcetype = linux_messages_syslog # tallylog (count of attempted logins/fails) # Ok to ignore? # [monitor:///var/log/tallylog] # disabled = false # index = main # sourcetype = ?? # ufw log (firewall) [monitor:///var/log/ufw.log] disabled = false index = main # sourcetype = linux_messages_syslog # wtmp (login records) # Splunk cannot index this data type # TODO figure out how to get this in splunk # [monitor:///var/log/wtmp] # disabled = false # index = main # sourcetype = ?? ## /var/log/subdirs # apache access log [monitor:///var/log/apache/access.log] disabled = false index = main sourcetype = access_combined # apache error log [monitor:///var/log/apache/error.log] disabled = false index = main sourcetype = apache_error # apache other vhosts access log [monitor:///var/log/apache/other_vhosts_access.log] disabled = false index = main sourcetype = access_combined # apt history log [monitor:///var/log/apt/history.log] disabled = false index = main # sourcetype = ?? # apt term log [monitor:///var/log/apt/term.log] disabled = false index = main # sourcetype = ?? # Ignoring /chrony (empty) # Ignoring /installer (tons of files) # Ignorning /journal (binaries) # letsencrypt log [monitor:///var/log/letsencrypt/letsencrypt.log] disabled = false index = main # sourcetype = ?? # mysql error log [monitor:///var/log/mysql/error.log] disabled = false index = main sourcetype = mysqld_error # unattended upgrades dpkg log [monitor:///var/log/unattended-upgrades/unattended-upgrades-dpkg.log] disabled = false index = main # sourcetype = ?? # unattended upgrades shutdown log [monitor:///var/log/unattended-upgrades/unattended-upgrades-shutdown.log] disabled = false index = main # sourcetype = ??

jackjack · ‎09-08-2021

I would like to retrieve the data in /var/log as correctly as possible. Currently I am simply monitoring the entire /var/log folder with no pre-selected source type. On the List of pretrained source types I see a few callouts for log files such as syslog but the majority of log files are not present in this list. Perhaps some of these types can be used elsewhere though? For example, I see the linux_messages_syslog pretrained type refers to logs in /var/log/messages and since syslog != messages I presume this type may be useful on other files as well? So I can use the few pretrained source types and then do I need to make my own source types for all the other log files? Is there any repository with user created source types? I have to imagine most log file types have had source types created for them by now? Or do people just not apply source types and simply search on the unstructured data?

jackjack · ‎09-07-2021

Did you ever solve the second piece of your question? "Additionally, I added the new field type to highlight whether or not the entry is a failed login or a success. The visualization appears however the count(type) does not separate b/w the different values in type by srcHost. In other words within an hour span I want a column for each host with different colors representing success or failure rather than representing them all as one color." I am trying to figure this out now.

Posts	32
Solutions	5
Karma Given	17
Karma Received	0
Member Since	‎08-31-2021

Online Status	Offline
Date Last Visited	‎12-10-2021 03:41 PM

Custom itsi_im_metrics / what is itsi_im_metrics_i...

Alert when host stops reporting data (IT Essential...

Why am I unable to filter COMMAND twice?

How do I properly configure schedule search/cron/a...

Is it possible to change the default search perfor...

Search for sudo (linux_secure does not exist?)

Where are IT Essentials and IT Essentials Learn Al...

Splunk Unix Add-On Alerts Troubleshooting

Source types in /var/log

Splunk bash startup / setup (with ansible)

Re: Alert when host stops reporting data (IT Essen...

Custom itsi_im_metrics / what is itsi_im_metrics_i...

Re: Alert when host stops reporting data (IT Essen...

Alert when host stops reporting data (IT Essential...

Re: Why am I unable to filter COMMAND twice?

Why am I unable to filter COMMAND twice?

Re: How do I properly configure schedule search/cr...

How do I properly configure schedule search/cron/a...

Re: Is it possible to change the default search pe...

Is it possible to change the default search perfor...

Re: Search for sudo (linux_secure does not exist?)

Re: Search for sudo (linux_secure does not exist?)

Re: Search for sudo (linux_secure does not exist?)

Re: Search for sudo (linux_secure does not exist?)

Search for sudo (linux_secure does not exist?)

Re: Securing HTTP Event Collector

Re: Where are IT Essentials and IT Essentials Lear...

Where are IT Essentials and IT Essentials Learn Al...

Splunk Unix Add-On Alerts Troubleshooting

Re: Splunk App for Unix and Linux - Alerts don't f...

Re: Splunk.License: failed to add because: cannot ...

Re: Source types in /var/log

Re: Source types in /var/log

Source types in /var/log

Re: How to edit my timechart search that shows the...