How do you conduct security monitoring in Linux?

[test_qweqwe]

Hello.
How do you monitor in Linux?
For example Splunk_TA_nix works not property. Scripts not parsing as well information what I need.
http://prntscr.com/jdb2ip (was tested on Ubuntu 18.04 LTS and CentOS7)
Do you know any another alternative for it?

Also, what about your personal best practice to security monitoring Linux boxes?
Preferring open source or not third-party.

Infrasturcture:

• Linux Auditd
• sudo technology add-on (also can be monitoring by auditd)
• Linux Secure Technology Add-On (for /var/log/secure)

Network:

• Linux Netfilter (iptables) Technology Add-On
• ASN Lookup Generator (mini-whois)

Webserver:

• ModSecurity Add-on/App for Splunk

So, what can you add to that list? Any advice? 🙂

[AndySplunks]

Here are some of my use cases. This requires a relatively professional Linux team and you'll likely have to buy them some donuts.

1. Monitor for SSH activity within the same subnet. No one should be moving laterally. They should be connecting from their endpoint or the gateway/jumphost.
2. Build a list of executed commands (./executable) for each system, alert on anything new. Congrats, you've just built application whitelisting via Splunk.
3. Alert on IP addresses at the command line. Everyone should be using DNS entries for connections, right?
4. If you've got any colleagues or peers that have recently taken the CEH or OSCP courses, ask them to sit down and walk you through what they learned the first day. They'll give you the script kiddie / noob commands to watch for.
5. If you're fancy, build profiling for your Linux users. The su people typically only su, the sudo people typically only sudo.

[test_qweqwe]

@AndySplunks, thanks! :3
I will reviewed some of my old use cases.
The main problem that I don't have any colleagues that are pro in Linux or had any CEH/OSCP cert. And it's hard for me to handle so much information in a solo. Ideally, I would have to study one thing to the ideal level, rather than focusing my attention on everything. Any security chats/forums it's only where I can get any people who works in InfoSec and can give me advice (what I am actively using)

Also, it's my own list of some use cases:

1. List of all log in attempts (either successful or not)
2. Alert after X failed log in attempts
3. List of all sudo commands executed, and the user who ran them
4. Addition/removal of users to the system
5. Activation/de-activation of services (e.g. A/V, IDS, firewall)
6. Unauthorized access attempts to deactivated accounts/files/services
7. Binaries with curvature rights to rais
8. If running any NMAP on machine
9. SSH, VPN on machines where it can't be installed
10. Detecting short-lived files in /temp (noise/false-positive)
11. Installing malware or non-upgraded software
12. Recording yourself to any startup
13. Change the settings of /etc/shadow, /etc/passwd, /etc/sudoers, including changing privileges for certain users
14. Detect what drives, flash drives are connected to the host system (including rubber ducky)
15. Monitoring browser extension

[doksu]

If you’re looking specifically for a list of behaviours to detect, MITRE’s ATT@CK Matrix is exactly what is required. On the other hand, if you want to know how to detect those behaviours, please see my conf talk already mentioned above.

[doksu · Accepted solution]

I wrote all but the ModSecurity app you listed so naturally those are what I use ;). I'd also recommend two of my other apps: https://splunkbase.splunk.com/app/3506/ and https://splunkbase.splunk.com/app/3446/ The first provides "free" whois lookups and the second provides the ability to pivot on field values such as a domain to VirusTotal.

If possible, I suggest investing in sensors to produce application-layer traffic metadata with something like Splunk Stream or bro (both of which have supported apps). For more information about how this is done (and many other great recommendations), please see MITRE's free publication: https://www.mitre.org/publications/all/ten-strategies-of-a-world-class-cybersecurity-operations-cent...

One of the things you can generate in Splunk from that traffic metadata is passive DNS information. DomainTools, Farsight DNSDB, and PassiveTotal are also useful (paid) services that provide a wider ranger of historical passive information, for which there are Splunk apps of varying quality.

Now that you've got those pieces in place, I recommend watching my .conf presentation from last year which discusses how to develop detection techniques: https://conf.splunk.com/sessions/2017-sessions.html#search=The%20Art%20of%20Detection%20Using%20Splu...

[doksu]

@test_qweqwe: Please also see the slides from another conference session I presented last year that talks about how to use Auditd for detection: https://redhat.slides.com/dobrown/deck?token=BcavmnEs

[test_qweqwe]

@doksu, yeap, I already saw your presentation and it was very useful for me, although I still can't make puzzle in my head about monitoring by Auditd.

1. Logging all commands. I still don't understand even with your guide-video on youtube how to logging TTY. Your guides not working on my machine or I don't know what I'm doing wrong.

For example, to have at least something logging from TTY l I need to run one more command:
echo "session required pam_tty_audit.so enable=*" >> /etc/pam.d/system
And after it i can logging all commands, but only by user root. Commands by another users I still can't logging.

1. Watch a context of file. It's possible by auditd to parse context of file? I think it's important. For example, I made rules for changing such files as: /etc/passwd, /etc/shadow, /etc/crontab and auditd only can said me that something happen, but I can't see context of file to appreciate the whole picture. I can not assess the risks. Of course I can write scripts that will cover it

[doksu]

@test_qweqwe: it’s a different presentation as I indicated and you’re asking questions in comments here that are well outside the scope of the original question. Please submit separate questions on the site for each.

With regards to the PAM config, that is neither a Splunk issue nor an issue with the app so I suggest you contact your distribution’s support if the operating system is not functioning as expected.

[test_qweqwe]

@doksu Great, thank you for answer! 🙂

@AndySplunks provided some use cases. What can you add to his list?

About Mitre, when I having a free time I'm trying to do some cases from that list (while only offense part):
https://attack.mitre.org/wiki/Linux_Technique_Matrix
The first question, can auditd cover these use cases?
And the second, how to monitor commands? Sudo commands - easy, but what about other? (many of there use cases seems that can be success monitoring if you can logging commands)

Enable TTY logging by echo "session required pam_tty_audit.so enable=*" >> /etc/pam.d/password-auth-ac
I used it only 4 times and only one time it was worked. And I really don't understand why I can't logging any key-strokes by it (was tested on CentOS7)

[doksu]

The Splunk Answers site isn’t really the most appropriate place to discuss this in detail and it’s something that comes with experience. If you watch the video attached to the Linux Auditd app on Splunkbase, it will address several of the concrete questions here.