topic Re: Extract Log Data in Splunk Search

Extract Log Data

GRC — Wed, 20 Oct 2021 01:10:02 GMT

Hi Team,

I am pulling hair to figure out a query to extract data into a table with following information.

stopping system logging (rsyslog)
stopping the Tripwire agent
stopping the Splunk agent

In hosts. Also want to know who stopped/disabled those services at what time so I can reconcile it with approved changes.

Really appreciate if someone can help.

Thank you.

Re: Extract Log Data

richgalloway — Wed, 20 Oct 2021 17:00:53 GMT

We need to know more to offer specific help. For example, what platform? What does the data look like?

Re: Extract Log Data

GRC — Wed, 20 Oct 2021 22:15:24 GMT

Hi @richgalloway

The data is in Splunk.

The logs are in the xxx_log..

Thank you in advance.

Re: Extract Log Data

nmohammed — Wed, 20 Oct 2021 23:07:44 GMT

@GRC

Sample log events data will help to work further on your request.

Re: Extract Log Data

richgalloway — Wed, 20 Oct 2021 23:55:17 GMT

Yes, I guessed the data is in Splunk, but is it Windows data, Linux data, or some other source?

As already suggested, it's very helpful to see sample events.

Re: Extract Log Data

GRC — Fri, 22 Oct 2021 06:02:55 GMT

@richgalloway @nmohammed Thank you.

Here are some sample logs:

This is one source:

Oct 21 15:13:51 swc-03-jals rsyslogd: [origin software="rsyslogd" swVersion="1.24.5-57.el5_9.1" x-pid="1313" x-info="http://www.rsyslog.com"] start
host = edc-03-jals
index = linux_logs
source = /var/log/messages
sourcetype = syslog

The other one is:

Oct 21 12:40:15 keybox sudosh: i3ev9zjyY46UEJPj jasan: service rsyslog start
host = keybox
index = sudosh_app_pro
source = /var/log/hosts/messages
sourcetype = sudosh_app

Oct 21 12:40:09 keybox sudosh: i3ev9zjyY46UEJPj jasan: service rsyslog stop
host = keybox
index = sudosh_app_pro
source = /var/log/hosts/messages
sourcetype = sudosh_app

I want to combine these 2 sources into a single table and found who started or stopped rsyslog service at what time.

Thank you

Re: Extract Log Data

PickleRick — Fri, 22 Oct 2021 09:08:32 GMT

It's not (mostly) about querying the data you already have. It's more about what information you have in your logs.

Typically, unless you have some other means of monitoring the system, you only get the info that a service has been started/stopped/restarted and so on coming from either the service itself or in some cases the init process (usually the systemd these days). But if you don't have any other method of monitoring your users activity, you won't know who was responsible for stopping the service because you don't have this information in your logs.

OK, you could do some correlations between logged-in sessions and the time of service start/stop but it's easily defeatable by running a process with a delayed action (like running a "sleep 3600 && systemctl restart rsyslogd" in a screen). You would also not be able to tell which user did something in case of more than one user logged in at the same time.

So the general problem is not how to search for this data in splunk but rather how to get this data from your source and - most importantly - whether the source does log this at all.

There are many ways of approaching user activity logging and that's definitely out of scope of the splunk forum - the only connection is that there are some ready-made apps for some solutions (like auditd) but that's it. You still have to do most of the work on the source side.

OK. I see that some of your logs show sudo activity. That can be traced indeed. But that's where it gets a bit more complicated. It all comes down what you want to find.

If you just want to find all sudo logs where someone launched a "sevice <whatever> <command>" command, that's relatively easy but it doesn't tell you anything about the service behaviour itself. So it all boils down on what _precisely_ would be your requirement.