topic Re: How to parse complete command line information into the event field ? in Getting Data In

How to parse complete command line information into the event field ?

Hemnaath — Tue, 29 Sep 2020 17:14:05 GMT

Hi All, Today we got an request from a user to include the entire information provided in the command line, when checked the values parsed for the COMMAND field, it only parses until there's a space.

search query: index=unix sourcetype=linux_secure COMMAND

For example, this log:
Dec 12 15:02:17 test01 sudo: solarwinds : TTY=pts/1 ; PWD=/opt/solarwinds ; USER=root ; COMMAND=/opt/solarwinds/utilities/check_file.sh -e -f /var/log/audit/audit.log

"Problem"- COMMAND is currently parsed as "/opt/solarwinds/utilities/check_file.sh"

"Exact Requirement"- Want it parses as "/opt/solarwinds/utilities/check_file.sh -e -f /var/log/audit/audit.log"

Details :
Inputs.conf for monitoring stanza
[monitor:///var/log]
whitelist=(.log|log$|messages|.out)
blacklist=(lastlog)
index = unix
disabled = 0

Props.conf Details:
[linux_secure]

Event extractions by type

REPORT-0authentication_for_linux_secure = ssh-login-events, ssh-session-close, ssh-disconnect, etc
REPORT-routing = iptables

Kindly guide me how to parse entire command line information in the command field.

thanks in advance.

Re: How to parse complete command line information into the event field ?

micahkemp — Wed, 13 Dec 2017 14:19:31 GMT

Your transforms.conf details didn't come through. Please edit your question to include it, and you may also want to use the code syntax to make it easier to read (using the 101010 button above the text input).

Re: How to parse complete command line information into the event field ?

Hemnaath — Wed, 13 Dec 2017 15:21:15 GMT

Hey micahkemp, thanks for your effort on this, I have updated the question with the Partial transforms.conf details. I am not sure how to make use of the code syntax to easier to read using 101010 button.

Kindly guide me on how to parsing issue.
thanks in advance.

Re: How to parse complete command line information into the event field ?

somesoni2 — Wed, 13 Dec 2017 15:48:25 GMT

Add this line to your props.conf entry (assuming that file is getting deployed to your search head)

[linux_secure]
..other existing entries...
EXTRACT-command = COMMAND\=(?<COMMAND>.+)$

FYR, to format the code, you can either select the code and press Ctrl+K, or, as @micahkemp said, 101010 button above the text box.

Re: How to parse complete command line information into the event field ?

DalJeanis — Wed, 13 Dec 2017 15:58:05 GMT

In order to fulfill this one, you are going to have to determine under what circumstances that COMMAND= appears, so we can help you establish what kind of regex can pull out the correct value. Is it only for sudo? is COMMAND= always the last key value? Does the real value never include a semicolon?

If so, then something like this should pull out what you need...

 | rex   "sudo:.*?COMMAND=(?<COMMAND>[^;]+)(;|$)"

Test that, and if it works as desired, then we can help you figure out where to drop it into your config.

Re: How to parse complete command line information into the event field ?

Hemnaath — Wed, 13 Dec 2017 16:04:05 GMT

Hi Somesoni2, thanks for your effort on this, hey props and transforms are placed in both indexer instances/ search head and inputs.conf is placed in the remote nodes.

So I need to update it like this right in props.conf

[linux_secure]

EXTRACT-command = COMMAND\=(?.+)$

So shall I push the changes in both the indexer instance as well as search head master instances.
Kindly guide me please.

thanks in advance.

Re: How to parse complete command line information into the event field ?

somesoni2 — Wed, 13 Dec 2017 16:08:45 GMT

Based on configurations you've defined in props.conf and transforms.conf (Only search time field extraction related settings are present), they are not required to be deployed on Indexers. Go ahead and push it to Search Heads.

Re: How to parse complete command line information into the event field ?

Hemnaath — Thu, 14 Dec 2017 15:25:36 GMT

Hi Dal Jeanis, thanks for your effort on this, Just checked all the events which are getting populated when we execute the below query and found that all the events are having SUDO. I have also tested the regex which you had posted above along with query and it works fine getting the complete command details in the command field.

   index=unix sourcetype=linux_secure COMMAND | rex   "sudo:.*?COMMAND=(?<COMMAND>[^;]+)(;|$)"

Dec 14 15:00:21 test02 sudo: solarwinds : TTY=pts/3 ; PWD=/opt/solarwinds ; USER=root ; COMMAND=/opt/solarwinds/utilities/check_file.sh -e -f /var/log/audit/audit.log

Can I push the below props.conf in the search head cluster environment.

[linux_secure]
 ## Event extractions by type
 REPORT-0authentication_for_linux_secure = ssh-login-events, ssh-session-close, ssh-disconnect, etc
 EXTRACT-command = "sudo:.*?COMMAND=(?[^;]+)(;|$)"

Kindly guide me on this.
thanks in advance.

Re: How to parse complete command line information into the event field ?

Hemnaath — Thu, 14 Dec 2017 17:45:54 GMT

Hi Dal Jeanis, Can you please guide me on the above comment, so that I can push the changes to the search head master.

thanks in advance.

Re: How to parse complete command line information into the event field ?

Hemnaath — Fri, 15 Dec 2017 11:10:36 GMT

Hi Dal Jeanis, Good Morning, can you please guide me on the above comment, so that I can push the props.conf to the search head cluster via deployer.

thanks in advance.

Re: How to parse complete command line information into the event field ?

Hemnaath — Fri, 15 Dec 2017 13:21:38 GMT

Hi All, Can you please guide me on this, I had pushed the above props.conf file into the search head via deployer but no luck still I could see only partial command information present in the command field.

 [linux_secure]
  ## Event extractions by type
  REPORT-0authentication_for_linux_secure = ssh-login-events, ssh-session-close, ssh-disconnect, sshd_authentication_kerberos_success, sshd_authentication_refused, sshd_authentication_tried
  REPORT-account_management_for_linux_secure = useradd, userdel
  REPORT-firewall = ipfw, ipfw-stealth, ipfw-icmp, pf
  REPORT-routing = iptables
  EXTRACT-command = "sudo:.*?COMMAND=(?[^;]+)(;|$)"

When I searched using the above regex its working, I could see the entire command information in command line. But same when I added into the Props.conf for the sourcetype = linux_secure and pushed the changes in splunk it did not work, I am getting the partial command information in Command line field.

Kindly guide me how to parse entire command line information in the command field.

Re: How to parse complete command line information into the event field ?

Hemnaath — Fri, 15 Dec 2017 14:39:13 GMT

HI Dal Jeanis, can you please guide me on this, i am not sure why its not taking the complete command line information in command line field. It worked well when i executed the query in the search head.

 index=unix sourcetype=linux_secure COMMAND | rex   "sudo:.*?COMMAND=(?<COMMAND>[^;]+)(;|$)"

12/15/17
9:34:31.000 AM  
Dec 15 09:34:31 test01 sudo: solarwinds : TTY=pts/2 ; PWD=/opt/solarwinds ; USER=root ; COMMAND=/opt/solarwinds/utilities/check_file.sh -e -f /var/log/audit/audit.log

But the same regex did not work when we pushed from props.conf with below stanza added and deployed to the search head cluster members.

[linux_secure]
EXTRACT-command = "sudo:.*?COMMAND=(?[^;]+)(;|$)"

Please guide me on this.
thanks in advance.

Re: How to parse complete command line information into the event field ?

Hemnaath — Fri, 15 Dec 2017 16:46:41 GMT

hi somesoni2, can you please guide me on this, i am not sure why its not taking the complete command line information in command line field(interesting field) . It worked well when executed the below query in the search head.

 index=unix sourcetype=linux_secure COMMAND | rex   "sudo:.*?COMMAND=(?<COMMAND>[^;]+)(;|$)"

12/15/17
9:34:31.000 AM

Dec 15 09:34:31 test01 sudo: solarwinds : TTY=pts/2 ; PWD=/opt/solarwinds ; USER=root ; COMMAND=/opt/solarwinds/utilities/check_file.sh -e -f /var/log/audit/audit.log

But the same regex did not work when pushed from props.conf with below stanza added and deployed to the search head cluster members.

 [linux_secure]

   EXTRACT-command = "sudo:.*?COMMAND=(?[^;]+)(;|$)"

"Problem"- COMMAND is currently parsed as "/opt/solarwinds/utilities/check_file.sh"

"Exact Requirement"- Want it parses as "/opt/solarwinds/utilities/check_file.sh -e -f /var/log/audit/audit.log"

Please guide me on this .... I am not sure where its going wrong.

Re: How to parse complete command line information into the event field ?

micahkemp — Wed, 20 Dec 2017 13:47:55 GMT

I don't think you need the double quotes in your EXTRACT line.

Re: How to parse complete command line information into the event field ?

Hemnaath — Thu, 21 Dec 2017 11:27:44 GMT

Hi micahkemp, it worked after removing the double quotes from the Extract command.

micahkemp, I am finding difficulties in Field Extraction (Parsing), I tried learning the Under Props.conf documentation -- Field Extraction but could not understand it, so could please let me know is there way where I can get an Video on Field Extraction (parsing). If you had come across any good video explaining on Field Extraction both Index time and Search time , please share the link with me.

thanks in advance .

Re: How to parse complete command line information into the event field ?

micahkemp — Thu, 21 Dec 2017 11:36:59 GMT

I don't have links to any videos or other trainings, other than the free fundamentals course, which may not have field extraction content. You might want to post another question asking for links to free training options.

Re: How to parse complete command line information into the event field ?

DalJeanis — Thu, 21 Dec 2017 15:37:07 GMT

@hemnath - remove the quotes from around the regex string. In a conf file, you don't need those quotes.