- Find Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- How do I search field extraction for table?
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I have below scenario. Image_Name and Name_Space are being ingested with below variations in table A. Image_name is a multivalued field as shown. I tried using makemv delim but it doesnt work because there is no delimiter e.g. space between the two. I need them separated out as in table B. Thanks in advance!
Table A:
|
Image_Name |
Name_Space |
|
<none> c-ecm-dev/das-dynamic-filter-services |
c-ecm-dev |
|
<none> |
cs-webapps-sat |
|
NULL |
NULL |
|
NULL |
c-aoic-dev |
|
c-ecm-dev/das-dynamic-filter-services c-ecm-sat/irtf-das-service |
c-ecm-sat |
|
c-ecm-dev/das-dynamic-filter-services cpopen/ibm-watson-speech-catalog |
openshift-marketplace |
|
c-ecm-sbx/das-pay-gov-services iam-essar-aqt1/iam-essar-aqt1 |
NULL |
|
c-ecm-sbx/das-rendering-service |
sysdig |
|
cs-webapps-sbx/baldue-bwas c-ecm-dev/das-rendering-service |
c-ecm-dev |
Table B:
|
Image_Name |
Name_Space |
|
<none> |
c-ecm-dev |
|
c-ecm-dev/das-dynamic-filter-services |
c-ecm-dev |
|
<none> |
cs-webapps-sat |
|
NULL |
NULL |
|
NULL |
c-aoic-dev |
|
c-ecm-dev/das-dynamic-filter-services |
c-ecm-sat |
|
c-ecm-sat/irtf-das-service |
c-ecm-sat |
|
c-ecm-dev/das-dynamic-filter-services |
openshift-marketplace |
|
cpopen/ibm-watson-speech-catalog |
openshift-marketplace |
|
c-ecm-sbx/das-pay-gov-services |
NULL |
|
iam-essar-aqt1/iam-essar-aqt1 |
NULL |
|
c-ecm-sbx/das-rendering-service |
sysdig |
|
cs-webapps-sbx/baldue-bwas |
c-ecm-dev |
|
c-ecm-dev/das-rendering-service |
c-ecm-dev |
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Perhaps I didn't explain my (and all these volunteers') difficulty with your explanation clearly enough. It has been clear early on that one or more of your events can have field values like the following.
| namespace | imageName |
| c-esm-sat | c-ecm-dev/das-dynamic-filter-services/sample (1) c-ecm-sat/irtf-das-service (2) |
(I'm using parenthesized numerals to indicate two different values.)
The problem is, you have never demonstrated a raw event has properties like that for which @VatsalJagani's mvexpand should not give you desired result, if you correct for the field name that was mistaken in your original description. After field name correction,
| mvexpand imageName
You also have not demonstrated the output of this command IF your events truly have multivalued imageName. Please note: the phrase "did not work" conveys little useful information and should be avoided in the best of scenarios, much less to volunteers with no intimate knowledge about your dataset.
In addition, just because Splunk's stats table displays a field like
c-ecm-dev/das-dynamic-filter-services/sample
c-ecm-sat/irtf-das-service
i.e., newline-delimited strings, the field is not necessarily multivalued. This is why everyone here insists that you demonstrate raw events with such characteristics. The events you have illustrated so far are made of text key-value pairs. It is not even possible for Splunk to give you any multivalued field with such constructs. (Unless you have some sort of secondary extraction somewhere else that is unbeknown to volunteers here. If you do, you must also explain those.)
Have you considered the possibility that imageName could be single valued multiline texts? Maybe you can try
| eval imageName = split(imageName, "
")
| mvexpand imageName
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This was an interesting and valuable discussion. Thanks.
I do genuinely appreciate volunteers and the willingness of people to give their time and experience/expertise to help, and if there is genuine interest in an exchange--versus simply trying to off-load doing work onto other people.
I hope in all of my interactions here I am helped, can help, and can be taught.
Cheers.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@mbasharat - It's possible if its already a multi-valued field and that's why you are seeing them like that. If so, try below at the end of your search and see if that works:
| mvexpand Image_Name
Loose nothing in trying out!!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As @gcusello mentioned kindly share the _raw events so we can guide you.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @mbasharat,
to help you, I need the raw events, could you share them?
anyway, probably with a regex it should be possible to separate values.
Ciao.
Giuseppe
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi folks,
See below 4 samples. Field names are namespace and imageName in the events. Much appreciated!!!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Gi @VatsalJagani,
ok, in each event there's one value for both the fields, so you have to use stats and mvexpand commands, something like this:
<your_search>
| stats values(Image_Name) AS Image_Name BY Name_Space
| mvexpand Image_Name
| table Image_Name Name_SpaceCiao.
Giuseppe
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This gets even more confusion. Exemplified data show no multivalued imageName in any event.
| imageName | namespace |
| raas/jggmb/graph-analysis | NULL |
| raas/cdw-api | xyz |
| raas/bpa-lab/rapidsai-22.08-cuda11.4-centos7-py3.8 | NULL |
| postgres | abcd |
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi yuanlu,
Yes it does. See below from Event #3:
imageName="raas/jggmb/graph-analysis"
Which is actually needed as below since there can be multiple imageName in each namespace separated by /:
namespace imageName
NULL raas
NULL iggmb
NULL graph-analysis
The challenge is not the /, it is that imageName can be multivalued field as below which I had mentioned in the very first post sample. The issue is the mv because there is no delimiter between the first and second value of mv as shown below:
namespace imageName
c-esm-sat c-ecm-dev/das-dynamic-filter-services/sample
c-ecm-sat/irtf-das-service
And this needs to be first extracted as below:
namespace imageName
c-esm-sat c-ecm-dev/das-dynamic-filter-services/sample
c-esm-sat c-ecm-sat/irtf-das-service
And final results are to be:
namespace Environment imageName
c-esm-sat c-ecm-dev /das-dynamic-filter-services
c-esm-sat c-ecm-dev /sample
c-esm-sat c-ecm-sat /irtf-das-service
Also, the first group is the Environment as I highlighted in red above. I am not worried about Environment because if I can have the value 1 and value 2 separated, It can them delimit Environment easily. I hope I explained better this time.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes it does. See below from Event #3:
imageName="raas/jggmb/graph-analysis"
...
The challenge is not the /,
In other words, you are aware that Splunk doesn't give you multiple values from extracting imageName="raas/jggmb/graph-analysis". Is this correct? To get three values based on that separator "/", you need additional processing such as split.
it is that imageName can be multivalued field as below which I had
mentioned in the very first post sample. The issue is the mv because there is no delimiter between the first and second value of mv as shown below:
namespace imageName
c-esm-sat c-ecm-dev/das-dynamic-filter-services/sample
c-ecm-sat/irtf-das-service
And this needs to be first extracted as below:
namespace imageName
c-esm-sat c-ecm-dev/das-dynamic-filter-services/sample
c-esm-sat c-ecm-sat/irtf-das-service
Here is the problem: You keep insisting that imageName has multiple values such as (c-ecm-dev/das-dynamic-filter-services/sample, c-ecm-sat/irtf-das-service), but you haven't demonstrated a single event that can be extracted as multivalued into (c-ecm-dev/das-dynamic-filter-services/sample, c-ecm-sat/irtf-das-service). @VatsalJagani's first reply already informed that IF field imageName has multiple values, mvexpand will take care of your requirement, i.e.,
| mvexpand imageNameThe only difference is that @VatsalJagani followed the original post's incorrect field name.
As far as SPL is concerned, it is impossible for mvexpand to not behave as you desired IF this field has multiple values. But if your so-called multivalued field has to come from separating parts delimited by "/", you can add a split, followed by mvexpand.
| eval imageName = split(imageName, "/")
| mvexpand imageNameHowever, if you continue to see events in which imageName has multiple values without performing additional commands such as split, AND that mvexpand does not give you multiple events corresponding to each of those values, you must first demonstrate at least ONE such event. Otherwise this is a waste of volunteers' time.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi yuanliu,
I did try that. See my notes in parenthesis. I will try to explain again.
namespace imageName
c-esm-sat c-ecm-dev/das-dynamic-filter-services/sample
(there is no space nor any delimiter between the 1st value at the top and the 2nd one below)
c-ecm-sat/irtf-das-service
Final results need to be:
namespace Environment imageName
c-esm-sat c-ecm-dev /das-dynamic-filter-services
c-esm-sat c-ecm-dev /sample
c-esm-sat c-ecm-sat /irtf-das-service
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Perhaps I didn't explain my (and all these volunteers') difficulty with your explanation clearly enough. It has been clear early on that one or more of your events can have field values like the following.
| namespace | imageName |
| c-esm-sat | c-ecm-dev/das-dynamic-filter-services/sample (1) c-ecm-sat/irtf-das-service (2) |
(I'm using parenthesized numerals to indicate two different values.)
The problem is, you have never demonstrated a raw event has properties like that for which @VatsalJagani's mvexpand should not give you desired result, if you correct for the field name that was mistaken in your original description. After field name correction,
| mvexpand imageName
You also have not demonstrated the output of this command IF your events truly have multivalued imageName. Please note: the phrase "did not work" conveys little useful information and should be avoided in the best of scenarios, much less to volunteers with no intimate knowledge about your dataset.
In addition, just because Splunk's stats table displays a field like
c-ecm-dev/das-dynamic-filter-services/sample
c-ecm-sat/irtf-das-service
i.e., newline-delimited strings, the field is not necessarily multivalued. This is why everyone here insists that you demonstrate raw events with such characteristics. The events you have illustrated so far are made of text key-value pairs. It is not even possible for Splunk to give you any multivalued field with such constructs. (Unless you have some sort of secondary extraction somewhere else that is unbeknown to volunteers here. If you do, you must also explain those.)
Have you considered the possibility that imageName could be single valued multiline texts? Maybe you can try
| eval imageName = split(imageName, "
")
| mvexpand imageName
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Well said!
Perhaps it is my age, but I find that my capacity for tolerance has diminished over time - I only have a certain amount of time to give (voluntarily) to answering Splunkers' questions, and those Splunkers who provide sufficient information are those who are most easy to aid. I applaud those in the community with sufficient capacity to continue with attempts to assist those in need! 😵🤓😎
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @ITWhisperer
Patience is a virtue and tolerance is the by product of it including many other things. Specially when the trunk has a sticker "Student driver"! 😁 Btw, I did explain everything well, however @yuanliu added a little more clarity with mentioning (1) and (2).
Appreciate ya' all as always!!! Don't lose patience 😋😜
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @gcusello
See few raw samples below. Field names are imageName and namespace. I have everything coming normalized except imageName which need to be split up inparallel with namespace the way I have provided in Table B.
Sample1:
07/13/2023 17:55:05 +0000, search_name="Sample", search_now=1689271860.000, info_min_time=1686614400.000, info_max_time=1689271894.612, info_search_time=1689271892.776, IP="1.2.3.4", OS="Red Hat Enterprise Linux CoreOS 4.11", DNS=sampledns1, GSS="sample1", qid=241759, AO="user1.com", cveids="CVE-2023-1667 CVE-2023-2283", result="#table cols=\"3\"
Package Installed_Version Required_Version
libssh-config 0.9.6-3.el8.noarch 0.9.6-10.el8__8
libssh 0.9.6-3.el8.x86__64 0.9.6-10.el8__8", imageId=05ac522d3e87, isDrift=false, CATEGORY=SERVER, ISSO="sampleisso1", PROJECTS=NULL, hostname=sampledns1, imageSha=0000000000000000000000000000000000000000, os_group=OTHER, LAST_SEEN="2023-07-12T20:18:15Z", imageName="<none>", imageUuid="0000000000000000000000", namespace="c-ecm-dev", vulnTitle="Red Hat Update for libssh (RHSA-2023:3839)", containerState=RUNNING, softwareFixVersion="0.9.6-10.el8__8", PRJ_GROUP_EMAIL="projemail1@sample.com", Business_Group=UNASSIGNED, vulnFirstfound="2023-07-13T06:35:03Z", imageScanType=null, POC_EMAIL="poc1@sample.come", cvss3TemporalScore="5.9", SYSTEMNAME=NULL, RESPONSIBILITY_CODE="sample respcode", vulnLastfound="2023-07-13T06:35:03Z", cvss3BaseScore="6.5", AO_EMAIL="sampleemail.com", POC_NAME="sample user", PRJ_NAME=ABC, Severity=Moderate, Environment=DEV, containerId=123456789, ISSO_EMAIL="sample3.com", containerSha=000000000000000000000000000000000000000000, softwareVersion="0.9.6-3.el8", softwareName="libssh libssh-config", vulnCategory=RedHat, vulnSolution="Refer to Red Hat security advisory <A HREF='https://access.redhat.com/errata/RHSA-2023:3839' TARGET='_blank'>RHSA-2023:3839</A> for updates and patch information.
<P>Patch:<BR>
Following are links for downloading patches to fix the vulnerabilities:
<P> <A HREF='https://access.redhat.com/errata/RHSA-2023:3839' TARGET='_blank'>RHSA-2023:3839:Red Hat Enterprise Linux</A>", containerCreated="2023-07-13T06:31:14Z", containerUpdated="2023-07-13T06:35:03Z"
Sample2:
07/14/2023 11:39:39 +0000, search_name="sample", search_now=1689336660.000, info_min_time=1686700800.000, info_max_time=1689336695.166, info_search_time=1689336692.365, IP=NULL, OS=NULL, DNS=NULL, GSS=NULL, qid=500500, AO=NULL, cveids="CVE-2022-0778", result="#table cols=\"3\"
Package Installed_Version Required_Version
libcrypto1.1 1.1.1k-r0 1.1.1n-r0
libssl1.1 1.1.1k-r0 1.1.1n-r0", imageId=24ae535b6904, isDrift=false, CATEGORY=NULL, ISSO=NULL, PROJECTS=NULL, hostname=samplehost, imageSha=000000000000000000000000000000, os_group=NULL, LAST_SEEN=NULL, imageName="c-ecm-dev/mtrdb-integration", imageUuid="000000000-00000000000000000-000000000000000", namespace="sysdig-sdc-cli", vulnTitle="Alpine Linux Security Update for Open Secure Sockets Layer (OpenSSL)", containerState=RUNNING, softwareFixVersion="1.1.1n-r0", PRJ_GROUP_EMAIL=NULL, Business_Group=UNASSIGNED, vulnFirstfound="2023-07-09T18:18:06Z", imageScanType=null, POC_EMAIL=NULL, cvss3TemporalScore="6.7", SYSTEMNAME=NULL, RESPONSIBILITY_CODE=NULL, vulnLastfound="2023-07-09T18:18:06Z", cvss3BaseScore="7.5", AO_EMAIL=NULL, POC_NAME=NULL, PRJ_NAME=NULL, Severity=High, Environment=NULL, containerId=123456789, ISSO_EMAIL=NULL, containerSha=000000000000000000000000000000000000, softwareVersion="1.1.1k-r0", softwareName="libcrypto1.1 libssl1.1", vulnCategory="Alpine Linux", vulnSolution="Refer to Alpine Linux advisory <A HREF='https://security.alpinelinux.org/srcpkg/openssl' TARGET='_blank'>openssl</A> for updates and patch information.
<P>Patch:<BR>
Following are links for downloading patches to fix the vulnerabilities:
<P> <A HREF='https://security.alpinelinux.org/srcpkg/openssl' TARGET='_blank'>openssl-1.1.1n-r0:Alpine Linux</A>", containerCreated="2023-07-09T18:13:48Z", containerUpdated="2023-07-09T18:18:06Z"
Sample3:
07/13/2023 17:40:56 +0000, search_name="sample", search_now=1689271860.000, info_min_time=1686614400.000, info_max_time=1689271894.612, info_search_time=1689271892.776, IP="0.0.0.0", OS="Red Hat Enterprise Linux CoreOS 4.11", DNS=sampledns, GSS="samplegss", qid=241757, AO="sampleuser@sample.com", cveids="CVE-2023-26604", result="#table cols=\"3\"
Package Installed_Version Required_Version
systemd-libs 239-68.el8__7.4.x86__64 239-74.el8__8.2", imageId=dcbb6b8e07e2, isDrift=false, CATEGORY=SERVER, ISSO="sample", PROJECTS=NULL, hostname=samplehostname, imageSha=00000000000000000000000000000000, os_group=OTHER, LAST_SEEN="2023-07-12T20:33:03Z", imageName="wi-irps-sat/ir-data-certification-sat
c-ecm-dev/irtf-das-service", imageUuid="0000000000-00000000000-000000000000", namespace="c-ecm-dev", vulnTitle="Red Hat Update for systemd (RHSA-2023:3837)", containerState=RUNNING, softwareFixVersion="239-74.el8__8.2", PRJ_GROUP_EMAIL="sampleuser@sample.com", Business_Group=UNASSIGNED, vulnFirstfound="2023-07-03T10:55:11Z", imageScanType="null
null", POC_EMAIL="sampleuser@sample.com", cvss3TemporalScore="7.0", SYSTEMNAME=NULL, RESPONSIBILITY_CODE="ENTERPRISE CONTAINER", vulnLastfound="2023-07-03T10:55:11Z", cvss3BaseScore="7.8", AO_EMAIL="sampleuser@sample.com", POC_NAME="sample user", PRJ_NAME=ECM, Severity=High, Environment=DEV, containerId=1234564897, ISSO_EMAIL="sampleuser@sample.com", containerSha=0000000000000000000000000000000000, softwareVersion="239-68.el8_7.4", softwareName="systemd-libs", vulnCategory=RedHat, vulnSolution="Refer to Red Hat security advisory <A HREF='https://access.redhat.com/errata/RHSA-2023:3837' TARGET='_blank'>RHSA-2023:3837</A> for updates and patch information.
<P>Patch:<BR>
Following are links for downloading patches to fix the vulnerabilities:
<P> <A HREF='https://access.redhat.com/errata/RHSA-2023:3837' TARGET='_blank'>RHSA-2023:3837:Red Hat Enterprise Linux</A>", containerCreated="2023-07-03T10:51:50Z", containerUpdated="2023-07-03T10:55:11Z"
Enterprise Security Content Update (ESCU) | New Releases
Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?
Index This | What are the 12 Days of Splunk-mas?
