About mbasharat

mbasharat

Hi, I have very large dataset that appears as multivalued as below: | makeresults | eval data1="Windows_7,Unknown,Windows_2012,Windows_7,Windows_8,Windows_10" | eval data2="LAPTOP PC,SERVER,APPLIANCE,DESKTOP,ROUTER,SWITCH" | makemv delim="," data1 | makemv delim="," data2 When I try to use mvexpand using below technique, it exceeds default memory limit of 500MB. I did increase it to 5000MB but problem remains. I did try to limit _raw and multiple other techniques but to no avail. I can not use stats because cardinality is high as I have more fields, ~ 8 fields to expand. Below is a great solution which I have been using at many places for smaller dataset but not for larger dataset. For example, my saved search produces ~6000 records. When those records are to be expanded, I am expecting them to be around 30,000 rows. I need output to be like below but without mvexpand so we do not have to worry about mvexpand memory limits yet still get complete dataset as needed. Is this possible? | makeresults | eval data1="Windows_7,Unknown,Windows_2012,Windows_7,Windows_8,Windows_10" | eval data2="LAPTOP PC,SERVER,APPLIANCE,DESKTOP,ROUTER,SWITCH" | makemv delim="," data1 | makemv delim="," data2 | eval Asset_Category=mvrange(0,mvcount(data1)) | mvexpand Asset_Category | eval data1=mvindex(data1,Asset_Category) | eval data2=mvindex(data2,Asset_Category) Thanks in-advance!!!!

mbasharat

Hi, I have below sample dataset. This dataset is for an asset being compliant or not compliant. What I need is: If an asset is tagged as compliant for some values and non compliant for some, I need non compliant to take precedence and show it as non compliant. Right now, I am seeing all values for example for asset1. If an asset is compliant only, it stays as Compliant. If an asset is NA only, it stays as Compliant. Is: To Be: Thanks in-advance!!!

mbasharat · ‎02-12-2021

Hi @ bowesmana, To test your "*" suggestion, I created another job that used below eval to create another field where % is replaced by * and output to a new lookup table which I then used to create new lookup definition. | eval TEXT_Updated = replace (TEXT, "%", "*") So all the Test value below became like this: *Cas is required to perform this test.* *Cas is required* *Cas is required * this test.* * Match Type as WILDCARD(TEXT_Updated) Then I adjusted and re-ran below query index=sample sourcetype=sample_vuln TEXT IN ("*Cas is required*") | stats count by TEXT | lookup sample_rules_updated TEXT_Updated AS TEXT OUTPUT TEXT_Updated, Priority Result was: TEXT=Cas is required to perform this test. TEXT_Updated=* At least a value (*) appeared this time. S o it did not match. Feedback pls?

mbasharat · ‎02-12-2021

Hi, I have scan dataset. It has a field name TEXT. This field contains the data for test results. I am provided a data from SQL table which I am ingesting via DBConnect. This also contains TEXT field but along with Priorities to assign to. I am using data from SQL to create a saved search that outputs into a lookup. My issue is that the data that is in the lookup has TEXT field with wildcard (%). I am not able to match what is in the raw data with what is in the lookup because of wildcard. Sample: In raw data: TEXT= Cas is required to perform this test. In lookup: TEXT= %Cas is required to perform this test.% %Cas is required% %Cas is required % this test.% % Test search: index=sample sourcetype=sample_vuln TEXT IN ("*Cas is required*") | stats count by TEXT | lookup sample_rules TEXT AS TEXT OUTPUT TEXT AS TEXT_Test, Priority How can I match the data in raw with the data in lookup containing same field values with wildcards in above combinations. I did edited the lookup's definition and configured Match Type option as WILDCARD(TEXT) and this is not helping. Thanks in advance!!!

mbasharat · ‎02-02-2021

Hi, I have vulnerability dataset. Each vulnerability comes with a score from the scanning tool. Scanning tool has its own calculation and assigns a severity based on that. We on customer side, calculate Severity based on customer defined score ranges below: 9.0-10.0 > CRITICAL 7.0-8.9 > HIGH 4.0-6.9 > MEDIUM 0.1-3.9 > LOW 0.0 > NONE Issue is that when data comes from source/scanning tool, it has its own severities which are not always lined up with the above ranges. Our Score ranges above is the main root guideline to use. Example: Often times, Severity from data does not match the Score that is passed by the tool as I mentioned above. A Severity of MAJOR in data coming with a Score of 3.0. A Severity of MINOR in data coming with a Score of 3.0. A Severity of CRITICAL in data coming with a Score of 0.0. A Severity of CRITICAL in data coming with a Score of 10.0 (This is correct and inline with our ranges above) I need both of the options below: Desired output 1 (based on score ranges): SEVERITY_Data Score_Data Severity_Adjusted_Score Severity_Adjusted_Code MAJOR 3.0 Median of 0.1-3.9 LOW Desired output 2 (based on SEVERITY_Data e.g. value is MAJOR): SEVERITY_Data Score_Data Severity_Adjusted_Score Severity_Adjusted_Code MAJOR 3.0 Median of 7.0-8.9 HIGH Likewise for the rest of the severities and score ranges. Thanks in advance!!!

mbasharat · ‎02-02-2021

@ scelikok, Doing some testing so will respond back shortly.

mbasharat · ‎01-29-2021

Hi @scelikok, Append has max results limit of 50,000. I have a lot more than this. Any join option? Thanks.

mbasharat · ‎01-28-2021

Re-initiation of an older question I had asked: Hi, I have a need for an alternative of | lookup abc field1 AS field2 OUTPUT field1, fieldA, fieldB, fieldC. For above, I have a lookup definition from a lookup that holds information about more than 50,000 vulnerabilities. I am using this lookup definition in my queries and result set is no more than 1000. 1000 is the maxmatch limit of lookup definition that Splunk supports. I need an alternative e.g. a subsearch using lookup itself or anything that allows me to do match for all the values in my lookup which is approximately 50,000 on average as efficiently as possible. Sample query (original query is much longer but I will be using your provided solution to consolidate) index=ABC sourcetype="XYZ" `comment (This is to reduce Splunk's internal fields to keep my table size smaller)` | fields - index, source, sourcetype, splunk_server, splunk_server_group, host, eventtype, field, linecount, punct, tag, tag::eventtype, _raw `comment (This is to limit to the only fields which I need)` | fields dns, vuln_id `comment (vuln_id is a multivalued field and I have to separate them to get accurate stats. When stats is run, it takes care of expanding them and it works as expected)` | makemv delim="," vuln_id | stats count by vuln_id, dns | lookup vuln_info VulnID AS vuln_id OUTPUT Scan_Type, OS, Environment The below approach is what I have tried that is not returning anything but it should. I am missing something in this: index=ABC sourcetype="XYZ" | fields - index, source, sourcetype, splunk_server, splunk_server_group, host, eventtype, field, linecount, punct, tag, tag::eventtype, _raw | fields dns, vuln_id | makemv delim="," vuln_id | stats count by vuln_id, dns [| inputlookup vuln_info.csv | fields VulnID, Scan_Type, OS, Environment | rename VulnID as vuln_id] Any solution that will work as efficiently as possible to get all records from lookup instead of incomplete dataset due to lookup definition's maxmatch limit of 1000 in Splunk. Thanks in-advance!!!

mbasharat · ‎11-25-2020

I tried many things with up until Splunk 7.x. Finally, we upgraded to 8.0.6 (8.x) and below worked. No custom stylesheets etc. needed. Just use the below code and adjust for your need as needed. If it works for you all, don't forget to thumbs up. Thanks. <form theme="light"> <label>Sample Text Box Sizing</label> <fieldset submitButton="true" autoRun="false"> <input id="splunk_input_text_1" type="text" token="text1" searchWhenChanged="false"> <label>2 values</label> <default>A,B</default> </input> <input id="splunk_input_text_2" type="text" token="text2" searchWhenChanged="false"> <label>5 values</label> <default>A,B,C,D,E</default> </input> <input id="splunk_input_text_3" type="text" token="text3" searchWhenChanged="false"> <label>8 values</label> <default>A,B,C,D,E,F,G,H</default> </input> <html> <style> #splunk_input_text_1{ width: 200px !important; } </style> </html> <html> <style> #splunk_input_text_2{ width: 300px !important; } </style> </html> <html> <style> #splunk_input_text_3{ width: 600px !important; } </style> </html> </fieldset>

mbasharat · ‎11-10-2020

mbasharat · ‎11-06-2020

Hi bowesmana, How to consolidate an additional field for netbiosDomain in your below eval please? | makeresults | eval netbiosName="ABC\HOST" | eval netbiosHost=replace(netbiosName,"[^\\\]*\\\(.*)","\1") | eval netbiosDomain=??? I have also tried below and this one works as well. However, which one will be a better approach? | eval Split=split(netbiosName,"\\") | eval SplitHost=mvindex(Split,1) | eval SplitDomain=mvindex(Split,0) By using * in place of UNKNOWN as you suggest produces below error: Regex: UTF-8 error: isolated byte with 0x80 bit set. Can you provide a RegEx option to extract both Domain and Host that I can add in props.conf and have the fields extracted at the backend so I can do my searches and calculations as normal?

mbasharat · ‎11-05-2020

Hi, I am dealing with an issue because data changed from my source. I was using a lookup as below to search only on the hosts that are in my lookup. This field name NETBIOS was always coming as UNKNOWN\samplehost so I did a simple eval and added UNKNOWN\ with host name in lookup query and that worked great. index=source1sample sourcetype="samplesourcetype" [| inputlookup sample.csv | table sample_netbios | eval sample_netbios=upper(sample_netbios) | rename sample_netbios as netbiosName | eval netbiosName="UNKNOWN"."\\". netbiosName] Now, the data has changed in a way that I am seeing domain coming from data source in netbiosName field e.g. ABC\host1 XYZ\host2 How can I structure a search where I can filter upfront as in search above regardless of the domain value that come in? I can get rid of the "\" but this means that I will have to eval or rex before everything and then do a match which takes a toll on query performance. My query was taking only about 10 seconds for ~5k hosts matching from lookup to index but aforementioned way cause it to run for ~ 20 mins because it has to go thru all hosts and then do a match on the ones in lookup. Thank in-advance!!!

mbasharat · ‎11-04-2020

Hi Rich, The issue is that users are creating clones of KOs and due to the lack of their training and knowledge, they try tweaking the cloned KOs in an un-optimized manner which is causing heavy load on infrastructure. We are asked by management to shutdown the cloning for users with read only access so they are unable to take clones to begin with. Thanks.

mbasharat · ‎11-03-2020

Hi, I am dealing with a situation where many users in Production are taking clones of knowledge objects e.g. scheduled reports, views etc. How to prevent this? Any settings in the configurations and roles capabilities I can apply to prevent this? Our configuration settings for access controls has read only for users. Write is for developers and admins only. This still does not prevent users from taking clones. Thanks in advance!!!

mbasharat · ‎10-14-2020

Perfect. Thank you!!!!

mbasharat · ‎10-14-2020

Hi ITWhisperer, So first solution produces results and I am testing them. I tried using the second one as well to test as well and it is giving error that mvmap is unsupported or undefined. I am @ Splunk Enterprise 7.x. Thanks.

mbasharat · ‎10-14-2020

Hi, I have data in XML format. Out of many fields that I have extracted, there is another field name pluginText which is in below format. I need to have some fields extracted from below. I need below two fields. Also, if there is a rex I can use to extract all fields in below tags using a universal logic, that will be great. Thanks in-advance!!! Nessus version Plugin feed version See sample below: pluginText: <plugin_output>Information about this scan : Nessus version : 7.6.3 Plugin feed version : 202010122335 Scanner edition used : Sample Scan type : Windows Agent Scan policy used : Windows_Server_2019 Scanner IP : 0.0.0.0 Thorough tests : no Experimental tests : no Paranoia level : 1 Report verbosity : 1 Safe checks : yes Optimize the test : yes Credentialed checks : yes Patch management checks : None Display superseded patches : yes (supersedence plugin did not launch) CGI scanning : disabled Web application tests : disabled Max hosts : Max checks : 5 Recv timeout : 5 Backports : None Allow post-scan editing: Yes Scan duration : unknown </plugin_output>

mbasharat · ‎09-30-2020

dedup does not work for me because I am dealing with a lot of data and stats runs the fastest. The drilldown dashboard is also used as an individual dashboard. USing your approach, tokens do get transferred now. The problem is if a value A is selected on dashboard A, filter logic is passing on value C or D. There is something in |n logic or something else causing this which I am not getting.

mbasharat · ‎09-29-2020

Hi @ ITWhisperer, So I used your method below, made progress and multiselect started passing the value from multiselect filter from dashboard A and drilldown stats table panel to filters in dashboard B. Now the issue is, the value that is selected on dashboard A is not the one that is getting transferred to dashboard B. A different value is getting passed which seems odd because the selected one needs to be passed over. Here is my code snippet for the filters: Dashboard A: <input type="dropdown" token="scantype" searchWhenChanged="false"> <label>Scan Type</label> <fieldForLabel>Scan_Type</fieldForLabel> <fieldForValue>Scan_Type</fieldForValue> <search base="menus"> <query>| stats dc(Scan_Type) AS Total by Scan_Type | fields - Total</query> </search> <default>*</default> <initialValue>*</initialValue> <choice value="*">All</choice> </input> <input type="multiselect" token="resporg" searchWhenChanged="false"> <label>Responsible Organization</label> <fieldForLabel>Responsible_Organization</fieldForLabel> <fieldForValue>Responsible_Organization</fieldForValue> <search base="menus"> <query>| search Scan_Type="$scantype$" | stats dc(Responsible_Organization) AS Total by Responsible_Organization | fields - Total</query> </search> <default>*</default> <initialValue>*</initialValue> <choice value="*">All</choice> <prefix>Responsible_Organization IN (</prefix> <suffix>)</suffix> <valuePrefix>"</valuePrefix> <valueSuffix>"</valueSuffix> <delimiter>,</delimiter> <change> <eval token="drilldownmultiorg">mvjoin('form.resporg',"&form.drilldownmultiorg=")</eval> </change> </input> Stats Panel A: <drilldown> <link target="_blank">/app/sample_app/drilldown_dashboard?form.cves=$row.CVE$&form.scantype=$scantype$&form.resporg=$drilldownmultiorg|n$</link> </drilldown> Dashboard B: <input type="text" token="cves"> <label>Select CVE</label> <default>CVE-2015-6550</default> <initialValue>CVE-2015-6550</initialValue> </input> <input type="multiselect" token="plugin" searchWhenChanged="false"> <label>Plugin ID/s</label> <choice value="*">All</choice> <default>*</default> <initialValue>*</initialValue> <fieldForLabel>Plugin_ID</fieldForLabel> <fieldForValue>Plugin_ID</fieldForValue> <search base="vulninfo"> <query>| search CVE="$cves$" | stats dc(Plugin_ID) AS count BY Plugin_ID | fields Plugin_ID</query> </search> <prefix>Plugin_ID IN (</prefix> <suffix>)</suffix> <delimiter>,</delimiter> <valuePrefix>"</valuePrefix> <valueSuffix>"</valueSuffix> </input> <input type="dropdown" token="scantype" searchWhenChanged="false"> <label>Scan Type</label> <choice value="*">All</choice> <default>*</default> <initialValue>*</initialValue> <fieldForLabel>Scan_Type</fieldForLabel> <fieldForValue>Scan_Type</fieldForValue> <search base="vulninfo"> <query>| search CVE="$cves$" AND $plugin$ | stats dc(Scan_Type) AS count BY Scan_Type | fields Scan_Type</query> </search> </input> <input type="multiselect" token="drilldownmultiorg" searchWhenChanged="false"> <label>Responsbile Organization</label> <choice value="*">All</choice> <fieldForLabel>KISAM_Responsible_Organization</fieldForLabel> <fieldForValue>KISAM_Responsible_Organization</fieldForValue> <search base="vulninfo"> <query>| search CVE="$cves$" AND $plugin$ AND Scan_Type="$scantype$" | stats dc(KISAM_Responsible_Organization) AS count BY KISAM_Responsible_Organization | fields KISAM_Responsible_Organization</query> </search> <prefix>KISAM_Responsible_Organization IN (</prefix> <suffix>)</suffix> <valuePrefix>"</valuePrefix> <valueSuffix>"</valueSuffix> <delimiter>,</delimiter> <initialValue>*</initialValue> <default>*</default> </input>

mbasharat · ‎09-27-2020

Hi @ ITWhisperer, Sorry, I was away from home. Will be posting my code snippet for filters today. Thanks!

Posts	202
Solutions	4
Karma Given	11
Karma Received	11
Member Since	‎01-10-2018

Online Status	Offline
Date Last Visited	an hour ago

mvexpand memory issue

Multivalue fields match issue

Issues with lookup matching on field values

Problem with ranges calculation in search

Incomplete search results because of lookup defini...

Field extraction and search issue

How to prevent users taking clones of knowledge ob...

Field extraction needed

Name Extraction needed

Increasing size of textbox input

mvexpand memory issue

Multivalue fields match issue

Re: Issues with lookup matching on field values

Issues with lookup matching on field values

Problem with ranges calculation in search

Re: Incomplete search results because of lookup de...

Re: Incomplete search results because of lookup de...

Incomplete search results because of lookup defini...

Re: How to increase the textbox size when you want...

Re: Field extraction and search issue

Re: Field extraction and search issue

Field extraction and search issue

Re: How to prevent users taking clones of knowledg...

How to prevent users taking clones of knowledge ob...

Re: Field extraction needed

Re: Field extraction needed

Field extraction needed

Re: How to pass multiselect token values from main...

Re: How to pass multiselect token values from main...

Re: How to pass multiselect token values from main...