About ak9092

ak9092 · ‎09-21-2021

Just the replace i was looking for. Thanks Much!! @bowesmana

ak9092 · ‎09-20-2021

@PickleRick Yes , there are some events in which it is extracting exactly the way you mentioned. But then there are some event which are coming as a K/V pairs inside another field, So the extraction which I am doing is for the field, where i am getting data like - field1="field2=\"some_string_value\", field3=\"some_path_value\", field4=\"again_some_value\", ... fieldN=\"valueN\"" So now from field1 to field4 everything is getting formatted perfectly fine using EVAL-field = replace(field,"\\\\(.),"") The issue comes in "fieldN" where an extra double quotes is added to close the quotes which is opened at field1. Now the problem is my fieldN can be any any field and can contain multiple back slashes , double quotes and any string, so need some robust solution here which will work fine on any kind of field value. If somehow I can make the below regex work with replace command , then i guess it should solve my problem, but unfortunately its giving me error with something like - "|ev...{snipped} {errorcontext = ce(conn1,[\\"]+],"")}'. https://regex101.com/r/glIfKz/1 |makeresults | eval conn1="\\\"08/24/2021\\\"\"" |eval conn=replace(conn1,[\\"]+],"")

ak9092 · ‎09-20-2021

@ashvinpandey , thank you for your response. I am trying to achieve this using props.conf. Also the data in "conn" field can be a mix of IP , string and other special characters just like the "loc" field which you can see in the question.

ak9092 · ‎09-20-2021

@PickleRick The eval which I am using removes the double quotes too, but its just not removing the additional quotes which is coming in some events. In below run anywhere search for conn2 field you will see one double quote remaining at the end, which I don't want. |makeresults |eval conn="\\\"08/24/2021\\\"\"" |eval conn2=replace(conn,"\\\\(.)","") Also there is no such any specific extraction in my props.conf and it roughly looks something like below [mysourcetype] EVAL-conn = replace(conn,"\\\\(.),"") With following regex, I am able to replace all the backslashes and double quotes the way I want but I am unable to make it work with Splunk replace command. https://regex101.com/r/glIfKz/1 @niketnilay , @ITWhisperer, @bowesmana Thanks in advance.

ak9092 · ‎09-17-2021

@PickleRick thank you for your response. The EVAL statements shared by you and the one which I have shared above both are working fine only if the field is like conn="\success\" , but not working if there is an additional double quotes at the end like conn=\"pass\"". So in both; the results looks like - Conn= pass", what I want is; conn=pass

ak9092 · ‎09-16-2021

Hello, I want to remove all the back slashes and double quotes from following fields - conn=\"pass\"" ip=\"10.23.22.1\"" I am trying to extract with EVAL-conn = replace(conn,"\\\\(.),"") and EVAL-ip= replace(ip,"\\\\(.),"") in my props.conf but it is not removing the last double quotes and give me following results - conn=pass" ip=10.23.22.1" Results I want : conn=pass & ip=10.23.22.1 Can someone please help/guide me with this extraction. Thanks in Advance

ak9092 · ‎09-10-2021

@ITWhisperer Thanks, I tried this but its missing out on some of the fields due to _raw data format being inconsistent. Like for e.g- field report_name its giving me path= but actually that field is empty. Also I am looking to extract this using props & transforms. Thanks.

ak9092 · ‎09-10-2021

@richgalloway Basically I was trying something like - props.conf [my_srctype] REPORT-extract_conn=extract_conn tranforms.conf [extract_conn] REGEX = conn="connmsg=([^,]*)|^conn=(.*) MV_ADD = true FORMAT = conn::$1::$2 But this doesn't seem to be working as REGEX/FORMAT is not extracting two values for a single field. So currently I have figured out an alternative way where I am first extracting the entire conn field and then other key value pairs in following way :- [my_srctype] REPORT-extract_conn=extract_conn,extract_connmsg,extract_user ... [extract_conn] REGEX = conn="(.*)" MV_ADD = true FORMAT = conn::$1 [extract_connmsg] REGEX = conn="connmsg=([^,]*) MV_ADD = true FORMAT = connmsg::$1 [extract_user] REGEX = conn=.*user=([^,]*) MV_ADD = true FORMAT = user::$1 .. and so on... If you can think of or suggest any better and more efficient way to this than i will be happy to try that too. Thanks

ak9092 · ‎09-09-2021

Hi, I am having difficulty in extracting key=value pairs from one of the auto extracted field. The problem is that, this field may contain just a text value but also could contain multiple key=value pairs in it, so whenever there are multiple key=value pairs in the event then I am not getting the desired results. Following are some of my _raw events - 2021-08-10T11:35:00.505 ip=10.1.10.10 id=1 event="passed" model="t1" conn="connmsg=\"controller.conn_download::message.clean\", file=\"/home/folder1/filename_8555c5s.ext\", time=\"21:22:02\", day=\"08/24/2021\"" 2021-08-10T11:35:00.508 ip=10.1.10.10 id=1 event="running" model="t1" conn="connmsg=\"model.log::option.event.view.log_view_conn, connname=\"model.log::option.event.view.log_view_conn_name\", user=\"xyz\", remote_conn=10.23.55.54, auth_conn=\"Base\"" 2021-08-10T11:35:00.515 ip=10.1.10.10 id=1 event="failed" model="t1" conn="Failed to connect to the file for \"file_name\"" 2021-08-10T11:35:00.890 ip=10.1.10.10 id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.logout.message\", user=, job_id=65, report_name=", path=\"{\"type\":1,\"appIds\":\"\",\"path\":\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/10.1.120.11\\/http_code\\/200\",\"restrict\":null}\"" 2021-08-10T11:36:00.090 ip=10.1.10.10 id=1 event="extracting" model="t1" conn="connmsg=\"model.log::option.event.view.audit.message, user=\"qic\\abc_pqr\, reason_msg=\"component.auth::message:unknown_user\", path=/abc/flows/timespan/2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00/ip_initiate/10.101.10.20/data.ext" 2021-08-10T11:36:00.380 ip=10.1.10.10 id=1 event="triggered" model="t1" conn="Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'" 2021-08-10T11:36:00.880 ip=10.1.10.10 id=1 event="triggered" model="t1" conn="connmsg=\"model.log::option.event.report.finished\", user=, job_id=65, report_name=", path=\"{\"type\":1,\"namespace\":\"flows\",\"appIds\":\"10,11,12\",\"path_bar\":\"[\\\"ip_initiate=10.1.120.11\\\"]\",\"2021-08-10T11:35:00+00:00_2021-08-10T12:35:00+00:00\\/ip_initiate\\/10.1.120.11\\/http_code\\/200\",\"restrict\":null}\"" The field which I am facing issue is "conn" field and I want data to be extracted in conn field in somewhat below manner - conn \"controller.conn_download::message.clean\" model.log::option.event.view.log_view_conn Failed to connect to the file for \"file_name\" \"model.log::option.event.view.logout.message\" \"model.log::option.event.view.audit.message\" "Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service'" but currently its just extracting the next value coming after conn= ,so basically current data in my conn field based on the above raw events looks like - conn connmsg=\ connmsg=\ Failed to connect to the file for \"file_name\" connmsg=\ connmsg=\ Rule 'Conn Web Service' was triggered by Indicator:'Conn Web Service' The "conn" field might contain even more key value pairs , so also wanted to know if there is some dynamic way to capture if any new key value pair pops in conn field other than those specified ? Also along with that, the other key value pairs in conn field is sometimes getting auto extracted and sometime it isn't. I am trying to write Search time field extraction using props and transforms but no luck so far in getting what I want, can someone please help ? Thanks in Advance.

ak9092 · ‎09-25-2020

Thank You @thambisetty for your response , but its not appending the results as i want. Its creating whole new row of results. So for e.g - i currently i have following data present in my csv file title 07 Jul 2020 08 Aug 2020 Title1 99.998 99.561 Title2 99.700 99.760 Title3 99.989 99.973 now when i append the new data for the month of September, then I want it to append just the Months column like below - title 07 Jul 2020 08 Aug 2020 09 Sep 2020 Title1 99.998 99.561 99.577 Title2 99.700 99.760 99.389 Title3 99.989 99.973 99.123 but its appending the results like below. title 07 Jul 2020 08 Aug 2020 09 Sep 2020 Title1 99.998 99.561 Title2 99.700 99.760 Title3 99.989 99.973 Title1 99.577 Title2 99.389 Title3 99.123 How do i fix this ?

ak9092 · ‎09-08-2020

Hi @thambisetty , thank you. I am now able to get the data in below format in splunk - title 07 Jul 2020 08 Aug 2020 title 07 Jul 2020 08 Aug 2020 Title1 99.998 99.561 Title2 99.700 99.760 Title3 99.989 99.973 is there a way to just append it to a csv file every month ? For example i will store this file on search head and on 1st of October i will like to append data for month of September in the csv before sending. If so ,then what changes i would need to make in following search so that it just adds the latest months data - my search .. | fillnull value=1000 response_code | eval success=case(response_code>=400, 0, timed_out == "True", 0, response_code="",0) | fillnull value=1 success | eval month=strftime(_time,"%m %b %Y") | stats count as total, sum(success) as successes by title month | eval availability=round(100*(successes/total),3) | chart list(availability) over title by month | eval month=replace(month, "\d+ (.*)","\1") Also I had added the month_num(%m) to sort the data by month but when i am trying to remove it using replace its not working. what am i doing wrong there any idea ?

ak9092 · ‎09-06-2020

Hi, I want to create a report through splunk that will send out an email consisting data of each months stats by auto appending the excel file with the past months data before sending. Following is my query which gives me the required data. Now i don't know how to distribute that data by month and then append in some excel file and send it through over email. my search.. | fillnull value=1000 response_code | eval success=case(response_code>=400, 0, timed_out == "True", 0, response_code="",0) | fillnull value=1 success |stats count as total, sum(success) as successes by title | eval availability=round(100*(successes/total),2) |stats count by title availability I want the data in excel file to look something like below - I want this to be done automatically through Splunk Schedule reports at the beginning of each month. Can someone please help me figure out a way if its possible through Splunk ? Thanks in advance.

ak9092 · ‎08-31-2020

@thambisetty this helps.. Thank You!!

ak9092 · ‎08-31-2020

oops ..my bad @niketnilay . Actually i did use sort before, but was viewing data in stats table itself and since the values was separated didn't paid much attention that it had already sorted 🙂 ..anyways thanks again for your help

ak9092 · ‎08-30-2020

Hi, I have a transaction that goes through multiple Status before its completed. Now the challenge I am facing here is , one status can be mapped multiple time before the transaction is completed and in some cases the same status can keep repeated until its move to the next state - For example ,my logs look something like below for one transaction ID - (Note-there can be many more statuses other than the one's below) 2020-08-27 08:00:40.000, ID="20", STATUS="CREATE" 2020-08-27 08:01:11.000, ID="20", STATUS="POST" 2020-08-27 08:01:42.000, ID="20", STATUS="POST" 2020-08-27 08:02:24.000, ID="20", STATUS="POST" 2020-08-27 08:03:46.000, ID="20", STATUS="REPAIR" 2020-08-27 08:03:56.000, ID="20", STATUS="PENDING" 2020-08-27 08:04:00.000, ID="20", STATUS="UPDATE" 2020-08-27 08:04:12.000, ID="20", STATUS="UPDATE" 2020-08-27 08:04:30.000, ID="20", STATUS="POST" 2020-08-27 08:04:46.000, ID="20", STATUS="COMPLETE" 2020-08-27 08:04:56.000, ID="20", STATUS="COMPLETE" Now , What i want to do is calculate the total duration of time a transaction spent in a particular status. So the final results should look something like below - ID STATUS max(_time) duration (sec) 20 CREATE 2020-08-27 08:00:40.487 31 20 POST 2020-08-27 08:02:24.265 155 20 REPAIR 2020-08-27 08:03:46.529 10 20 PENDING 2020-08-27 08:03:56.097 4 20 UPDATE 2020-08-27 08:04:12.715 30 20 POST 2020-08-27 08:04:30.366 16 20 COMPLETE 2020-08-27 08:04:56.517 As of now, with below query I am able to map the status according to time but the duration is not being calculated accurately. Can someone please help me figure this out. my search ... | sort 0 _time | streamstats current=false last(STATUS) as newstatus by ID | reverse | streamstats current=false last(_time) as next_time by ID | eval duration=next_time-_time | reverse | streamstats count(eval(STATUS!=newstatus)) as order BY ID | stats max(_time) as _time, sum(duration) as "duration(sec)" BY ID order STATUS Thanks in advance.

ak9092 · ‎08-30-2020

@niketnilay Thanks a lot for documenting it once again, this was something i was looking for . Just one more question though - how do i sort the statuses by count ?

ak9092 · ‎08-28-2020

Hello, I have very basic query which is giving me the desired results and visualization but even after lot of researching what I am not able to do is color the bars according to the field values. Below is something what my query looks like - my search ...| chart count by status now this status fields has around 15-20 status values like complete,pending, repair, canceled, posted,etc, etc.. so as of now my Bar chart has Status field on X-axis with the field values name mapped to it like complete, cancelled,etc and my Y -axis has the count of those statuses. Now what i want is to keep the visualization as it is and change colors of few statuses like green for Complete, red for canceled, amber for repair and so on. I tried charting.fieldColors">{"COMPLETE":#32a838,"CANCELED":#e81324,"REPAIR":#FFC200} but is not helping. I also tried to transposing rows to column with which i am able to change the colors but then the mapping of field values onto to the Y-axis is being removed and converted to legends, which is not looking good. Can this be achieved by keeping my current visualization intact ? I have gone through multiple pages here on Splunk Answers but no luck. Thanks in advance.

ak9092 · ‎08-07-2020

Hi @rwardwell , Have you had any luck in achieving this ? If yes, then can you please share your inputs as I am struggling with the same.

ak9092 · ‎07-10-2020

@bowesmana perfect ..Thanks!

ak9092 · ‎07-08-2020

Hi @bowesmana , this seems to be doing the trick but i am unable to filter events by any of that extracted fields. So for Example - i want to get event which are in STOP state, but when i tried executing below query it didn't returned any results ....|table LM GRP SR ID STATE SDL QUEUE SK FAIL | where STATE=STOP

Posts	40
Solutions	1
Karma Given	10
Karma Received	1
Member Since	‎11-01-2019

Help with Regex

Extract multiple key value pairs from one of the p...

How to create a report that appends the data of ev...

How to calculate for what duration the status was ...

Bar Chart Color change for dynamically changing st...

How to format raw events into splunk table when th...

Is it possible to sort field values present in one...

How to compare and match timestamp of one field wi...

if one field is returning null value then how do i...

How to reflect Disabled Data Input with some descr...

Re: Help with Regex

Re: Help with Regex

Re: Help with Regex

Re: Help with Regex

Re: Help with Regex

Help with Regex

Re: Extract multiple key value pairs from one of t...

Re: Extract multiple key value pairs from one of t...

Extract multiple key value pairs from one of the p...

Re: How to create a report that appends the data o...

Re: How to create a report that appends the data o...

How to create a report that appends the data of ev...

Re: How to calculate for what duration the status ...

Re: Bar Chart Color change for dynamically changin...

How to calculate for what duration the status was ...

Re: Bar Chart Color change for dynamically changin...

Bar Chart Color change for dynamically changing st...

Re: Drilldown to Show Source

Re: How to format raw events into splunk table whe...

Re: How to format raw events into splunk table whe...