About PanKokos

PanKokos · ‎01-04-2017

Hi, Yes, that looks like something I am looking for. I will test that and let you know if that is the thing. Do you know if this have a big impact of query execution time?

PanKokos · ‎01-03-2017

Hmm... Not sure if I got this. If I will modify my log files to be in a last form from above, for 2 tasks 1 [Time1] TaskId="1" TaskSender="ClientA" TaskType="TypeAAA" SectionName="Context" 2 [Time1] TaskId="1" ElapsedTime="100" SectionName="BuildingEnv" 3 [Time1] TaskId="1" ElapsedTime="34" SectionName="CalculatingResults" 4 [Time1] TaskId="2" TaskSender="ClientA" TaskType="TypeBBB" SectionName="Context" 5 [Time1] TaskId="2" ElapsedTime="100" SectionName="BuildingEnv" 6 [Time1] TaskId="2" ElapsedTime="34" SectionName="CalculatingResults" 4 .... Will it work for query: source | stats sum(ElapsedTime) as TotalTime by TaskType Note that TaskType is log only once for given task and is not present in other log lines (which have the time). Where is the place for dedup here?

PanKokos · ‎01-03-2017

Hi, I think that my question was misunderstood - or I have asked it not precise enough. We do not have duplicate entries, e.g. for same time and with same values: 1 [Time1] TaskId="1" Measure1="Value1" Measure2="Value2" GlobalContextVariable="xxx" GlobalContextVariable2="vvv" LocalContextVariable="aaa" .... 2 [Time1] TaskId="1" Measure1="Value1" Measure2="Value2" GlobalContextVariable="xxx" GlobalContextVariable2="vvv" LocalContextVariable="aaa" ... Basically, we are logging (mostly) execution time of critical section of application so it will be: 1 [Time1] TaskId="1" ElapsedTime="100" TaskSender="ClientA" TaskType="TypeA" SectionName="BuildingEnv" 2 [Time1] TaskId="1" ElapsedTime="34" TaskSender="ClientA" TaskType="TypeA" SectionName="CalculatingResults" 3 .... For each of this kind of logs we have number of distinct value (e.g. SectionName) and number of repeated values in every log entry (e.g. type of task, requestor etc.). What we want to achieve, is to compress the log files so we will have: 1 [Time1] TaskId="1" TaskSender="ClientA" TaskType="TypeA" SectionName="Context" **<-- logging static information for given task only once** 2 [Time1] TaskId="1" ElapsedTime="100" SectionName="BuildingEnv" 3 [Time1] TaskId="1" ElapsedTime="34" SectionName="CalculatingResults" 4 .... Because Splunk is, obviously, non-sql I am not sure if above is possible without reducing the performance of the queries and if that is even possible? Kind regards,

PanKokos · ‎01-02-2017

Hi, In my project we are using Splunk mainly for performance monitoring of application and we have created a dedicated logs for that. Currently they have following format: 1 [Time] TaskId="1" Measure1="Value" Measure2="Value" GlobalContextVariable="xxx" GlobalContextVariable2="vvv" LocalContextVariable="aaa" .... 2. [Time] TaskId="1" Measure1="Value" Measure2="Value" GlobalContextVariable="xxx" GlobalContextVariable2="vvv" LocalContextVariable="aaa" ... 3 [Time] TaskId="1" Measure1="Value" Measure2="Value" GlobalContextVariable="xxx" GlobalContextVariable2="vvv" LocalContextVariable="bbb" .... 4 [Time] TaskId="1" Measure1="Value" Measure2="Value" GlobalContextVariable="xxx" GlobalContextVariable2="vvv" LocalContextVariable="bbb" .... That was the easiest way to easily write Splunk queries and produce nice graphs. However, we are repeating a lots of data here. Also, it is making this logs harder to read for humans. Do you know if there is an easy alternative to remove this redundancy and still effectively query log files? E.g. to get at least a global context variables logged only once for a given task, something like: 1 [Time] TaskId="1" Key="CONTEXT" GlobalContextVariable="xxx" GlobalContextVariable2="vvv" 2 [Time] TaskId="1" Measure1="Value" Measure2="Value" LocalContextVariable="aaa" .... 3 [Time] TaskId="1" Measure1="Value" Measure2="Value" LocalContextVariable="bbb" ... Please forgive me if I am asking for something obvious 🙂 In the end, we want to be able to present e.g. Measure1 values grouped by GlobalContextVariable(s). Thanks a lot in advance for any help. Michal

PanKokos · ‎03-21-2016

Found out - finally it was a typo in a query tags in base query: <query> <query>

PanKokos · ‎03-21-2016

Hi, I have removed the fieldset from sample to reduce XML. Here it is: <fieldset submitButton="true" autoRun="false"> <input type="text" token="Name" searchWhenChanged="false"> <label>Name</label> <default>.*</default> </input> <input type="time" token="field3" searchWhenChanged="false"> <label>Time range</label> <default> <earliest>-6h@h</earliest> <latest>now</latest> </default> </input> </fieldset>

PanKokos · ‎03-21-2016

Hi, I have created quite large dashboard and want to add some optimizations to it. I choose to use base search as a starter here. However I could not make it working. Probably I am missing something basic - could anyone point me how to correct this? form> <label></label> <fieldset submitButton="true" autoRun="false">  </fieldset> <search id="baseSearch" > <query> <query> sourcetype="source" | where Type="Profiling" | regex Name ="$Name$" | eval ElapsedTime = ElapsedTime_ms / 1000 / 60 | eval Id = if(IsChild="True", ParentId, ID) | eval Reference = "(".RefSec."-".Name.")-".Id </query> <earliest>$field3.earliest$</earliest> <latest>$field3.latest$</latest> </query> </search> <row> <panel> <title></title> <chart> <search base="baseSearch"> <query> chart sum(ElapsedTime) as TotalTime over Reference by SectionName | addtotals fieldname=OTHER | eval OTHER=2*TotalExecutionTime - OTHER | fields - TotalExecutionTime | sort -OTHER | head 10</query> </search>  What I am missing here?

PanKokos · ‎03-14-2016

Child job is a job for which there is no other job which has it ID as a ParentJobId. I have defined this (more or less) with a SQL Pseudo Query). So I need to do something like this - get all jobs ids to kind of dictionary. Then, when filtering the events, do something like: .... | where JobId != ParentJobsIds | . In other words, if this job id was defined as parent job id somewhere, it means that this job is parent job so this event should be ignored. The structure could not be deeper than three nodes Root -> Parent -> Child or Parent->Child.

PanKokos · ‎03-14-2016

Hi, the problem is that I do not have the name ChilJob/ParentJob/Root available. I have only generic entry for root, parent and child jobs. I have updated the question to avoid confusion. Or I didn't get your suggestion?

PanKokos · ‎03-14-2016

Hey, Our tool has a root, parent and child jobs which we are monitoring using Splunk. For a short example: Job JobId="1" ParentJob="0" Job JobId="2" ParentJob="1" Job JobId="3" ParentJob="1" Job JobId="4" ParentJob="2" Job JobId="5" ParentJob="3" Job JobId="6" ParentJob="2" So here, the child jobs are only the jobs with ID = 4,5,6. I want to get events only from ChildJobs and the only way to do that is to execute following queries (pseudo SQL): 1. AllJobsId = SELECT DISTINCT JobId FROM JobStatuses 2. ChildJob = SELECT * FROM JobStatuses WHERE ParentJobId != 0 AND ParentJobId NOT IN [AllJobsId] Then I want to execute some stats on all events which were from ChildJobs. Is that possible? Regards, Michal

PanKokos · ‎03-09-2016

Hi, Clever! Also I didn't know that I can directly call fields in eval (e.g. ... | eval SUM = A + B). How I have missed that? This simplified the things a lot! Thanks!

PanKokos · ‎03-08-2016

Hi, We are trying to use Splunk to provide some nice diagrams showing execution time of critical sections in reference to total execution time. Let's say this is our input: Type ="Perf" Section="TOTAL" FlowType="F1" RequestType="R1" Time="23" Type="Perf" Section="A" FlowType="F1" RequestType="R1" Time="3" Type="Perf" Section="B" FlowType="F1" RequestType="R1" Time="13" Type="Perf" Section="TOTAL" FlowType="F2" RequestType="R2" Time="45" Type="Perf" Section="A" FlowType="F2" RequestType="R2" Time="30" Type="Perf" Section="B" FlowType="F2" RequestType="R2" Time="3" What we would like to have is a stacked bar chart, which will be high as value in Total and inside will have a bar for each Section and the difference will shown as OTHER (in case of first 3 rows OTHER = 7) So the y axis is time and the x axis is FlowType + RequestType. I am trying different queries but could not get anything yet... any ideas? This is hwat I got so far: source | where Type="Perf" | stats avg(Time) as AvgTime sum(Time) as Time values(Section) as SectionName by Section FlowType RequestType | eval Reference = "(".FlowType."-".SectionType.")" | eventstats avg(Time) as "AvgSectionTime" by FlowType RequestType SectionName | chart values(AvgSectionTime) as Time over Reference by SectionName This is not resolving all the issues, but it generates the stacked bar chart. Best regards, Michal

Posts	12
Solutions	1
Karma Given	1
Karma Received	0
Member Since	‎03-03-2016

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

Reducing logs redundancy

"No search query provided" when using base search

Removing events from "parent" source using ID

Struggling with stacked bar chart to show time spe...

Re: Reducing logs redundancy

Re: Reducing logs redundancy

Re: Reducing logs redundancy

Reducing logs redundancy

Re: "No search query provided" when using base sea...

Re: "No search query provided" when using base sea...

"No search query provided" when using base search

Re: Removing events from "parent" source using ID

Re: Removing events from "parent" source using ID

Removing events from "parent" source using ID

Re: Struggling with stacked bar chart to show time...

Struggling with stacked bar chart to show time spe...