topic How to make a loop including a dataframe for each jobname in Deployment Architecture

How to make a loop including a dataframe for each jobname

nsantiago17 — Mon, 03 Jun 2019 14:16:25 GMT

I have a dataset like this:

jobname     date                   avg      start             end
A              30/03/2019            84900  03:13:25           20:59:47
B              02/04/2019            60798  16:53:05           16:00:05
C              02/04/2019            60798  16:53:05           16:00:04
D              02/04/2019            79200  22:00:01           15:00:39

The length is 100k, and I need to do a loop including a dataframe for each jobname, for example:

df1 = A, 30/03/2019, 84900, 03:13:25,20:59:47
df2 = B, 02/04/2019,60798,16:53:05,16:00:05

So I can use the kmeans algorithm to cluster all of the data.

How can I do this?

Re: How to make a loop including a dataframe for each jobname

martynoconnor — Mon, 03 Jun 2019 18:56:14 GMT

Hi There,

What's the end use case you're trying to achieve here? I have a similar sort of search that I use in a dashboard of my own. The data you have looks fairly simple in structure and so you could index the data, ensuring the fields are correctly extracted and then use a search something like the following:

index=<your_index>  
| kmeans k=<how_many_clusters> dt=<your_distype> <numeric_field_1> <numeric_field_2> cfield="Cluster number"
| stats mode(<field1>) mode(<field2>) by "Cluster number"
| rename mode(<field1>) AS "<Better Name>" mode(<field2>) AS "<Better Name>"

The search above is one I use to cluster patterns in airline traffic, grouping them by heading and altitude.

Hopefully that's enough to get going with - let me know if you have any more questions.

Regards,

Martyn

Re: How to make a loop including a dataframe for each jobname

nsantiago17 — Tue, 04 Jun 2019 13:19:04 GMT

Hi Martyn,

I need to cluster my dataset in 4 groups: those who have outliers early in the month, 15th, 21th and last day of the month by the jobname, and I have to use the avg, date and one other numerical column.
So I can analyze each jobname already knowing their pattern. I'm trying to separate each jobname with their information and then start to figure how to cluster them. If you have some idea, please share with me.

Regards,

Nick

Re: How to make a loop including a dataframe for each jobname

evania — Fri, 07 Jun 2019 15:44:29 GMT

Hi @nsantiago17 ,

Did you have a chance to check out an answer? If it worked, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help you.

Thanks for posting!

Re: How to make a loop including a dataframe for each jobname

nsantiago17 — Mon, 24 Jun 2019 17:37:07 GMT

Hi, I replied Martyn but he didn't answer anymore, so I'm still without a solution or something that can help me to solve the question.