topic Re: Need help on Search in Splunk Search

Need help on Search

pramit46 — Fri, 02 May 2014 06:12:09 GMT

This question is related to 'This' one. I wanted to extract multiple fields from different sourcetypes and indexes and print them.

Field1(F1) & Field2(F2) (belong to Sourcetype S1) and Field (F3) (belongs to Sourcetype S2). All the fields are multivalued.

Now, I want pass the data to a java program which would transform the data into an XML. for example, in the following format:
<Data> <F1-Value1> <F2-Value1/> <F3-Value1/> </F1-Value1> <F1-Value2> <F2-Value2/> <F3-Value2/> </F1-Value2> <F1-Value3> //No respective data for the value3 </F1-Value3> </Data>

In the output CSV file, I see that Splunk prints all the distinct values while I run the search. But I need all the values to be passed and in a way so that the java program is able to identify the relation among the values of F1, F2 & F3 while creating the XML.

What is the best way to gather the above data in Splunk? How can I list all the values of these fields (including the duplicates) and also indicate the java program about the relation between the fields so that it can form the XML?

I know this question is a little complicated to understand. Please let me know if you need more clarification.

Re: Need help on Search

lguinn2 — Fri, 02 May 2014 07:02:16 GMT

What IS the relationship between F1 and F2 and F3? I expect that F1 and F2 will occur in the same event, since they are both of sourcetype1, so I can see that relationship. But how does F3 fit in? What is the relationship between sourcetype1 and sourcetype2?

I don't think this is a complicated question, but I think you need to give more details. Perhaps you should show data that is not so abstract...

Re: Need help on Search

pramit46 — Fri, 02 May 2014 07:32:06 GMT

@lguinn, Unfortunately I cannot show you the actual data, so I'll try to put a relevant example.
I have some logs coming in from one subsystems which may be linked with the logs generated from other subsystems (may be by input file name or some other fields).
Now, all the sub-systems process the input file differently and store different fields in the respective logs. Thus, the values of F1 & F2 extracted from S1 may or may not be related to the value of F3 coming from S2.
But from the input filename, I can identify which values are related to that filename and put them in the same XML block.

Re: Need help on Search

pramit46 — Fri, 02 May 2014 07:46:35 GMT

@lguinn, when I use outputcsv, I receive the data in following format:
F1,F2,F3,
"F1-Value1" "F1-Value2" "F1-Value3",
"F2-Value1" "F2-Value2" "F2-Value1",
"F3-Value1" "F3-Value2" "F3-Value3"

Now how can I format the data in a way so that my java program can form the above XML by identifying the relation between the data? Do I need to add or remove something in my search query?

Not sure if I gave you enough information, so please let me know if you still have confusions

Re: Need help on Search

lguinn2 — Fri, 02 May 2014 20:32:37 GMT

Try

sourcetype=S1 OR sourcetype=S2
| stats list(F1) as F1 list(F2) as F2 list(F3) as F3 by inputFilename

You might add this at the end

| eval Fields = mvzip( mvzip(F1,F2,","),F3,",")
| fields - F1 F2 F3 inputFilename
| mvexpand Fields

Re: Need help on Search

pramit46 — Mon, 05 May 2014 13:22:52 GMT

Thanks @lguinn, it worked. But I guess, if you can help me with one more thing, I should be in a good shape.
Suppose, I'm publishing 15 fields (columns) in my search result, is there any way I can club certain columns together and give them a logical name? for example: If I have (mv)fields F1-F15 in output, now along with them, can I also print the following logical fields L1(by clubbing F1, F9, F13), L2(by clubbing F1, F9, F10), L3(by clubbing F4, F8, F13).
Please note that I have certain (actual) fields common in the values of logical fields.
Can I use mvzip/mvexpand for these?