Thanks dcarmack , This helped a lot .. However, I could not get the storage volume calculator from License master. basically, I want to calculate how much of storage is consumed by each index/host ... View more

I faced the similar problem on TCP and the same worked fine when sent as File. The solution I found is to use the below in props.conf SHOULD_LINEMERGE=false now all the events are broken properly .... ... View more

The final Query that solved my requirement is base search | rex field=_raw mode=sed "s/\<\d+\>\-[^\{]+//g" this Query, even without spath command provides afield value exactly the way we need for analytics. ... View more

Sridhar, I had come across a similar problem. I am explaining the approach I have taken, may be useful for you. 1. Combine all the three log files into an eventtype. Example: eventtype myevent="index=* host=hostname source="source1.log" OR source="source2.log" 2. in each log file define field extracts for the fields as you are currently doing. Use same field name across files (or use field aliases) 3. search for the common fields using eventtype. example: eventtype="myevent" ID=* this query will give you values for ID from all three files. -Bharath ... View more

I got to a partial solution to this issue as below: eventtype="Myevent" source="Mysource" ID=* | dedup ID | transaction ID | bin span=1d _time | eval formatted_time=strftime(_time, "%c") | eval Status="Pass" | chart values(Status) over ID by formatted_time similarly I used other fields to plot required values. coloring is the only challenge I have now . ... View more

The problem is not in the color . merging the content of two tables. i..e one table has time range as x-axis and the other has fields basically I want to merge the output of the below two: sourcetype=File* ID=* | dedup ID | stats count by ID| rangemap field=count low=0-100 elevated=101-1000 and sourcetype=File* ID=* | dedup ID | bucket span=1h _time | eval formatted_time=strftime(_time, "%c") | chart count over ID by formatted_time ... View more

Here is some sample data that can be used for generating this view. File1 belongs to App1 File2 belongs to App2 File3 belongs to App3 the application status should be shown based on the failures. Fails are identified by an Error() string. Note that the third file is a Error dump and not actually a structured log. File1: Mon Nov 16 2015 08:26:51 [0x00350016][mgmt][notice] source-mq(CommonBinaryPassthrough): tid(111): Service installed on port,414d512050423120202020565d7c2320099b2a, Error(Invalid Data) Mon Nov 16 2015 08:26:51 [0x00350014][mgmt][notice] source-mq(CommonMsgPassthrough): tid(111): Operational state up,414d5120505344322020202006f84f53cd9bb002, Error() Mon Nov 16 2015 08:26:51 [0x00350014][mgmt][notice] mpgw(CommonBinaryPassthrough): tid(111): Operational state up,414d5120505344322020202006f84f53cd9bb002, Error() Mon Nov 16 2015 08:26:51 [0x80e00344][mq][notice] mq-qm(Common_EAIT): tid(9171729): Connection succeeded,414d512050423120202020565d7c2320099b2a, Error(URL ….) Mon Nov 16 2015 08:26:51 [0x80e00344][mq][notice] mq-qm(Common_EAIT): tid(9171633): Connection succeeded,414d51205042312020202055a1b9d422a6e502, Error() Mon Nov 16 2015 08:26:51 [0x00350016][mgmt][notice] source-mq(CommonDataPassthrough): tid(111): Service installed on port, Error(" Could not get response") Mon Nov 16 2015 08:26:51 [0x00350014][mgmt][notice] source-mq(CommonBinaryPassthrough): tid(111): Operational state up, Error(" Invalid Data found") File2: 07:27:52.820',X'414d5120505344322020202006f84f53cd9bb002',X'414d51205042513157452055c28d792f442e8f',6,'TEST.MSG.TEST1' 2015-11-16 07:28:00.176457,'TEST.MSG.TEST2',NULL,X'414d512050423120202020565d7c2320099b2a',X'414d51205042513157452055c28d792f442e8f' 2015-11-16 07:28:00.178487,'TEST.MSG.TEST3',NULL,X'414d5120505344322020202006f84f53cd9bb002',X'414d512050425131574544312020202055c28d792f442e8f' 2015-11-16 07:28:02.709618,'TEST.MSG.TEST1',DATE '2015-11-16' GMTTIME '07:28:00.950',X'414d5120505344322020202006f84f53cd9bb002',X'414d51205042513157452055c28d792f442e8f',6 2015-11-16 07:28:04.066394,'TEST.REPLY',NULL,NULL,NULL,NULL,X'414d5120505344322020202006f84f53cd9bb002',X'414d51205042513157452055c28d792f442e8f' 2015-11-16 07:40:31.533186,'TEST.MSG.TEST1 '2015-11-16' GMTTIME '07:40:31.510',X'414d51205042312020202055a1b9d422a6e502',X'000000000000000000000000000000000000000000000000',4,'' File3: ******* MessageID: X'414d5120505344322020202006f84f53cd9bb002' ******* ******* 2015-11-03 11:26:29.663561 TEST.MSG.TEST1 ******* ( ['APP1' : 0x11d1f2b0] (0x01000000:Name):RecoverableException = ( (0x03000000:NameValue):File = '/mypath/Comptest.cpp' (CHARACTER) (0x03000000:NameValue):Line = 497 (INTEGER) (0x03000000:NameValue):Function = 'test' (CHARACTER) (0x03000000:NameValue):Type = 'TestNode' (CHARACTER) (0x03000000:NameValue):Name = 'TEST_MSG' (CHARACTER) (0x03000000:NameValue):Label = 'TEST_MSG' (CHARACTER) (0x03000000:NameValue):Catalog = 'msgs' (CHARACTER) (0x03000000:NameValue):Severity = 3 (INTEGER) (0x03000000:NameValue):Number = 2230 (INTEGER) (0x03000000:NameValue):Text = 'Caught exception and rethrowing' (CHARACTER) ) ) ******* MessageID: X'414d51205042312020202055a1b9d422a6e502' ******* ******* 2015-11-03 11:26:45.663461 TEST.MSG.TEST2 ******* ( ['APP2' : 0x11d1f2b1] (0x01000000:Name):RecoverableException = ( (0x03000000:NameValue):File = '/mypath/Comptest.cpp' (CHARACTER) (0x03000000:NameValue):Line = 497 (INTEGER) (0x03000000:NameValue):Function = 'test' (CHARACTER) (0x03000000:NameValue):Type = 'TestNode' (CHARACTER) (0x03000000:NameValue):Name = 'TEST_MSG' (CHARACTER) (0x03000000:NameValue):Label = 'TEST_MSG' (CHARACTER) (0x03000000:NameValue):Catalog = 'msgs' (CHARACTER) (0x03000000:NameValue):Severity = 3 (INTEGER) (0x03000000:NameValue):Number = 2230 (INTEGER) (0x03000000:NameValue):Text = 'Caught another exception and rethrowing' (CHARACTER) ... View more

this is the view I am trying to get in later on. ... View more

Agreed, and that has got as Excellent appreciation in today's management presentation as well. Thank you all ..Special thanks for Sundaresh. answer to Rich: the target is to process millions of transactions per day from different users coming into this app and display them based on User and transaction ID. log size is currently about 1GB per day in one app and it is growing. so it is useful to create such views in different colors and trigger events on failures too. ... View more

I am not allowed to upload file/photo because of point restrictions. if you can ping me on bharath dot kumar19 at wipro dot com I can provide more details. ... View more

I am not replacing an existing tool, but I am trying to do a POC for log analytics. The requirement is to present in a dashboard the overall statistics of events in each log file and status message for each transaction based on the ID that gets attached to it. second requirement is to present a table that has correlation between logs and when I select a transaction ID it should take me to the search page which lists all the transactions, then ofcourse there is a need for displaying few graphs . I am able to show graphs and statistics ...however I am struggling with tables , specially combining multiple tables into one ... View more

Due apologies, I missed to mention that I am searching on multiple files of each type, i,e. there are 10 files of type File1, 5 of type File2 and 1 of type File3 ... View more

you have given me a way to get the number or occurances . that is useful . however I tried a lot , but still could not reach to : Here is what I am trying out to get Transaction Status time 414d5120505344322020202006f84f53cd9bb002 Pass time 414d512050423120202020565d7c2320099b2a Fail time 414d51205042312020202055a1b9d422a6e502 Fail TransactionFail App1 TransactionFail App2 time 414d5120505344322020202006f84f53cd9bb002 time 414d512050423120202020565d7c2320099b2a time 414d51205042312020202055a1b9d422a6e502 ... View more

I needed all this in one search and your suggestion worked fine when I know the string 414d5120505344322020202006f84f53cd9bb002 before . My problem is that I get this from File1 at runtime. They are generated by application1 and passed to application2 , application3 for keeping track of message. so I have to extract ID at runtime from File1 and then search for it in other two. ... View more

Quite a bit work on it. On File3 I just wanted check if the MessageID is present, if so update the STATUS as fail. another tricky bit here is that I want to classify and display which application it failed. if Error(blahblah) is present in File1 - status is failed in app1 if no Error found then pass If ID entry found in File3 Fail this to be displayed in a table. currently I am able to extract these values, displaying them by relating them is still a problem ... View more

File3 does not have a line break. that is mistake while pasting it here. I have created a common eventtype that combines the three files for searching. however the error string is foung only in File1. I have also set field extractions and I am able to get these fields independently from each file. what I am failing is to correlate between them and generate a common table with Status based on error message. I have also created Field aliasing. Finally I saved the search output from File1 into a lookup test.csv file. however I am not clear how to use this for searching in other two files ... View more

Yes, I have created a common eventtype that combines the three files for searching. however the error string is foung only in File1. I have also set field extractions and I am able to get these fields independently from each file. what I am failing is to correlate between them and generate a common table with Status based on error message. I have also created Field aliasing. Finally I saved the search output from File1 into a lookup test.csv file. however I am NOT clear how to use this for searching in other two files ... View more

Your answer does not provide me the expected solution. the value 414d512050423120202020565d7c2320099b2a is runtime generated ..hence we have extract it dynamically secondly, First I want search for Error and if it failed i.e. there is come data within Error(....) I want to use the extracted ID values to search in other files and then display them in a single table ... View more

Thanks Rich, Answers to your question on break in the third file: NO, the log does not break, that line was my mistake while posting it. The log is continuous. however the log structure remains same. What I am trying to do is create a table that displays time, message(without duplicate) , status (based on the search described earlier) and may be few more related fields. from this table if the user clicks on a particular message ID, it should list all log entries with that ID ( from all three files) . currently it goes to the search result which has all entries. hope this answers your doubt. ... View more