Activity Feed
- Karma Re: Parsing SIP multiline, multiformat events for bwooden. 06-05-2020 12:46 AM
- Got Karma for syslog field over-rides host_segment. 06-05-2020 12:46 AM
- Got Karma for syslog field over-rides host_segment. 06-05-2020 12:46 AM
- Got Karma for Add newline into table cell?. 06-05-2020 12:46 AM
- Got Karma for Parsing SIP multiline, multiformat events. 06-05-2020 12:46 AM
- Got Karma for Does a HiddenPostProcess limit the dynamic dashboard modules?. 06-05-2020 12:46 AM
- Posted Re: Parsing SIP multiline, multiformat events on Getting Data In. 06-27-2012 04:12 AM
- Posted Parsing SIP multiline, multiformat events on Getting Data In. 06-26-2012 03:42 AM
- Tagged Parsing SIP multiline, multiformat events on Getting Data In. 06-26-2012 03:42 AM
- Tagged Parsing SIP multiline, multiformat events on Getting Data In. 06-26-2012 03:42 AM
- Tagged Parsing SIP multiline, multiformat events on Getting Data In. 06-26-2012 03:42 AM
- Posted Re: syslog field over-rides host_segment on Getting Data In. 05-25-2012 05:51 AM
- Posted syslog field over-rides host_segment on Getting Data In. 05-23-2012 06:12 AM
- Tagged syslog field over-rides host_segment on Getting Data In. 05-23-2012 06:12 AM
- Tagged syslog field over-rides host_segment on Getting Data In. 05-23-2012 06:12 AM
- Posted Re: Parser finding time but not date. on Getting Data In. 03-29-2012 01:22 AM
- Posted Parser finding time but not date. on Getting Data In. 03-28-2012 12:55 PM
- Tagged Parser finding time but not date. on Getting Data In. 03-28-2012 12:55 PM
- Tagged Parser finding time but not date. on Getting Data In. 03-28-2012 12:55 PM
- Posted Re: Does a HiddenPostProcess limit the dynamic dashboard modules? on Dashboards & Visualizations. 03-19-2012 06:12 AM
Topics I've Started
Subject | Karma | Author | Latest Post |
---|---|---|---|
1 | |||
2 | |||
0 | |||
1 | |||
1 | |||
0 |
06-27-2012
04:12 AM
Excellent, thanks.
I came across a "2-phase" similar strategy in a question about FIX logs. Its a really useful way of working with ugly log formats. I can pull out other values with rex in the search command.
You also resolved some other issues on linebreaking I was having.
... View more
06-26-2012
03:42 AM
1 Karma
Hi, I'm trying to parse some logs generated by Broadsoft SIP servers. The log formats follow a general pattern but the detail can vary from event to event and field meanings can be context-sensitive.
The events are multiline broken by datetime string and the first portion is pipe-separated. The fields here can differ in number and meaning, and if I use DELIMS on the pipe character it works except for the last field which flows into the remainder of the event.
The first thing I'd like to do is stop the delims at a defined point which seems to be a newline character. The following transform using "| or newline" doesn't work. If I make it "| or tab", it works better for the first line but also matches unwanted fields in the remainder of the event (many of which start with tab).
[transform-bsft-xslog-test1]
# delims are pipe OR newline.
DELIMS = "|
"
FIELDS = "szDateTime" logLevel logType sipField1 sipField2 sipField3
Event sample:
2012.06.21 02:48:15:155 EST | Info | CallP | SIP Endpoint | +155512345678 | Service Delivery | localHost1234:5678
Processing Event: com.broadsoft.events.sip.SipReferEvent
2012.06.21 02:48:15:157 EST | Info | Accounting
SERVICE INVOCATION ACCOUNTING EVENT
Time Stamp: Thu Jun 21 02:48:15 EST 2012 (1340264895157)
Accounting ID: [id]
Service Name: Call Transfer
Related Accounting ID: [id]
2012.06.21 02:48:14:773 EST | Info | SipMedia | +155512345678 | localHost1234:5678
udp 391 Bytes IN from 10.10.10.10:5060
SIP/2.0 200 OK
[various amounts (10 - 30+ lines) of SIP information trimmed]
... View more
05-23-2012
06:12 AM
2 Karma
Hi,
I have a syslog server (Centos 6) with splunk 4.3.1 that receives syslog using the rsyslog daemon. The folder structure is /var/log/remote/1.2.3.4/syslog.log and I want to use the source IP address as the 'host' field.
The docs say to use host_segment, which I've done (inputs.conf shown below) but this seems to be ignored in favour of the syslog event hostname which could be IP, or could be hostname.
[monitor:///var/log/remote]
blacklist = *.gz
disabled = false
followTail = 0
index = test
sourcetype = syslog
whitelist = *.log
host_segment = 4
I've also tried manually setting it to a fixed string, and it still prefers the syslog headings. Sometimes the syslog message is that the last message repeated n times, in which case host=last.
Thanks
... View more
03-28-2012
12:55 PM
I have an event that starts something like this:
2012-03-20 06:07:00.000,BLANK,11.12.13.14,,,IP,Linux hostname 2.6.18-194.el5 1 SMP Tue Mar 16 21:52:39 EDT 2010 x86 64,
The first field is the timestamp of the event, I've inserted a blank value to separate it from the IP it seemed to not be identifying that as a proper timestamp. The problem I've got is the parser is using the time portion of the timestamp (06:07:00) but the date from the kernel string ( "Mar 16 .... 2010" ).
Within inputs.conf I've tried adding a prefix to lock the lookahead to the start of the event and not look beyond the end of the timestamp, but it still picks out the wrong thing.
TIME_FORMAT = %Y-%m-%d %H:%M:$S.000
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 25
... View more
03-19-2012
06:12 AM
No answers yet, but I did switch to JSChart (no difference) and run splunk in debug mode to have a look at the _internal indexes. There is a repeating pattern around the time that the drilldown is run:
class=js_chart.js, XHR clear for takeoff for module JSChart_1_15_0
class=js_chart.js, XHR in-flight destroyed for module JSChart_1_15_0 for job 1332161165.8 and replaced with new one
class=js_chart.js, Splunk.Module.JSChart .getResults() aborted
class=js_chart.js, Aborting getResults request for Splunk.Module.JSChart
Any ideas why the module would abort the search?
... View more
03-14-2012
07:29 AM
Here are the relevant parts of the dashboard. Search performance is reasonably and the drilldown triggers searches containing the correct intentions but the chart never appears.
<module name="HiddenSearch">
<param name="search">index=main sourcetype=rpt-pur-2 filter="foo" | eval dVolGB=(DOWNSTREAM_VOLUME/1000000) | bucket span=1h _time | stats sum(dVolGB) as downGB by _time, date_wday, PackageName, ServiceType | fields + _time date_wday PackageName downGB ServiceType
</param>
<module name="HiddenPostProcess">
<param name="search">timechart sum(downGB) by ServiceType</param>
<module name="SimpleResultsHeader" layoutPanel="panel_row2_col2">
<param name="entityName">results</param>
<param name="headerFormat">Selected Packages: Download Activity per day</param>
</module>
<module name="HiddenChartFormatter" layoutPanel="panel_row2_col2">
<param name="chart">column</param>
<param name="chart.stackMode">stacked</param>
<param name="primaryAxisTitle.text"></param>
<param name="secondaryAxisTitle.text">GBytes downloaded</param>
<param name="legend.placement">none</param>
<module name="FlashChart">
<param name="width">100%</param>
<param name="height">320px</param>
<param name="enableResize">False</param>
</module>
</module> <!-- end chart stuff -->
</module> <!-- end hidden post process -->
<module name="HiddenPostProcess">
<param name="search">
eval dtg=strftime(_time, "%H")
| stats sum(downGB) as downGBytes by dtg, date_wday, PackageName
| dedup 1 date_wday, PackageName sortby -downGBytes
| chart values(dtg) over PackageName by date_wday
| fields PackageName sunday monday tuesday wednesday thursday friday saturday
</param>
<module name="SimpleResultsHeader" layoutPanel="panel_row2_col1">
<param name="entityName">results</param>
<param name="headerFormat">Select Packages: Peak hour per day.</param>
</module>
<module name="SimpleResultsTable" layoutPanel="panel_row2_col1">
<param name="displayRowNumbers">false</param>
<param name="drilldown">all</param>
<param name="count">0</param>
<module name="HiddenSearch">
<param name="search">index=main sourcetype=rpt-pur-2 NOT "TIME_STAMP" | eval downMBytes=(DOWNSTREAM_VOLUME/1000) | chart sum(downMBytes) by date_hour, ServiceType
</param>
<module name="ConvertToIntention">
<param name="intention">
<param name="name">addterm</param>
<param name="arg">
<param name="date_wday">$click.name2$</param>
</param>
<param name="flags"><list>indexed</list></param>
</param>
<module name="ConvertToIntention">
<param name="intention">
<param name="name">addterm</param>
<param name="arg">
<param name="PackageName">$click.value$</param>
</param>
<param name="flags"><list>indexed</list></param>
</param>
<module name="SimpleResultsHeader" layoutPanel="panel_row2_col1">
<param name="entityName">results</param>
<param name="headerFormat">$click.value$ : utilisation on $click.name2$</param>
</module>
<module name="HiddenChartFormatter" layoutPanel="panel_row2_col1">
<param name="chart">column</param>
<param name="chart.stackMode">stacked</param>
<param name="primaryAxisTitle.text">Hour</param>
<param name="secondaryAxisTitle.text">MBytes downloaded</param>
<param name="legend.placement">none</param>
<module name="FlashChart">
<param name="width">100%</param>
<param name="height">320px</param>
<param name="enableResize">False</param>
</module>
</module> <!-- end chart stuff -->
</module> <!-- end c2i click.value -->
</module> <!-- end C2I click.value2 -->
</module> <!-- end HiddenSearch -->
</module> <!-- end simple results drilldown -->
</module> <!-- end hidden post process-->
</module> <!-- End HiddenSearch (or drilldown table postprocess) -->
... View more
03-14-2012
04:33 AM
1 Karma
Hi
I've been building a dashboard that contains a number of dynamic elements. Two modules are fed by a search that will return anything from a few thousand to a few million events depending on time range. One of these modules is a context sensitive table which triggers some more charts to appear using intentions.
First structure was this:
HiddenSearch
HiddenChartFormatter
FlashChart
HiddenSearch
SimpleResultsTable with drilldown
ConvertToIntention
ConvertToIntention
HiddenChartFormatter
FlashChart
This worked, but because I was running identical base queries at the same time each returning millions of events the performance was poor. It would take over a minute to get anything back and we lost the chart preview building during the wait time.
I swapped this for a single HiddenSearch and two HiddenPostProcess clauses but then the chart triggered by the click-through stopped appearing. I can see in the logs that the intentions are working and the query it generates does what it should do when I run it manually.
I played around with the structure and it seems that either the HiddenChartFormatter or the FlashChart cant live inside a HiddenSearch => HiddenPostProcess structure.
Is there some constraint?
... View more
- Tags:
- hiddenpostprocess
02-28-2012
02:09 PM
Thanks. Works well, although getting into the guts of CSS is something I was hoping to avoid.
... View more
02-28-2012
12:59 PM
1 Karma
Hi,
I'm splunking some network traffic and want to generate a table showing the peak hour for different categories of data, for each day through a week. To add some context I'd like to embed more information (e.g. total throughput) into the table cell which I can do using eval to generate a result string.
Is there any way of creating a newline within the result string so that the values wrap onto multiple lines?
Thanks
... View more
- Tags:
- newline
09-16-2011
02:31 AM
OK, so I played around an piping into eval started working. Go figure. Here's my current solution:
search and filter on Category
| stats first(_time) as timestr, values(Y) as y, sum(X) as x by date_year, date_month, date_mday, date_hour, SubCategory
| convert timeformat="%Y-%m-%d %H:00" ctime(timestr) as DateTime
| stats sum(y) as y, sum(x) as x by DateTime, SubCategory
| eval r=x/y
| chart sum(r) by DateTime, SubCategory
The raw data is grouped into 300s clumps comprising several thousand unique events. The first stats command creates the period over which the summaries are built and includes a timestamp which is used by the second stats to build the actual table. Then we eval the average and pipe into chart to produce something that can be put into a graph.
This allows me to keep or drop the SubCategory grouping in the second stats and the chart clauses and calculate the correct average over the Category.
By using stats I've lost the event timestamp so needed to manually re-insert it. I works, but not as nicely as timechart as the X-Axis strings are quite long and a bit ugly when plotting over a week (168 points per series). Changing it to summarise over different time periods would involve changing the 2nd and 3rd lines.
... View more
09-15-2011
02:35 AM
I did try that. I was expecting that the first stats would generate a table which can be piped onwards through more functions so I could generate averages over that summary table (4 or 5 rows) however I cant get the value of 'r' to appear anywhere. Piping the above into another stats results in no results.
... View more
09-14-2011
05:06 AM
This has stumped me for too long so I'm opening it up to the experts.
I have some event data of format "timestamp, Category, SubCategory, X, Y". The data has already been processed by the source system so that there are, say, 5000 events per timestamp. I want to filter on the Category, leaving about 500 relevant events to process.
X is a data volume (e.g. through an interface). I can generate the total using sum(X).
Y gives me the number of users. There is a fixed value of Y per SubCategory, so that for a given Category values(Y) would contain 4 or 5 data points. This is not really a problem as I can return the correct value in a subsearch through "join Category,SubCategory [search | stats | mvexpand | stats ]"
I want to calculate the average per-user volume for X for a given category and also for each subCategory within the category.
Using stats gives me:
SubCategory UsersInSubCategory sum(X) sum(X/Y)
A 100 100MB 1MB
B 200 200MB 1MB
Totals 300 300MB 2MB
This is correct when breaking out by SubCategory, but for the whole Category I cannot use sum(X/Y) as what I want is sum(X)/sum(Y). Since the underlying events are in the thousands avg(X/Y) gives me averages per event, not per SubCategory.
I'm beginning to think two queries might be easier.
... View more
- Tags:
- stats