Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Join the Conversation
Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
- About phoenixdigital
phoenixdigital
Builder
Member since:
02-08-2011
04-01-2026
Community Statistics
| Posts | 280 |
| Solutions | 25 |
| Karma Given | 33 |
| Karma Received | 77 |
| Member Since | 02-08-2011 |
Activity Feed
- Got Karma for Splunk Addon Builder 4.5.1 import/export broken on Splunk 10.2.1. Thursday
- Posted Splunk Addon Builder 4.5.1 import/export broken on Splunk 10.2.1 on All Apps and Add-ons. 04-01-2026 01:53 AM
- Got Karma for Re: Terminate User Session. 08-05-2022 12:07 AM
- Got Karma for Re: Realtime clock of Splunk server time on dashboard?. 06-10-2022 05:49 AM
- Got Karma for Indexing a CSV data file with more than one set of data. 04-06-2021 04:39 PM
- Got Karma for Can you hard code a color for a value range in Calendar Heat Map?. 06-05-2020 12:49 AM
- Karma Re: unarchive_cmd for decoding binary file with python script for MuS. 06-05-2020 12:48 AM
- Karma Re: unarchive_cmd for decoding binary file with python script for micahkemp. 06-05-2020 12:48 AM
- Karma Re: Splunk App for Stream: How to configure a universal forwarder to monitor DNS and DHCP? for vshcherbakov_sp. 06-05-2020 12:48 AM
- Karma Re: Is there a logging or debug tool to identify all props and transforms that are applied to a particular sourcetype to isolate rogue field extractions? for rjthibod. 06-05-2020 12:48 AM
- Karma Re: Is there documentation on forwarder behavior for various types of inputs when an indexer goes offline? for ryanoconnor. 06-05-2020 12:48 AM
- Karma Re: Search Head Deployer in a SH Cluster: What happens to local? for teunlaan. 06-05-2020 12:48 AM
- Karma Re: How is Server Identified After clone-prep-clear-config Script is Run? for jconger. 06-05-2020 12:48 AM
- Got Karma for Search Head Deployer in a SH Cluster: What happens to local?. 06-05-2020 12:48 AM
- Got Karma for Is there a logging or debug tool to identify all props and transforms that are applied to a particular sourcetype to isolate rogue field extractions?. 06-05-2020 12:48 AM
- Got Karma for How to update a global lookup file via REST API for a particular app in a search head cluster?. 06-05-2020 12:48 AM
- Got Karma for How to update a global lookup file via REST API for a particular app in a search head cluster?. 06-05-2020 12:48 AM
- Got Karma for setup.xml to modify/create an entry in app.conf (or anywhere else). 06-05-2020 12:48 AM
- Got Karma for Re: Chart not obeying charting.axisY.includeZero=false. 06-05-2020 12:48 AM
- Karma SA-Eventgen doesn't work? for laiyongmao. 06-05-2020 12:47 AM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 1 | |||
| 1 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 | |||
| 0 |
05-12-2015
06:40 PM
1 Karma
A quick question about how the asset and identity list is populated for Splunk ES.
I can see it is happening from a Identity Management modular input under (with associated python scripts)
Settings -> Data Inputs -> Identity Management
However sometimes the list used by Splunk fails to populate. Generally if there is an issue with the asset or identity csv.
The problem I have is this failure to parse the asset or identity list csv fails silently. I get no error to indicate that this didn't work.
This is probably the biggest issue I am concerned about as if the auto generation routines for these list puts something in that breaks the ingestion we might not know for weeks that the list is old due to a failure of parsing the newer csv.
Lastly how often does Splunk ES update this list when csv entries are changed?
Sometimes it appears to be immediate other times it appears to take a few minutes.
... View more
03-17-2015
06:01 PM
Agreed it would make things a hell of a lot easier. I am guessing they don't want to support tweaking the Python libraries and in a way I can see why.
Sadly it really limits the power of what we can achieve with modular and scripted inputs though.
... View more
03-17-2015
04:38 PM
Thanks this has definitely stopped the importing of the library issue.
cp -r /usr/lib64/python2.7/site-packages/systemd /opt/splunk/lib/python2.7/site-packages/
chown -R splunk:users /opt/splunk/lib/python2.7/site-packages/
I did want to try to avoid this however as I know Splunk officially does not recommend this.
I note in your thread Splunk were looking into resolving the need for this in 2003. It seems they moved onto more important issues since.
... View more
03-16-2015
08:58 PM
Hi All,
So I am trying to make a modular input which ingests the journal in newer LInux distributions which no longer log to a file.
I have this test Python script which works from the command line.
import select
import systemd
from systemd import journal
j = systemd.journal.Reader()
#j.log_level(journal.LOG_DEBUG)
# j.add_match(_SYSTEMD_UNIT="systemd-udevd.service")
j.seek_tail()
# need to get previous event as seek_tail went right to the very end of the journal
j.get_previous()
p = select.poll()
p.register(j, j.get_events())
while p.poll():
if j.process() != journal.APPEND:
continue
for entry in j:
event = str(entry['__REALTIME_TIMESTAMP'])
for key, value in entry.iteritems():
event = event + ',' + str(key) + '="' + str(value).replace('"','\\"') + '"'
print event
Now the issue I have is when I try to add it to a modular input script. Splunk's Python obviously doesn't have the library for systemd and I get this error
[root@phoenix bin]# /opt/splunk/bin/splunk cmd /opt/splunk/bin/python /opt/splunk/etc/apps/journal_input/bin/journal_input.py
Traceback (most recent call last):
File "/opt/splunk/etc/apps/journal_input/bin/journal_input.py", line 20, in <module>
from systemd import journal
ImportError: No module named systemd
Everything I have read says to leave Splunk's python alone and not to try to install custom modules.
Does anyone have any suggestions how I can get this into a modular input or to get systemd accessible to Splunk's Python binary without having support freak out?
... View more
- Tags:
- modular-inputs
- python
03-16-2015
08:46 PM
1 Karma
Hi P,
There really is no rule or set of procedures to follow here as everyone has different needs and there are so many different type of log data out there.
You are on the right track though with
(1) Data stopped ingestion - That search should definitely satisfy checking of data
(2) Data truncation - I can't think of anything specific here but the tests in item (4) might catch these errors.
(3) Correct timestamps recognition - This would be the hardest but you could say compare _indextime with _time to check for large drifts. This will only work for logs that are coming in real time. Historic or delayed logs would have issues.
(4) Field extraction Integrity - You would likely have to craft a scheduled search for each data type and check for specific critical fields missing.
ie sourcetype="fred" NOT criticalField=*
if that search returns results then your field extractions may be broken.
or you could check for a change in the number of fields extracted for a sourcetype (not tried this and doesn't look too friendly)
http://answers.splunk.com/answers/91632/getting-a-count-of-the-number-of-fields-associated-with-a-sourcetype.html
Hope that helps
... View more
02-19-2015
06:19 PM
It looks like that is the right props.conf config setting.
Try it on the server that is running the DBX app first then if that doesnt work add it to the indexer.
Ensure also that the inputs.conf is marking the relevant data as sourcetype=oracle_datasource
... View more
11-26-2014
08:35 PM
Is this still the case?
having local changes being overwritten would be pretty bad.
... View more
11-05-2014
05:20 PM
1 Karma
OK I upgraded Splunk and the DBX app and things are back to normal again.
The original search (database injection) is working as expected.
After updating the DBX app I had to run setup again for the app which I suspect removed whatever corrupt cache data file was there.
If a DBX support person can tell us where the DBX app might have been caching information that would be great. I am certain if I could have just deleted the relevant corrupted file this would have been resolvable within minutes not 10 days.
... View more
10-30-2014
10:52 PM
I thought you were on to something there as that makes perfect sense.
However adding a dedup did not fix the issue.
sourcetype="myStuff" earliest=-6m latest=-1m | dedup _time, SplunkField2, SplunkField3 | .... the rest of the search...
There must be some cache file somewhere that dbx is using. This search (database injection) worked perfectly prior to the power supply failure.
I even upgraded to Splunk 6.2 but the issue is still there.
I am trying to submit a support ticket but the Splunk support portal is broken for my clients account. It has been about 2 days now and they still havent resolved access to that.
... View more
10-29-2014
09:29 PM
OK based on that other answer I linked to in the question it was definitely the right track.
I messed around with the "Tag Cloud Autodiscover" dashboard in the Splunk 6 Example searches app and got it sorted.
Ref : http://your.splunk.intance:8000/en-US/app/simple_xml_examples/custom_viz_tag_cloud_autodiscover?earliest=0⪭st=
The key bit was just making a postprocessmanager which referenced the original manager.
Full code here
<dashboard script="autodiscover.js">
<label>Custom Visualization with Auto-Discovery</label>
<row>
<table>
<title>Table</title>
<searchString>index=_internal source=*metrics.log group=pipeline | stats max(cpu_seconds) as cpu_seconds by processor | sort - cpu_seconds</searchString>
<earliestTime>-24h</earliestTime>
<latestTime>now</latestTime>
</table>
<html>
<h2>Custom Visualization: Tag Cloud</h2>
<div id="tagcloud-search"
class="splunk-manager splunk-searchmanager "
data-require="splunkjs/mvc/searchmanager"
data-options='{
"app": "myapp",
"preview": true,
"search": "index=_internal source=*metrics.log group=pipeline | stats max(cpu_seconds) as cpu_seconds by processor",
"earliest_time": "-24h"
}'>
</div>
<div id="tagcloud-searchpostprocess"
class="splunk-manager splunk-searchmanager "
data-require="splunkjs/mvc/postprocessmanager"
data-options='{
"managerid": "tagcloud-search",
"search": "| where like(processor,\"s%\")"
}'>
</div>
<div id="tagcloud"
class="splunk-view"
data-require="app/infratil/components/rawdump/rawdump"
data-options='{
"minFontSize": 14,
"maxFontSize": 55,
"managerid": "tagcloud-search",
"valueField": "cpu_seconds",
"labelField": "processor"
}'>
</div>
</html>
</row>
<row>
<html>
<h2>Custom Visualization: Tag Cloud 2</h2>
<div id="tagcloud2"
class="splunk-view"
data-require="app/infratil/components/rawdump/rawdump"
data-options='{
"minFontSize": 14,
"maxFontSize": 55,
"managerid": "tagcloud-searchpostprocess",
"valueField": "cpu_seconds",
"labelField": "processor"
}'>
</div>
</html>
</row>
</dashboard>
... View more
10-28-2014
03:47 PM
Sadly this did not work.
/opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/dbx/bin/dbx_shpinst.py localhost:8089 --user myuser --db "MSSQL Instance"
splunk password:
database password:
Password at myserver.mydomain set successfully.
Are there any files being cached anywhere?
I even tried creating the DB connection again but it still fails.
We can manually run the INSERT statements against the database too btw.
... View more
10-27-2014
10:39 PM
I have a client who currently has a very complex dashboard with a few custom visualisations written in AdvancedXML.
I am wanting to see if I can convert it to the "new" Splunk Web Framework however there seem to be a few limitations that I have not resolved yet.
My current dashboard has panels that all work from one parent realtime saved search 'Dashboard realtime data search'.
<module name="SavedSearch" layoutPanel="viewHeader" autoRun="True">
<param name="name">Dashboard realtime data search</param>
<module name="JobProgressIndicator">
<module name="EnablePreview">
<param name="display">False</param>
<param name="enable">True</param>
</module>
<module name="PostProcess" layoutPanel="panel_row1_col1_grp1" group="Station State" autoRun="False">
<param name="search">where this="that" AND sectionCode="common" | fields totals, range | table totals, range</param>
<param name="groupLabel">Station State</param>
......
<module name="HiddenPostProcess" layoutPanel="panel_row2_col1" group="Generator Data">
<param name="search">where this="theother" AND like(sectionCode,"g%")</param>
<module name="MyCustomVisualisation"></module>
</module>
<module name="HiddenPostProcess" layoutPanel="panel_row2_col1" group="Generator Data">
<param name="search">where this="theNextOne" AND like(sectionCode,"g%")</param>
<module name="MyOtherCustomVisualisation"></module>
</module>
........
Basically it brings in the results of one big parent search in the 'SavedSearch' module.
Each panel then filters out the data it needs from this parent search.
Doing it this way reduces the load on the server as only one realtime search is running for all people viewing this dashboard.
So how to convert it to Splunk Web Framework
Working from the examples provided here in this Splunk app
https://apps.splunk.com/app/1603/
I am wanting to do something like this
SearchPostProcess seems to be the way to get parent/child searches working (note replace your.splunk.intance for links below)
http://your.splunk.intance:8000/en-US/app/simple_xml_examples/simple_search_postprocess?earliest=-24h&latest=now
How do I incorporate this into a custom visualisation like this?
http://your.splunk.intance:8000/en-US/app/simple_xml_examples/custom_viz_tag_cloud_autodiscover?earliest=0&latest=
I can see that the tagcloud visualisation references the search manager 'tagcloud-search'
"managerid": "tagcloud-search"
But is there a way to filter the results of 'tagcloud-search'?
And then do the same with every other panel?
I worked out you can definitely reference the same search from other panels also but just not sure how to add additional filters to the search.
<label>Custom Visualization with Auto-Discovery</label>
<row>
<table>
<title>Table</title>
<searchString>index=_internal source=*metrics.log group=pipeline | stats max(cpu_seconds) as cpu_seconds by processor | sort - cpu_seconds</searchString>
<earliestTime>-24h</earliestTime>
<latestTime>now</latestTime>
</table>
<html>
<h2>Custom Visualization: Tag Cloud</h2>
<div id="tagcloud-search"
class="splunk-manager splunk-searchmanager "
data-require="splunkjs/mvc/searchmanager"
data-options='{
"app": "infratil",
"preview": true,
"search": "index=_internal source=*metrics.log group=pipeline | stats max(cpu_seconds) as cpu_seconds by processor",
"earliest_time": "-24h"
}'>
</div>
<div id="tagcloud"
class="splunk-view"
data-require="app/infratil/components/rawdump/rawdump"
data-options='{
"minFontSize": 14,
"maxFontSize": 55,
"managerid": "tagcloud-search",
"valueField": "cpu_seconds",
"labelField": "processor"
}'>
</div>
</html>
</row>
<row>
<html>
<h2>Custom Visualization: Tag Cloud 2</h2>
<div id="tagcloud2"
class="splunk-view"
data-require="app/infratil/components/rawdump/rawdump"
data-options='{
"minFontSize": 14,
"maxFontSize": 55,
"managerid": "tagcloud-search",
"valueField": "cpu_seconds",
"labelField": "processor"
}'>
</div>
</html>
</row>
</dashboard>
Is this potentially how I should be doing it?
http://answers.splunk.com/answers/169754/splunk-django-webframework-cascading-form-search.html
Just not 100% sure how to translate this to SimpleXML
... View more
10-27-2014
05:56 PM
I even just tried this and got the same error.
sourcetype="myStuff" earliest=-6m latest=-1m | eval time = strftime(_time,"%Y-%m-%d %H:%M:%S") | eval table_field2 = round(SplunkField2,2) | rename Splunkfield3 AS table_field3 | rename Splunkfield4 AS table_field4 | fields time, table_field2, table_field3, table_field4 | table time, table_field2, table_field, table_field3, table_field4 | dboutput type=sql database="MSSQL Instance" "INSERT INTO TableForData (time, table_field2,table_field3,table_field4) values ($time$, $table_field2$,$table_field3$,$table_field4$)"
... View more
10-27-2014
05:22 PM
I should also note that even running this search manually gives this error. It is not just the saved search.
... View more
10-27-2014
05:19 PM
Hi All,
We had a server shutdown due to a power failure and now a saved search for inserting data into a database no longer works.
sourcetype="myStuff" earliest=-6m latest=-1m | eval time = _time | eval table_field2 = round(SplunkField2,2) | rename Splunkfield3 AS table_field3 | rename Splunkfield4 AS table_field4 | fields time, table_field2, table_field3, table_field4 | table time, table_field2, table_field, table_field3, table_field4 | dboutput type=insert database="MSSQL Instance" table="TableForData" time, table_field2,table_field3,table_field4
OK the table in the database has a primary key based on
time, table_field2, table_field3
Now remember this saved search worked perfectly prior to the power outage but now I get this error.
command="dboutput", Unexpected error while performing dboutput: java.sql.BatchUpdateException: Violation of PRIMARY KEY constraint 'PK_Name_of_Primary_Key_On_Database'. Cannot insert duplicate key in object 'TableForData'. The duplicate key value is (Oct 28 2014 10:07AM, Field2Data, Field3Data).
Is there somewhere on the local Splunk instance where primary keys are cached that I need to delete?
For the record the database has no data in it since the power outage (2 days ago) so there is definitely no primary key conflict.
I have tried restarting Splunk and the server but it had no effect.
Any ideas?
... View more
10-15-2014
06:07 PM
2 Karma
Ok it appears that threading was the wrong track to go down.
The SDK supports single or multiple instances meaning that
single instance - one script handles all of the data inputs
multi instance - each input fires off its own version of the script
http://docs.splunk.com/Documentation/Splunk/6.1.4/AdvancedDev/ModInputsScripts#Single_or_multiple_instances_of_a_script
I tested it out again with the random numbers example
single instance - http://pastebin.com/77MMLqPz
multi instance example - http://pastebin.com/idDqG7XY
Now to build the systemd modular input based on this new found knowledge.
... View more
10-13-2014
06:58 PM
I just had a similar issue and thought it was a realtime issue but it wasnt.
The trick with SearchTemplates is you need to specify the fields you want to pass down to use in your SearchPostProcess searches.
<dashboard>
<label>Search Templates</label>
<description>An example of search templates</description>
<searchTemplate>sourcetype="mysourcetype" station="thisone" | fields _time, sectionCode, other, interesting, fields, station</searchTemplate>
<earliestTime>rt-80m</earliestTime>
<latestTime>rtnow</latestTime>
<row>
<table>
<searchPostProcess>search sectionCode=this | eval ontime=strftime(_time, "%H:%M:%S") | table *</searchPostProcess>
</table>
<table>
<searchPostProcess>search sectionCode=that | eval ontime=strftime(_time, "%H:%M:%S") | table *</searchPostProcess>
</table>
</row>
</dashboard>
... View more
10-09-2014
10:03 PM
Adding threading doesnt seem to work.
http://pastebin.com/r5QyYb0T
Is there a better way I should be going about this?
... View more
10-09-2014
09:43 PM
Never mind resolved it with this change
http://pastebin.com/JfTNxQEv
My intervals were not defined when I was referencing them.
Going to have to change this a fair bit to spin off separate thread for each input in the loop
for input_name, input_item in inputs.inputs.iteritems():
Would the best way to achieve this to have stream_events definition loop through all inputs and fire off a thread for each which has a while 1: in it?
What closes these threads when new inputs are added via the GUI?
Because if stream_events is called again there will be a second set of threads started. Do I have to put the smarts in to ensure two threads for the same input dont exist?
... View more
10-09-2014
09:22 PM
So I have been reading the documentation on how to create modular inputs using the Python SDK here
http://dev.splunk.com/view/python-sdk/SP-CAAAER3
And I tested the random_number example from the SDK and it seems the random number is only generated when Splunk starts.
So as a learning exercise I tried to add a new parameter called 'interval' which will cause the number to generate every x seconds.
After a few failed attempts with indent errors in the script I finally got this to run without error
/opt/splunk/bin/splunk cmd /opt/splunk/bin/python /opt/splunk/etc/apps/random_numbers/bin/random_numbers.py
I restarted Splunk no data is coming in?
My new module input script is here
http://pastebin.com/TwVbHiUp
and my input.spec.conf is
[random_numbers://<name>]
*Generates events containing a random floating point number.
min = <value>
max = <value>
interval = <value>
Am I going about this the wrong way?
My final goal is to write something to listen to systemd journal. But I need to get past this hurdle of having something permanently running in the background either generating numbers in this case or listening to systemd down the track.
http://www.freedesktop.org/software/systemd/python-systemd/journal.html
... View more
- Tags:
- modular-inputs
08-25-2014
05:07 PM
Not sure why a workaround solution to this was removed from this thread but I am re-adding it again.
As of Splunk 6.1.3 (build 220630) there is still no support for BTRFS.
A workaround which I recommend for non production environments is to create a blockfile formatted to ext4 and mount this at /opt/splunk/var.
A 10G file can be created with the following on Fedora 20 (likely the same for most Linux flavours)
dd if=/dev/zero of=/opt/splunk-var-blockfile bs=1M count=10240
mkfs.ext4 /opt/splunk-var-blockfile
mount /opt/splunk-var-blockfile /opt/splunk/var
In order for the file to be mounted on reboot edit /etc/fstab and add the following line
/opt/splunk-var-blockfile /opt/splunk/var/ auto loop 0 0
... View more
08-07-2014
03:07 PM
I cracked it... not ideal but it works.
I am sure there is probably a better way to "repeat" that regular expression pattern but it works!!!!!
As you can see the first entry of FORMAT for each REPORT sets the "type" of data to allow you to search for a specific set.
INTERCONNECTOR_SOLN::1
allows for
sourcetype=predispatchis INTERCONNECTOR_SOLN=1
Now while I would much prefer to have different sourcetypes for each of these this is the next best solution.
props.conf
[predispatchis]
KV_MODE = none
SHOULD_LINEMERGE = false
TIME_PREFIX=\d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}
TRANSFORMS-a-filterprices = setnull,PREDISPATCH-getInterconnector,PREDISPATCH-getRegionSolution,PREDISPATCH-getRegionPrices
REPORT-extract_regionSolution = PREDISPATCH-regionSolutionCsvExtract
REPORT-extract_interconnect = PREDISPATCH-interconnectorSolnCsvExtract
REPORT-extract_regionPrices = PREDISPATCH-regionPricesCsvExtract
transforms.conf
[PREDISPATCH-getInterconnector]
REGEX = ^D,PREDISPATCH,INTERCONNECTOR_SOLN,(.*)
DEST_KEY = queue
FORMAT = indexQueue
[PREDISPATCH-getRegionSolution]
REGEX = ^D,PREDISPATCH,REGION_SOLUTION,(.*)
DEST_KEY = queue
FORMAT = indexQueue
[PREDISPATCH-getRegionPrices]
REGEX = ^D,PREDISPATCH,REGION_PRICES,(.*)
DEST_KEY = queue
FORMAT = indexQueue
[PREDISPATCH-interconnectorSolnCsvExtract]
REGEX = ^D,PREDISPATCH,INTERCONNECTOR_SOLN,([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)
FORMAT = INTERCONNECTOR_SOLN::1 PREDISPATCHSEQNO::$2 RUNNO::$3 INTERCONNECTORID::$4 PERIODID::$5 INTERVENTION::$6 METEREDMWFLOW::$7 MWFLOW::$8 MWLOSSES::$9 MARGINALVALUE::$10 VIOLATIONDEGREE::$11 LASTCHANGED::$12 DATETIME::$13 EXPORTLIMIT::$14 IMPORTLIMIT::$15 MARGINALLOSS::$16 EXPORTGENCONID::$17 IMPORTGENCONID::$18 FCASEXPORTLIMIT::$19 FCASIMPORTLIMIT::$20 LOCAL_PRICE_ADJUSTMENT_EXPORT::$21 LOCALLY_CONSTRAINED_EXPORT::$22 LOCAL_PRICE_ADJUSTMENT_IMPORT::$23 LOCALLY_CONSTRAINED_IMPORT::$24
[PREDISPATCH-regionSolutionCsvExtract]
REGEX = ^D,PREDISPATCH,REGION_SOLUTION,([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)
FORMAT = REGION_SOLUTION::1 PREDISPATCHSEQNO::$2 RUNNO::$3 REGIONID::$4 PERIODID::$5 INTERVENTION::$6 TOTALDEMAND::$7 AVAILABLEGENERATION::$8 AVAILABLELOAD::$9 DEMANDFORECAST::$10 DISPATCHABLEGENERATION::$11 DISPATCHABLELOAD::$12 NETINTERCHANGE::$13 EXCESSGENERATION::$14 LOWER5MINDISPATCH::$15 LOWER5MINIMPORT::$16 LOWER5MINLOCALDISPATCH::$17 LOWER5MINLOCALPRICE::$18 LOWER5MINLOCALREQ::$19 LOWER5MINPRICE::$20 LOWER5MINREQ::$21 LOWER5MINSUPPLYPRICE::$22 LOWER60SECDISPATCH::$23 LOWER60SECIMPORT::$24 LOWER60SECLOCALDISPATCH::$25 LOWER60SECLOCALPRICE::$26 LOWER60SECLOCALREQ::$27 LOWER60SECPRICE::$28 LOWER60SECREQ::$29 LOWER60SECSUPPLYPRICE::$30 LOWER6SECDISPATCH::$31 LOWER6SECIMPORT::$32 LOWER6SECLOCALDISPATCH::$33 LOWER6SECLOCALPRICE::$34 LOWER6SECLOCALREQ::$35 LOWER6SECPRICE::$36 LOWER6SECREQ::$37 LOWER6SECSUPPLYPRICE::$38 RAISE5MINDISPATCH::$39 RAISE5MINIMPORT::$40 RAISE5MINLOCALDISPATCH::$41 RAISE5MINLOCALPRICE::$42 RAISE5MINLOCALREQ::$43 RAISE5MINPRICE::$44 RAISE5MINREQ::$45 RAISE5MINSUPPLYPRICE::$46 RAISE60SECDISPATCH::$47 RAISE60SECIMPORT::$48 RAISE60SECLOCALDISPATCH::$49 RAISE60SECLOCALPRICE::$50 RAISE60SECLOCALREQ::$51 RAISE60SECPRICE::$52 RAISE60SECREQ::$53 RAISE60SECSUPPLYPRICE::$54 RAISE6SECDISPATCH::$55 RAISE6SECIMPORT::$56 RAISE6SECLOCALDISPATCH::$57 RAISE6SECLOCALPRICE::$58 RAISE6SECLOCALREQ::$59 RAISE6SECPRICE::$60 RAISE6SECREQ::$61 RAISE6SECSUPPLYPRICE::$62 LASTCHANGED::$63 DATETIME::$64 INITIALSUPPLY::$65 CLEAREDSUPPLY::$66 LOWERREGIMPORT::$67 LOWERREGLOCALDISPATCH::$68 LOWERREGLOCALREQ::$69 LOWERREGREQ::$70 RAISEREGIMPORT::$71 RAISEREGLOCALDISPATCH::$72 RAISEREGLOCALREQ::$73 RAISEREGREQ::$74 RAISE5MINLOCALVIOLATION::$75 RAISEREGLOCALVIOLATION::$76 RAISE60SECLOCALVIOLATION::$77 RAISE6SECLOCALVIOLATION::$78 LOWER5MINLOCALVIOLATION::$79 LOWERREGLOCALVIOLATION::$80 LOWER60SECLOCALVIOLATION::$81 LOWER6SECLOCALVIOLATION::$82 RAISE5MINVIOLATION::$83 RAISEREGVIOLATION::$84 RAISE60SECVIOLATION::$85 RAISE6SECVIOLATION::$86 LOWER5MINVIOLATION::$87 LOWERREGVIOLATION::$88 LOWER60SECVIOLATION::$89 LOWER6SECVIOLATION::$90 RAISE6SECACTUALAVAILABILITY::$91 RAISE60SECACTUALAVAILABILITY::$92 RAISE5MINACTUALAVAILABILITY::$93 RAISEREGACTUALAVAILABILITY::$94 LOWER6SECACTUALAVAILABILITY::$95 LOWER60SECACTUALAVAILABILITY::$96 LOWER5MINACTUALAVAILABILITY::$97 LOWERREGACTUALAVAILABILITY::$98 DECAVAILABILITY::$99 LORSURPLUS::$100 LRCSURPLUS::$101 TOTALINTERMITTENTGENERATION::$102 DEMAND_AND_NONSCHEDGEN::$103 UIGF::$104 SEMISCHEDULE_CLEAREDMW::$105 SEMISCHEDULE_COMPLIANCEMW::$106
[PREDISPATCH-regionPricesCsvExtract]
REGEX = ^D,PREDISPATCH,REGION_PRICES,([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)
FORMAT = REGION_PRICES::1 PREDISPATCHSEQNO::$2 RUNNO::$3 REGIONID::$4 PERIODID::$5 INTERVENTION::$6 RRP::$7 EEP::$8 RRP1::$9 EEP1::$10 RRP2::$11 EEP2::$12 RRP3::$13 EEP3::$14 RRP4::$15 EEP4::$16 RRP5::$17 EEP5::$18 RRP6::$19 EEP6::$20 RRP7::$21 EEP7::$22 RRP8::$23 EEP8::$24 LASTCHANGED::$25 DATETIME::$26 RAISE6SECRRP::$27 RAISE60SECRRP::$28 RAISE5MINRRP::$29 RAISEREGRRP::$30 LOWER6SECRRP::$31 LOWER60SECRRP::$32 LOWER5MINRRP::$33 LOWERREGRRP::$34
... View more
08-07-2014
12:25 AM
I've even tried this to just get them into the same sourcetype but with different extractions. In the hope that I can isolate them with a search on a field unique to each different data set.
This works and only pulls in the rows I am interested in however the fields dont extract. 😞
So close.....
inputs.conf
[monitor:///opt/web-data/PreDispatchIS_Reports]
disabled = false
followTail = 0
host = aemo-web
sourcetype = predispatchis
whitelist = PUBLIC_PREDISPATCHIS_[0-9_]*.CSV
crcSalt = <SOURCE>
props.conf
[predispatchis]
KV_MODE = none
SHOULD_LINEMERGE = false
TIME_PREFIX=\d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2}
TRANSFORMS-a-filterprices = setnull,getInterconnector,getRegionSolution
#TRANSFORMS-z-sourcetype-rewrite=setInterconnectorSource,setRegionSolutionSource
REPORT-extract_region = predispatchisCsvExtract
REPORT-extract_interconnect = interconnectorSolnCsvExtract
transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[getInterconnector]
REGEX = ^D,PREDISPATCH,INTERCONNECTOR_SOLN,(.*)
DEST_KEY = queue
FORMAT = indexQueue
[getRegionSolution]
REGEX = ^D,PREDISPATCH,REGION_SOLUTION,(.*)
DEST_KEY = queue
FORMAT = indexQueue
[interconnectorSolnCsvExtract]
REGEX = ^D,PREDISPATCH,INTERCONNECTOR_SOLN,([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)
FORMAT = PREDISPATCHSEQNO::$2 RUNNO::$3 INTERCONNECTORID::$4 PERIODID::$5 INTERVENTION::$6 METEREDMWFLOW::$7
[predispatchisCsvExtract]
REGEX = ^D,PREDISPATCH,REGION_SOLUTION,([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)
FORMAT = PREDISPATCHSEQNO::$2 RUNNO::$3 REGIONID::$4 PERIODID::$5 INTERVENTION::$6 TOTALDEMAND::$7
... View more
08-06-2014
10:56 PM
OK I have encountered this issue again and this time I need to resolve it.
The data is publicly available so I suppose there is no need to keep it private here is an example contents of the CSV file.
http://pastebin.com/pFSQNdru
or grab a full file to unzip here http://www.nemweb.com.au/REPORTS/CURRENT/PreDispatchIS_Reports/
The first sourcetype I want to call "predispatchis" and matches these formats
D,PREDISPATCH,REGION_SOLUTION,........
I want _time to be the second date in each row
The second sourcetype I want to call "interconnector_soln" and matches these formats
D,PREDISPATCH,INTERCONNECTOR_SOLN,
I want _time to be the second date in each row
So my thinking was have the inputs.conf pulling into a "temp" sourcetype then reassign the sourcetype based on which row matches from the transforms. Then this new sourcetype will (hopefully) find the timestamp and finally a few search time field extractions to get out the fields based on the known format.
Sadly it aint working at the moment and nothing is coming in.
Any assistance would be greatly appreciated.
inputs.conf
[monitor:///opt/web-data/PreDispatchIS_Reports]
disabled = false
followTail = 0
host = aemo-web
sourcetype = predispatchis_init_sourcetype
index=test
whitelist = PUBLIC_PREDISPATCHIS_[0-9_]*.CSV
crcSalt = <SOURCE>
props.conf
[predispatchis_init_sourcetype]
KV_MODE = none
SHOULD_LINEMERGE = false
TRANSFORMS-filterprices = setnull,getInterconnector,getRegionSolution
[predispatchis]
KV_MODE = none
SHOULD_LINEMERGE = false
TIME_PREFIX = ^D,PREDISPATCH,REGION_SOLUTION,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,
REPORT-extracts = predispatchisCsvExtract
[interconnector_soln]
KV_MODE = none
SHOULD_LINEMERGE = false
TIME_PREFIX = ^D,PREDISPATCH,INTERCONNECTOR_SOLN,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*
REPORT-extracts = interconnectorSolnCsvExtract
transforms.conf
[getInterconnector]
REGEX = ^D,PREDISPATCH,INTERCONNECTOR_SOLN,(.*)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::interconnector_soln
[getRegionSolution]
REGEX = ^D,PREDISPATCH,REGION_SOLUTION,(.*)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::predispatchis
[interconnectorSolnCsvExtract]
DELIMS = ","
FIELDS = "I","PREDISPATCH,INTERCONNECTOR_SOLN","3","PREDISPATCHSEQNO","RUNNO","INTERCONNECTORID","PERIODID","INTERVENTION","METEREDMWFLOW","MWFLOW","MWLOSSES","MARGINALVALUE","VIOLATIONDEGREE","LASTCHANGED","DATETIME","EXPORTLIMIT","IMPORTLIMIT","MARGINALLOSS","EXPORTGENCONID","IMPORTGENCONID","FCASEXPORTLIMIT","FCASIMPORTLIMIT","LOCAL_PRICE_ADJUSTMENT_EXPORT","LOCALLY_CONSTRAINED_EXPORT","LOCAL_PRICE_ADJUSTMENT_IMPORT","LOCALLY_CONSTRAINED_IMPORT"
[predispatchisCsvExtract]
DELIMS = ","
FIELDS = "I","PREDISPATCH","REGION_SOLUTION","4","PREDISPATCHSEQNO","RUNNO","REGIONID","PERIODID","INTERVENTION","TOTALDEMAND","AVAILABLEGENERATION","AVAILABLELOAD","DEMANDFORECAST","DISPATCHABLEGENERATION","DISPATCHABLELOAD","NETINTERCHANGE","EXCESSGENERATION","LOWER5MINDISPATCH","LOWER5MINIMPORT","LOWER5MINLOCALDISPATCH","LOWER5MINLOCALPRICE","LOWER5MINLOCALREQ","LOWER5MINPRICE","LOWER5MINREQ","LOWER5MINSUPPLYPRICE","LOWER60SECDISPATCH","LOWER60SECIMPORT","LOWER60SECLOCALDISPATCH","LOWER60SECLOCALPRICE","LOWER60SECLOCALREQ","LOWER60SECPRICE","LOWER60SECREQ","LOWER60SECSUPPLYPRICE","LOWER6SECDISPATCH","LOWER6SECIMPORT","LOWER6SECLOCALDISPATCH","LOWER6SECLOCALPRICE","LOWER6SECLOCALREQ","LOWER6SECPRICE","LOWER6SECREQ","LOWER6SECSUPPLYPRICE","RAISE5MINDISPATCH","RAISE5MINIMPORT","RAISE5MINLOCALDISPATCH","RAISE5MINLOCALPRICE","RAISE5MINLOCALREQ","RAISE5MINPRICE","RAISE5MINREQ","RAISE5MINSUPPLYPRICE","RAISE60SECDISPATCH","RAISE60SECIMPORT","RAISE60SECLOCALDISPATCH","RAISE60SECLOCALPRICE","RAISE60SECLOCALREQ","RAISE60SECPRICE","RAISE60SECREQ","RAISE60SECSUPPLYPRICE","RAISE6SECDISPATCH","RAISE6SECIMPORT","RAISE6SECLOCALDISPATCH","RAISE6SECLOCALPRICE","RAISE6SECLOCALREQ","RAISE6SECPRICE","RAISE6SECREQ","RAISE6SECSUPPLYPRICE","LASTCHANGED","DATETIME","INITIALSUPPLY","CLEAREDSUPPLY","LOWERREGIMPORT","LOWERREGLOCALDISPATCH","LOWERREGLOCALREQ","LOWERREGREQ","RAISEREGIMPORT","RAISEREGLOCALDISPATCH","RAISEREGLOCALREQ","RAISEREGREQ","RAISE5MINLOCALVIOLATION","RAISEREGLOCALVIOLATION","RAISE60SECLOCALVIOLATION","RAISE6SECLOCALVIOLATION","LOWER5MINLOCALVIOLATION","LOWERREGLOCALVIOLATION","LOWER60SECLOCALVIOLATION","LOWER6SECLOCALVIOLATION","RAISE5MINVIOLATION","RAISEREGVIOLATION","RAISE60SECVIOLATION","RAISE6SECVIOLATION","LOWER5MINVIOLATION","LOWERREGVIOLATION","LOWER60SECVIOLATION","LOWER6SECVIOLATION","RAISE6SECACTUALAVAILABILITY","RAISE60SECACTUALAVAILABILITY","RAISE5MINACTUALAVAILABILITY","RAISEREGACTUALAVAILABILITY","LOWER6SECACTUALAVAILABILITY","LOWER60SECACTUALAVAILABILITY","LOWER5MINACTUALAVAILABILITY","LOWERREGACTUALAVAILABILITY","DECAVAILABILITY","LORSURPLUS","LRCSURPLUS","TOTALINTERMITTENTGENERATION","DEMAND_AND_NONSCHEDGEN","UIGF","SEMISCHEDULE_CLEAREDMW","SEMISCHEDULE_COMPLIANCEMW"
... View more
07-09-2014
11:14 PM
I am seeing this exact same issue. I am surprised it hasnt been fixed almost a year on.
My row is like this but it still leaves everything encoded
<row>
<panel>
<table>
<title>Asset threats over last 60 days</title>
<searchString>index=cve-details vendor_id=$vendor_id$ | join vendor_id [ | inputlookup asset-list.csv ] | eval url=urldecode(url) | sort update_date desc | table asset_code, asset_name, vendor_name, cve_id, cwe_name, publish_date, update_date, summary, url</searchString>
<earliestTime>-60d</earliestTime>
<latestTime>now</latestTime>
<option name="wrap">true</option>
<option name="rowNumbers">false</option>
<option name="dataOverlayMode">none</option>
<option name="drilldown">cell</option>
<option name="count">10</option>
<drilldown>
<link><![CDATA[ $row.url$ ]]></link>
</drilldown>
</table>
</panel>
</row>
My link ends up looking like this. (not it is still encoded for URLs which is should not be)
So for a row with
url = http://www.cvedetails.com/cve/cve-2013-1763//
The drilldown link is
http://my.splunk.server.com.au:8000/en-US/app/cvedetails/http%3A%2F%2Fwww.cvedetails.com%2Fcve%2FCVE-2013-1763%2F
Is there anyway to get the raw text of the field to be the link?
... View more
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
04-01-2026
01:40 AM
|
