Hi I am taking in data and making a new source type, so i need to use a transform for this. The issue is when i use this i cant seem to use SEDCMD to trim some of the lines i am taking in. Props.conf [AMBER_RAW:METRIC] SEDCMD-remove_header = s/^.*?{/{/1 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N TIME_PREFIX = \"ts\":\" INDEXED_EXTRACTIONS = JSON Transform.conf [AMBER_RAW_json_METRIC] DEST_KEY = MetaData:Sourcetype REGEX = {"v":"1.0\" FORMAT = sourcetype::AMBER_RAW:METRIC If i don't use the Transform the SEDCMD-remove_header works, but in the case i need to, for another issue i am having. Any ideas, how to get around this? Example of the data, but i have it working if i take it in difectly. However i have to use a transform in this case as i have multiple sourcetype in one files. 2018-01-10 15:50:03 [metrics-application-1-thread-1] INFO METRIC:41 - {"v":"1.0","t":"MTR","ts":"2018-01-10T15:50:03.704Z","h":"mx7654vm","pid":12483,"src":{"c":"authn-app","d":"auth"},"mtr":{"counters":{"process":{"cpu":{"time_cumulated_s":35},"memory":{"gc":{"ps_marksweep":{"total_duration_ms":814},"ps_scavenge":{"total_duration_ms":539}}}}},"gauges":{"com.murex.serviceframework.rest.datalayer.DataSourceMetrics.datasources.authn-authn-app-1":{"availableConnectionCount":1,"borrowedConnectionCount":0,"currPoolSize":1,"maxPoolSize":50,"poolName":"authn-authn-app-1"},"process":{"cpu":{"percentage":0.0014801778579070887},"files":{"open_files":37},"memory":{"jvm":{"heap":{"committed_kb":195072,"used_kb":111654},"nonheap":{"committed_kb":91456,"used_kb":89829}},"rss_kb":32880864,"vsz_kb":2295108}}},"histograms":{},"meters":{},"timers":{"process":{"memory":{"gc":{"ps_marksweep":{"events":{"count":1,"rate_1m":0.010541994097562058,"rate_5m":0.0030413993186727347,"rate_15m":0.001077675326868502,"rate_mean":0.023586525047214212},"duration_ms":{"max":620.0,"mean":620.0,"median":620.0,"min":620.0,"percentile_75":620.0,"percentile_95":620.0,"percentile_98":620.0,"percentile_99":620.0,"percentile_999":620.0,"standard_deviation":0.0}},"ps_scavenge":{"events":{"count":32,"rate_1m":1.3370365746688775,"rate_5m":1.8460208181687348,"rate_15m":1.9473463934310977,"rate_mean":0.7547234936660224},"duration_ms":{"max":18.0,"mean":9.125,"median":6.5,"min":3.0,"percentile_75":13.0,"percentile_95":18.0,"percentile_98":18.0,"percentile_99":18.0,"percentile_999":18.0,"standard_deviation":5.014495118187132}}}}}}}} Thanks, in advance Rob ... View more

HI I have used the below answer to get me 95% to a full solution, but i just cant get the last bit. https://answers.splunk.com/answers/567087/how-to-split-data-into-separate-sourcetypes-with-t.html I take in one file with multiple JSON and splits it into multiple source types. However i have a sub issue, one of the source types is like below Text + JSON trace. 2018-01-10 15:52:03 [metrics-application-1-thread-1] INFO METRIC:41 - {"v":"1.0","t":"MTR","ts":"2018-01-10T15:52:03.700Z","h":"mx7654vm","pid" I am looking to get only the JSON and removing the other data (at the start). So, i think i need a SED in the props? but not sure. I am trying not to use a heavy forwarder. props.conf [AMBER_RAW] SHOULD_LINEMERGE=false NO_BINARY_CHECK=true TRANSFORMS-sourcetye_routing = AMBER_RAW_json_METRIC [AMBER_RAW:METRIC] TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N TIME_PREFIX = \"ts\":\" INDEXED_EXTRACTIONS = JSON SEDCMD-REGEX_ONLY = s/^.({"v".).*$/\1/ Transforms.conf [AMBER_RAW_json_METRIC] DEST_KEY = MetaData:Sourcetype REGEX = {"v":"1.0\" FORMAT = sourcetype::AMBER_RAW:METRIC Thanks in Advance ... View more

Hi I have one file with multiple JSON types in it. What is the best way to get this data into Splunk. I dont think i can use a universal forwarder as i cant specify the sourcetype as i is multiple. Someone said use a heavy forward and do the work of splitting the data into different source types before i send it. Is this the correct approach? Thanks Robert Lynch ... View more

Hi I need my appendcols to take values from my first search. Specifically two values of time produce in the first search Start_epoc and Stop_epoc. For each row as the first search will produce multiple rows, and i need the second search to produce the same amount. Then i want to use them in the second search like below. earliest=$Start_epoc$ latest=$Stop_epoc$. | inputlookup Saved_Tests.csv | where Host="UBS-RC_QCST_MASTER" | where 1=1 | search Dev_Optimization="*" | search Functional_Optimization="*" | eval Start_epoc=Start | eval Stop_epoc=Stop | convert ctime(Start) | convert ctime(Stop) | table ID, host, Start_epoc , Stop_epoc | head 1001 | sort 0 - by ID | appendcols [| tstats count where index="mlc_live" host=UBS-RC_QCST_MASTER sourcetype="MX_TIMING2" earliest=$Start_epoc$ latest=$Stop_epoc$ by _indextime host | stats sum(count) as No_Of_MXTIMING_lines by host | table No_Of_MXTIMING_lines ] ... View more

Hi I have a very large base search. Full of tokens that can be driven from the user dashboard. It is built of 2 tstat commands doing a join. The issue is the second tstats gets updated with a token and the whole search will re-run. In the perfect world the top half does'tre-run and the second tstat re-use the 1st half's data from the original run and saves time I have but in a cut down version of the base search. + the whole thing below it. Can i get put a second base search inside the first base search, so it only reruns and does trigger the whole thing to re run? <search id="First_Base_Search"> <query>| tstats summariesonly=$summariesonly_token$ ...... etc.. | join Machine_Name NPID type=$join_type_token$ [| tstats summariesonly=$summariesonly_token$ ......etc ] | lookup MXTIMING_lookup_Base ..etc.. </query> <earliest>$time_token.earliest$</earliest> <latest>$time_token.latest$</latest> </search> <search id="First_Base_Search"> <query>| tstats summariesonly=$summariesonly_token$ max(MXTIMING.Elapsed) AS Elapsed max(MXTIMING.CPU) AS CPU max(MXTIMING.CPU_PER) AS CPU_PER values(MXTIMING.RDB_COM1) AS RDB_COM values(MXTIMING.RDB_COM_PER1) AS RDB_COM_PER max(MXTIMING.Memory) AS Memory max(MXTIMING.Elapsed_C) AS Elapsed_C values(source) AS source_MXTIMING avg(MXTIMING.Elapsed) AS average, count(MXTIMING.Elapsed) AS count, stdev(MXTIMING.Elapsed) AS stdev, median(MXTIMING.Elapsed) AS median, exactperc95(MXTIMING.Elapsed) AS perc95, exactperc99.5(MXTIMING.Elapsed) AS perc99.5, min(MXTIMING.Elapsed) AS min,earliest(_time) as start, latest(_time) as stop FROM datamodel=$MXTIMING_DATASET$ WHERE host=$host_token$ AND MXTIMING.Elapsed &gt; $MAX_TIME$ GROUPBY _time MXTIMING.Machine_Name MXTIMING.Context+Command MXTIMING.NPID MXTIMING.Date MXTIMING.Time MXTIMING.MXTIMING_TYPE_DM source MXTIMING.UserName2 MXTIMING.source_path MXTIMING.Command3 MXTIMING.Context3 span=1s | rename MXTIMING.Context+Command as Context+Command | rename MXTIMING.NPID as NPID | rename MXTIMING.MXTIMING_TYPE_DM as TYPE | rename MXTIMING.Date as Date | rename MXTIMING.Time as Time | rename MXTIMING.Machine_Name as Machine_Name | rename MXTIMING.UserName2 as UserName | rename MXTIMING.source_path as source_path | eval Date=strftime(strptime(Date,"%Y%m%d"),"%d/%m/%Y") | eval Time = Date." ".Time | eval FULL_EVENT=Elapsed_C | eval FULL_EVENT=replace(FULL_EVENT,"\d+.\d+","FULL_EVENT") | join Machine_Name NPID type=$join_type_token$ [| tstats summariesonly=$summariesonly_token$ count(SERVICE.NPID) AS count2 values(source) AS source_SERVICES FROM datamodel=SERVICE_V5 WHERE ( host=$host_token$ earliest=$service_earliest_time$ latest=$service_latest_time$) AND SERVICE.NICKNAME IN ($NICKNAME_TOKEN$) GROUPBY SERVICE.Machine_Name SERVICE.NICKNAME SERVICE.NPID | rename SERVICE.NPID AS NPID | rename SERVICE.NICKNAME AS NICKNAME | rename SERVICE.Machine_Name as Machine_Name | table NICKNAME NPID source_SERVICES Machine_Name ] | lookup MXTIMING_lookup_Base Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | appendpipe [ | where isnull(Threshold) | rename TYPE AS BACKUP_TYPE | eval TYPE="*" | lookup MXTIMING_lookup_Base Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | rename BACKUP_TYPE AS TYPE] | dedup Time, NPID,Context+Command | where Elapsed > Threshold OR isnull('Threshold') | fillnull Tags | eval Tags=if(Tags=0,"PLEASE_ADD_TAG",Tags) | makemv Tags delim="," | eval Tags=split(Tags,",") | search Tags IN ($TAG_TOKEN$) | eval source_SERVICES_count=mvcount(split(source_SERVICES, " ")) | eval NICKNAME=if(source_SERVICES_count > 1, "MULTIPLE_OPTIONS_FOUND",NICKNAME)</query> <earliest>$time_token.earliest$</earliest> <latest>$time_token.latest$</latest> </search> ... View more

Hi I need advice on the best approach for Datamodels and how to preload in the data on know points of interest. I have developed a Splunk_ID system in Splunk that will remember the start end and host name in a lookup table. So, for example, a user can store last weeks activity as Splunk_ID=123. This makes it quicker to get to your data. However this can take 30 seconds to load, 200 Million lines of complex tstat into a base search of a dashboard. How can I get splunk to have this ready in memory? Can I get a saved search to run and put it into a base search? Should I use an Add a "root search dataset" to get the query ready in memory? What I would like to do is I have over 1000 SPLUNK_ID. I would like to get splunk to have the last 100 in memory and ready for use. Below is the base search. It feeds about 15 drop downs or visualizations. | tstats summariesonly=$summariesonly_token$ max(MXTIMING.Elapsed) AS Elapsed max(MXTIMING.CPU) AS CPU max(MXTIMING.CPU_PER) AS CPU_PER values(MXTIMING.RDB_COM1) AS RDB_COM values(MXTIMING.RDB_COM_PER1) AS RDB_COM_PER max(MXTIMING.Memory) AS Memory max(MXTIMING.Elapsed_C) AS Elapsed_C values(source) AS source_MXTIMING avg(MXTIMING.Elapsed) AS average, count(MXTIMING.Elapsed) AS count, stdev(MXTIMING.Elapsed) AS stdev, median(MXTIMING.Elapsed) AS median, exactperc95(MXTIMING.Elapsed) AS perc95, exactperc99.5(MXTIMING.Elapsed) AS perc99.5, min(MXTIMING.Elapsed) AS min,earliest(_time) as start, latest(_time) as stop FROM datamodel=$MXTIMING_DATASET$ WHERE host=$host_token$ AND MXTIMING.Elapsed &gt; $MAX_TIME$ GROUPBY _time MXTIMING.Machine_Name MXTIMING.Context+Command MXTIMING.NPID MXTIMING.Date MXTIMING.Time MXTIMING.MXTIMING_TYPE_DM source MXTIMING.UserName2 MXTIMING.source_path MXTIMING.Command3 MXTIMING.Context3 span=1s | rename MXTIMING.Context+Command as Context+Command | rename MXTIMING.NPID as NPID | rename MXTIMING.MXTIMING_TYPE_DM as TYPE | rename MXTIMING.Date as Date | rename MXTIMING.Time as Time | rename MXTIMING.Machine_Name as Machine_Name | rename MXTIMING.UserName2 as UserName | rename MXTIMING.source_path as source_path | eval Date=strftime(strptime(Date,"%Y%m%d"),"%d/%m/%Y") | eval Time = Date." ".Time | eval FULL_EVENT=Elapsed_C | eval FULL_EVENT=replace(FULL_EVENT,"\d+.\d+","FULL_EVENT") | join Machine_Name NPID type=$join_type_token$ [| tstats summariesonly=$summariesonly_token$ count(SERVICE.NPID) AS count2 values(source) AS source_SERVICES FROM datamodel=SERVICE_V5 WHERE ( host=$host_token$ earliest=$service_earliest_time$ latest=$service_latest_time$) AND SERVICE.NICKNAME IN ($NICKNAME_TOKEN$) GROUPBY SERVICE.Machine_Name SERVICE.NICKNAME SERVICE.NPID | rename SERVICE.NPID AS NPID | rename SERVICE.NICKNAME AS NICKNAME | rename SERVICE.Machine_Name as Machine_Name | table NICKNAME NPID source_SERVICES Machine_Name ] | lookup MXTIMING_lookup_Base Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | appendpipe [ | where isnull(Threshold) | rename TYPE AS BACKUP_TYPE | eval TYPE="*" | lookup MXTIMING_lookup_Base Context_Command AS "Context+Command" Type as "TYPE" OUTPUT Tags CC_Description Threshold Alert | rename BACKUP_TYPE AS TYPE] | dedup Time, NPID,Context+Command | where Elapsed > Threshold OR isnull('Threshold') | fillnull Tags | eval Tags=if(Tags=0,"PLEASE_ADD_TAG",Tags) | makemv Tags delim="," | eval Tags=split(Tags,",") | search Tags IN ($TAG_TOKEN$) | sort Elapsed | eval source_SERVICES_count=mvcount(split(source_SERVICES, " ")) | eval NICKNAME=if(source_SERVICES_count > 1, "MULTIPLE_OPTIONS_FOUND",NICKNAME) Thanks in Advance Robert Lynch ... View more

Hi I have the following data written to one field. When i run source_SERVICES_count=mvcount(source_SERVICES) i get the answer of 1. However there are 3 results! So when i am writing the answer do i have to tell Spunk expect a multi value field? Or how can i get the mvcount to count the number of spaces to display 2? /dell872srv/apps/UBS_QCST_SEC2/logs/archive/300118_0937/mxsmcobjectrepository/smcobjectrepository_engine/566460-109829-service.log /dell872srv/apps/UBS_QCST_SEC2/logs/mxsmcobjectrepository/smcobjectrepository_engine/566460-109829-service.log /dell872srv/apps/UBS_QCST_SEC2/logs/mxsmcobjectrepository/smcobjectrepository_engine/986086-109829-service.log Thanks in Advance Rob ... View more

Hi Is there a way to get my eval token to wait for the full result. I am setting a token time_selection_BUFFER.earliest from a complex query so it take 5 seconds, however the eval below it is not waiting on the result. It looks like it is passing in the value $start$ to the token time_selection_BUFFER.earliest, before it is ready. This is an issues as i am specifically working on not refreshing the token with a CASE statement. So the questions, how do i get eval token to wait for set token to finish? <selection> <set token="time_selection_BUFFER.earliest">$start$</set> <eval token="time_selection.earliest">case($time_selection.earliest$ != $time_token.earliest$ ,$time_selection_BUFFER.earliest$,true(),$time_selection.earliest$)</eval> </selection> ... View more