About schose

schose · ‎01-13-2021

Hi all, I'm having non-indexed-extracted json in events. When there is a json "host" field host, which is different from the indexed "host", then the search view is showing you 2 values for host in smart or verbose mode. you can't work with the searchtime extracted json host field - clicking on it gives you no results - as host is an indexed field. .. when you are doing a ... | stats count by host, then only "indextimehost" is reported back - as expected. this behaviour differers from "normal" kv searchtime detection: i found multiple posts regarding this like: https://community.splunk.com/t5/Getting-Data-In/Duplicate-host-field-after-indexing-JSON-event/m-p/292472 unfortunately i'm not able to change the json field name at the source. Rewriting is also no good option for me. This more looks like a display bug for me.. but drives the poweruser crasy. best Regards, Andreas

schose · ‎01-12-2021

a working transforms.conf: [ilookuptest1] INGEST_EVAL = pod = "testpod1" [ilookuptest2] INGEST_EVAL = annotation =json_extract(lookup( "testlookup.csv" ,json_object( "pod" ,pod), json_array(annotation)), "annotation" ) message: WARN CsvDataProvider - Unable to find filename property for lookup=testlookup.csv will attempt to use implicit filename. still there, but working. Regards, Andreas

schose · ‎01-11-2021

Hi all, I'm receiving a lot of splunk-optimize errors in splunkd.log. ls -l /opt/splunk/var/lib/splunk/audit/db/hot_v1_455 | grep tsidx | wc -l 105 [root@indexer-2 splunk]# /opt/splunk/bin/splunk-optimize -v -d /opt/splunk/var/lib/splunk/audit/db/hot_v1_455 Logging configuration: verbose=1, log2splunk=0 tm= 1610364481 INFO splunk-optimize start: dir=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455 mode=0 isfinal=false max_iteration=2147483647 min_src_count=8 lex_tpb=64 write_level=1 target_size=1572864000 tm= 1610364481 DEBUG source_0=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610362948-1610362948-15554879922328683899.tsidx sz=1080 tm= 1610364481 DEBUG source_1=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363147-1610363147-16387064457660043774.tsidx sz=1168 tm= 1610364481 DEBUG source_2=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363267-1610363267-16901658911893982757.tsidx sz=1168 tm= 1610364481 DEBUG source_3=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363272-1610363272-16923132382574368602.tsidx sz=1176 tm= 1610364481 DEBUG source_4=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363092-1610363092-16150633440092436534.tsidx sz=1176 tm= 1610364481 DEBUG source_5=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363178-1610363178-16519774093793408615.tsidx sz=1176 tm= 1610364481 DEBUG source_6=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363057-1610363057-16000335144082864920.tsidx sz=1176 tm= 1610364481 DEBUG source_7=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363122-1610363122-16279466559003501729.tsidx sz=1176 tm= 1610364481 DEBUG source_8=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363058-1610363058-16004583287645382032.tsidx sz=1176 tm= 1610364481 DEBUG source_9=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363212-1610363212-16665396587756881875.tsidx sz=1176 tm= 1610364481 DEBUG source_10=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363152-1610363152-16408391753423458229.tsidx sz=1176 tm= 1610364481 DEBUG source_11=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363242-1610363242-16797814191883895619.tsidx sz=1176 tm= 1610364481 DEBUG source_12=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363238-1610363238-16777050395829684027.tsidx sz=1176 tm= 1610364481 DEBUG source_13=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363062-1610363062-16021755056521050397.tsidx sz=1176 tm= 1610364481 DEBUG source_14=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363298-1610363298-17035162030350365663.tsidx sz=1176 tm= 1610364481 DEBUG source_15=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363182-1610363182-16537360674905484228.tsidx sz=1176 tm= 1610364481 DEBUG source_16=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363288-1610363288-16996035449514477598.tsidx sz=1224 tm= 1610364481 DEBUG source_17=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363221-1610363221-16704811249229177472.tsidx sz=1240 tm= 1610364481 DEBUG source_18=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363248-1610363248-16826109427840005476.tsidx sz=1248 tm= 1610364481 DEBUG source_19=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363226-1610363226-16726339824340034748.tsidx sz=1248 tm= 1610364481 DEBUG source_20=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363103-1610363103-16197833266659683219.tsidx sz=1248 tm= 1610364481 DEBUG source_21=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363038-1610363038-15927433915388732711.tsidx sz=1248 tm= 1610364481 DEBUG source_22=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363278-1610363278-16953970350838872388.tsidx sz=1248 tm= 1610364481 DEBUG source_23=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363048-1610363048-15965922895444683963.tsidx sz=1248 tm= 1610364481 DEBUG source_24=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363296-1610363296-17026567070646661147.tsidx sz=1248 tm= 1610364481 DEBUG source_25=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363268-1610363268-16910815734924420355.tsidx sz=1248 tm= 1610364481 DEBUG source_26=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363218-1610363218-16695936197528204421.tsidx sz=1248 tm= 1610364481 DEBUG source_27=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363118-1610363118-16266549629544376987.tsidx sz=1248 tm= 1610364481 DEBUG source_28=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363138-1610363138-16352697168739542857.tsidx sz=1248 tm= 1610364481 DEBUG source_29=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363084-1610363084-16116233719873926123.tsidx sz=1248 tm= 1610364481 DEBUG source_30=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363238-1610363238-16782065727925460598.tsidx sz=1248 tm= 1610364481 DEBUG source_31=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455/1610363174-1610363174-16502826878692865729.tsidx sz=1248 tm= 1610364481 ERROR optimize finished: failed, see rc for more details, dir=/opt/splunk/var/lib/splunk/audit/db/hot_v1_455, rc=-29 (unsigned 227), errno=2 tm= 1610364481 INFO exiting splunk-optimize process with rc=-29 (unsigned 227) I haven't found any documentation what returncode -29 means or how to get more logs for this issue. Could anybody help out? Regards, Andreas

schose · ‎01-06-2021

Hi, This sounds like a typical multiline issue. Check crio logs on the worker node where your application pods are running. If the displayed json is line by line - what i would guess - then you will need to setup multiline support in fluentd.. https://github.com/splunk/splunk-connect-for-kubernetes/blob/8ba455e7f988d129c9bfa13fdcf2025c6c4ae724/helm-chart/splunk-connect-for-kubernetes/charts/splunk-kubernetes-logging/values.yaml#L191 .. meaning you need to concat the event to fluentd before sending it to hec. Another technical working solution is to fix the events in splunk on hec side. When your sourcetype for the damaged application is brokensourcetype set at SCK you can rename it, write it to aggQueue and so fix it with splunk props.conf capabilities. props.conf [brokensourcetype] TRANSFORMS - streams = renamest , write - agg [jsonfixed] ... fix your json here transforms.conf: [ write - agg ] REGEX = .* DEST_KEY = queue FORMAT = aggQueue [ renamest ] REGEX = .* FORMAT = sourcetype ::jsonfixed DEST_KEY = MetaData : Sourcetype Regards, Andreas

schose · ‎01-06-2021

Hi all, I'm trying to ingest data using a lookup like descripted in: https://docs.splunk.com/Documentation/Splunk/8.1.1/Data/IngestLookups props.conf: [ilookuptest] TRANSFORMS-a = ilookuptest1 TRANSFORMS-b = ilookuptest2 transforms.conf: [ilookuptest1] INGEST_EVAL = pod = "testpod1" [ilookuptest2] INGEST_EVAL = annotation =lookup( "testlookup.csv" , json_object( "pod" , "pod" ), json_array( "annotation" )) lookup testlookup.csv: pod,annotation testpod1,testannotation1 testpod2,testannotation2 ingest data using: curl -k http://192.168.208.5:8088/services/collector -H 'Authorization: Splunk f05eedbb-a706-427e-9606-baa3e8036411' -d '{"index": "test", "sourcetype": "ilookuptest", "event":"this is for testing ingest eval lookup12"} props.conf and transforms.conf are located at $SPLUNK_HOME/etc/system/local .. lookup at $SPLUNK_HOME/etc/system/lookups . I'm getting errors in splunkd.log: WARN CsvDataProvider - No valid lookup table file found for this lookup=testlookup ERROR CsvDataProvider - The lookup table ' testlookup ' does not exist or is not available. ERROR pipeline - Runtime exception in pipeline=typing processor=regexreplacement error='Invalid function argument' confkey='source::http:test|host::192.168.208.5:8088|ilookuptest|' ERROR pipeline - Uncaught exception in pipeline execution (regexreplacement) - getting next event The event is not indexed... When defining transforms.conf INGEST_EVAL = annotation =lookup( "testlookup" , json_object( "pod" , "pod" ), json_array( "annotation" )) I'm getting errors in splunkd.log: WARN CsvDataProvider - Unable to find filename property for lookup=testlookup.csv will attempt to use implicit filename. Event is indexed but not getting the value from the lookup. File is there, read permissions are set, "| inputlookup testlookup.csv" is displaying results. Any hints or a working INGEST_EVAL using lookups example? Best Regards, Andreas

schose · ‎12-11-2020

Hi, Sure, you can configure fluentd to write to a file and read that file by any splunk instance (UF/HF). You can also write to kafka and read from Kafka to splunk, or use S3 .. here is complete list of fluentd output plugins: https://docs.fluentd.org/output/forward you just need to pick what helps you most. Regards, Andreas

schose · ‎12-09-2020

Hi @gcusello , I tried to share some demo data within the curl commands. 🙂 Anyway... the example you posted worked for me, awesome! Thanks and best regards, Andreas

schose · ‎12-09-2020

Hi all, I'm trying to ingest (multiline) events with the string "public_ip" and remove the rest props.conf: [public_ips] TRANSFORMS-removeallnonsense = remove_unneeded transforms.conf [remove_unneeded] REGEX = (?m)^((?!public_ip).)*$ DEST_KEY = queue FORMAT = nullQueue When i now try to ingest Data using HEC: curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk 4f40e8ab-99a6-479f-ba13-7352feb11111' \ -d '{"sourcetype": "public_ips", "event":"foobar"}' is not indexed - fine. curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk 4f40e8ab-99a6-479f-ba13-7352feb11111' \ -d '{"sourcetype": "public_ips", "event":"foobar public_ip: 1.2.3.4 foobar1"}' this is indexed - fine when running: curl -k http://localhost:8088/services/collector -H 'Authorization: Splunk 4f40e8ab-99a6-479f-ba13-7352feb11111' \ -d '{"sourcetype": "public_ips", "event":"foobar public_ip: 1.2.3.4 foobar\nline2"}' This is not indexed - not fine. It seems i have a regex multiline issue i do not see.. Thanks for your help in advance, Andreas

schose · ‎09-23-2020

Hi, normal tar behavior.. --no-same-owner extract files as yourself (default for ordinary users) --owner=NAME force NAME as owner for added files https://linux.die.net/man/1/tar regards, Andreas

schose · ‎08-24-2020

hi all, As pod is an indexed field you have to configure fields.conf or search bei pod::mypod.. it's stated in the documentation: "Splunk Connect for Kubernetes sends events to Splunk which can contain extra meta-data attached to each event. Metadata values such as "pod", "namespace", "container_name","container_id", "cluster_name" will appear as fields when viewing the event data inside Splunk. There are two solutions for running searches in Splunk on meta-data. Modify search to usefieldname::value instead of fieldname=value. Configure fields.conf on your downstream Splunk system to have your meta-data fields available to be searched using fieldname=value. Example: fields.conf.example " fields.conf example: https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/fields.conf.example sorry, Andreas

schose · ‎08-24-2020

Hi all, I’m getting strange results when splunking container logs collected by splunk connect for k8s… when searching for the pod logs with normal SPL: index=kubernetes earliest=08/24/2020:17:00:00 latest=08/24/2020:17:45:00 pod=nextcloud-dev-84ff5f7dfb-jj2gz | stats count result=0 when forcing running the stats on shd: index=kubernetes earliest=08/24/2020:17:00:00 latest=08/24/2020:17:45:00 | noop | search pod=nextcloud-dev-84ff5f7dfb-jj2gz | stats count result=131 - looks correct when running tstats: | tstats count where index=kubernetes pod=nextcloud-dev-84ff5f7dfb-jj2gz earliest=08/24/2020:17:00:00 latest=08/24/2020:17:45:00 result=131 - looks correct… What may be the issue for the "normal" search noch working for the user? Running Splunk Enterprise v8.0.5. best regards, Andreas

schose · ‎12-10-2019

Hi, you can just copy one index to another host (as far as there are single-instances) - there are no issues with that. just make sure your settings for indexes.conf are the same and file permissions are set accordingly. This also works for clustered enviorments with a little more efford. Regards, Andreas

schose · ‎11-29-2019

I downvoted this post because data loss.

schose · ‎11-12-2019

You can use a lookup definition to point to the recent .csv file. This would help you not to revise the searches.

schose · ‎10-15-2019

Hi all, I want to configure a Datamodel in different apps. On app should define the datamodel (here search). The seconds app should (here: dm_acc) should define schedule and acceleration. andreas@art-mb-2.local:~/splunk/etc/apps$ cat search/metadata/local.meta [datamodels/metricbeat] access = read : [ * ], write : [ admin ] owner = testuser [models/metricbeat] access = read : [ * ], write : [ admin ] owner = testuser andreas@art-mb-2.local:~/splunk/etc/apps$ cat search/local/datamodels.conf [metricbeat] #acceleration = true #acceleration.earliest_time = 0 acceleration.hunk.dfs_block_size = 0 andreas@art-mb-2.local:~/splunk/etc/apps/dm_acc$ cat default/datamodels.conf [metricbeat] acceleration = true acceleration.earliest_time = 0 acceleration.cron_schedule = * * * * * andreas@art-mb-2.local:~/splunk/etc/apps/dm_acc$ cat metadata/local.meta [] access = read : [ * ], write : [ admin ] export = system this seems to work fine so far. I can query the DM using | tstats summariesonly=t count values(host) from datamodel=metricbeat Unfurtunaly I'm receiving an error message: "Error in data model "metricbeat" : JSON file contents not available." when using the WebGUI and access Datamodels. Any hint's?! Does this look like a GUI bug?! Regards, Andreas

schose · ‎10-01-2019

Hi, i created a custom search command | dobump . Feel free to test it out and give feedback. https://github.com/schose/TA-dobump Cheers, Andreas

schose · ‎10-01-2019

Hi, i created a custom search command "| dobump ". Feel free to test it out and give feedback. https://github.com/schose/TA-dobump Cheers, Andreas

schose · ‎09-02-2019

Hi, The results of your SPL search are passed to your custom alert action script from stdin as json format. This example will create a file testResult.txt within bin directory.. you can check out the json there.. from __future__ import print_function from future import standard_library standard_library.install_aliases() import sys, json, urllib.request, urllib.error, urllib.parse if __name__ == "__main__": if len(sys.argv) < 2 or sys.argv[1] != "--execute": print("FATAL Unsupported execution mode (expected --execute flag)", file=sys.stderr) sys.exit(1) else: #settings = json.loads(sys.stdin.read()) result = sys.stdin.read() settings = json.loads(result) file = open("testResult.txt", "w") file.write(result) file.close() print("here we go", settings) sys.exit(0) resulting json for search: index=_internal | head 10 | rename host as testhost sourcetype as testsourcetype source as testsource | table testhost testsourcetype testsource {"app":"search"...","result":{"testhost":"hostname","testsourcetype":"splunkd_ui_access","testsource":"/Users/andreas/splunk/var/log/splunk/splunkd_ui_access.log"}}

schose · ‎08-28-2019

Hi all, I'd like to create a custom alert action, which doesn't get it's custom parameters from savedsearches.conf, but from another (custom) configuration file. I guess from the alert action (python) script it should be no issue to read and parse another configfile instead of using the json which is coming from stdin. so far I have no idea how to display the values for the parameters in the webgui of the alert from another source than savedsearches.conf. Did anyone may fall over a custom alert action not using the default way to save and display parameters? Thanks for you help in advance, Andreas

schose · ‎07-29-2019

Hi forum, we are facing large increasing delays between dispatch_time and scheduled_time in scheduler log. We see delays up to 200 seconds for searches which are scheduled every minute - resulting in not-scheduled searches. this directly correlates to "ERROR SHCRepJob - failed to delegate job job=SHPDelegateSearchJob peer="peername", guid="466C6F6D-2779-4BFD-AE1F-64A71D0A5AC8" saved_search=system;; err=" messages we see at the SHC Captain. Any hints? Other Single-Search-head instances do not show any issues. Using v.7.1.6 Best Regards, Andreas

Posts	135
Solutions	16
Karma Given	19
Karma Received	54
Member Since	‎06-28-2012

Online Status	Offline
Date Last Visited	‎03-17-2021 03:01 PM

duplicate host field when _raw is json

splunk-optimize fails rc=-29 (unsigned 227)

ingest_eval lookup example

ingest events with a specific string - remove the ...

Inconsistent results when searching splunk connect...

errormessage "JSON file contents not available." w...

custom alert action parameters not from savedsearc...

large delay between dispatch_time and scheduled_ti...

Why are the columns reordered when implementing re...

eventgen missing in v3.5

duplicate host field when _raw is json

Re: ingest_eval lookup example

splunk-optimize fails rc=-29 (unsigned 227)

Re: Splunk Connect for Kubernetes is parsing json ...

ingest_eval lookup example

Re: Log ingestion from fluentd

Re: ingest events with a specific string - remove ...

ingest events with a specific string - remove the ...

Re: Upgrading Splunk as root modified file system ...

Re: Inconsistent results when searching splunk con...

Inconsistent results when searching splunk connect...

Re: Move bucket between 2 different Splunks

Re: Why am I getting "Missing enough suitable cand...

Re: Use of multiple CSV files in one search with w...

errormessage "JSON file contents not available." w...

Re: how do you programmatically bump a search head...

Re: How to Update Splunk after JavaScript Changes ...

Re: How to create a custom alert action Python scr...

custom alert action parameters not from savedsearc...

large delay between dispatch_time and scheduled_ti...