About mathiask

mathiask · ‎03-22-2019

I am not sure if I understand this correctly According to the documentation the field using field.cidr should be converted to a canonical CIDR address https://docs.splunk.com/Documentation/Splunk/7.2.5/RESTREF/RESTkvstore#CIDR Could you sort elaborate on what you mean with implicit?

mathiask · ‎03-11-2019

Sadly, I did not further investigate or resolve this issue yet.

mathiask · ‎08-31-2018

Hello Splunkers I just noticed that there is a field type "cidr" for the KV Store. According to the API documentation this should handle any kind of IP ranges nicely in canonical form. http://docs.splunk.com/Documentation/Splunk/7.1.2/RESTREF/RESTkvstore#CIDR Until now we used field type string field.netrange = string I created a new collection for testing with field.netrange = cidr and transferred the content with | inputlookup | outputlookup But upon inspection | inputlookup I still observe the previous non-canonical IP ranges like 2001:620:2000::/48 Did I do something wrong? What is the benefit of using the field type cidr when there are no changes?

mathiask · ‎08-30-2018

Thanks. - apparently it seems that this option is available for lookup files - but is not available for KV store lookups @lguinn is this working as intended or just a minor oversight? since it is possible to set this in the config file for KV store lookups.

mathiask · ‎08-30-2018

Is this option still available? I am using 7.1.2. and there is no such GUI option This option needs to be set in the config file, which is not very optimal when someone opens and saves it and accidently removes it

mathiask · ‎02-14-2018

Could you check compatibility with current Splunk and CIM version and if so please update App Meta Data Thx

mathiask · ‎02-22-2017

We are trying to use the Splunk Stream REST API https://docs.splunk.com/Documentation/StreamApp/7.0.1/DeployStreamApp/SplunkAppforStreamRESTAPI We are able to see the configs curl -k https://localhost:8000/en-US/custom/splunk_app_stream/streams/udp curl -X GET -H "X-Requested-With: XMLHttpRequest" -H "Content-Type: application/json" -k https://localhost:8000/en-US/custom/splunk_app_stream/streams/udp {"sourcetype": "stream:udp", "modifiedBy": "admin", "protocolName": "UDP", "fields": [{"term": "flow.bytes", "desc": "The total number of bytes transferred", "enabled": true, "name": "bytes", "aggType": "value"}, {"term": "flow.c-ip", "desc": "Client IP Address", "enabled": true, "name": "src_ip", "aggType": "value"}, {"term": "flow.c-mac", "desc": "Client packets MAC address in hexadecimal format", "enabled": true, "name": "src_mac", "aggType": "value"}, {"term": "flow.c-port", "desc": "Client port number", "enabled": true, "name": "src_port", "aggType": "value"}, {"term": "flow.cs-bytes", "desc": "The number of bytes sent from client to server", "enabled": true, "name": "bytes_in", "aggType": "value"}, {"term": "flow.cs-content", "desc": "All raw payload content sent from client to server", "enabled": false, "name": "src_content", "aggType": "value"}, {"term": "flow.cs-packets", "desc": "The total number of packets sent from client to server", "enabled": true, "name": "packets_in", "aggType": "value"}, {"term": "flow.interface-name", "desc": "Name of network interface", "enabled": false, "name": "network_interface", "aggType": "value"}, {"term": "flow.hostname", "desc": "Hostname where flow was captured", "enabled": false, "name": "capture_hostname", "aggType": "value"}, {"term": "flow.protocol", "desc": "Level 7 protocol name (http, ftp, etc.)", "enabled": true, "name": "app", "aggType": "value"}, {"term": "flow.s-ip", "desc": "Server IP Address", "enabled": true, "name": "dest_ip", "aggType": "value"}, {"term": "flow.s-mac", "desc": "Server packets MAC address in hexadecimal format", "enabled": true, "name": "dest_mac", "aggType": "value"}, {"term": "flow.s-port", "desc": "Server port number", "enabled": true, "name": "dest_port", "aggType": "value"}, {"term": "flow.sc-bytes", "desc": "The number of bytes sent from server to client", "enabled": true, "name": "bytes_out", "aggType": "value"}, {"term": "flow.sc-content", "desc": "All raw payload content sent from server to client", "enabled": false, "name": "dest_content", "aggType": "value"}, {"term": "flow.sc-packets", "desc": "The total number of packets sent from server to client", "enabled": true, "name": "packets_out", "aggType": "value"}, {"term": "flow.time-taken", "desc": "Number of microseconds that it took to complete a flow event, from the end user's perspective", "enabled": true, "name": "time_taken", "aggType": "value"}, {"term": "flow.vlan-id", "desc": "VLAN ID from 802.1Q header", "enabled": false, "name": "vlan_id", "aggType": "value"}, {"term": "flow.flow-id", "desc": "Flow Id", "enabled": true, "name": "flow_id", "aggType": "value"}, {"term": "flow.protocol-stack", "desc": "Protocol stack of flow", "enabled": true, "name": "protocol_stack", "aggType": "value"}], "_key": "udp", "streamType": "event", "aggregated": false, "app": "Stream", "filters": {"matchAllComparisons": true, "comparisons": []}, "statsOnly": false, "extras": {"eventType": "flow.udp-event"}, "_user": "nobody", "index": null, "category": "Networking", "isReferenceStream": true, "id": "udp", "name": "UDP Flow Events", "enabled": true}[ But we cannot figure out how to edit them curl -X PUT -H "Cookie: splunkd_8000=PRlg_PB8THrM8tZYOebt6K6^ooApy73FJRvJgY2RliptRtPWTeEUuUmArMi9fia5Vpw2eP7HVsENKnw2rag6HhPqCDDJVT52YMjEA0GIM_4opmWNJkVrtsCnqJ8wOTWlVvmLIcC; splunkweb_csrf_token_8000=6914067146718907469" -H "X-Splunk-Form-Key: 6914067146718907469" -H "X-Requested-With: XMLHttpRequest" -H "Content-Type: application/json" -k http://localhost:8000/en-US/custom/splunk_app_stream/streams/udp/enable We could not figure out how do we valid header tokens? It is mentioned that we should be able to get this information of the GET request, but all we get is HTTP/1.1 200 OK Date: Wed, 22 Feb 2017 10:42:16 GMT Content-Type: text/json;charset=utf-8 X-Content-Type-Options: nosniff Content-Length: 3383 Vary: Cookie Connection: Keep-Alive X-Frame-Options: SAMEORIGIN Set-Cookie: session_id_8000=bda9573630ba5e19964c235345a9b53c8d449e96; expires=Thu, 23 Feb 2017 10:42:16 GMT; httponly; Path=/; secure Server: Splunkd How do we get the necessary information to set the following parameters? -H "Cookie: splunkd_8000=PRlg_PB8THrM8tZYOebt6K6^ooApy73FJRvJgY2RliptRtPWTeEUuUmArMi9fia5Vpw2eP7HVsENKnw2rag6HhPqCDDJVT52YMjEA0GIM_4opmWNJkVrtsCnqJ8wOTWlVvmLIcC; splunkweb_csrf_token_8000=6914067146718907469"

mathiask · ‎02-20-2017

Thanks for your input. We have a message processing engine in place, which is written in python 3. We want to use this engine to interact with the KV store, like inserting, updating some lookup data. Whether there is a python2 env on the system or Splunk or not, does not really help. At the moment we access the KV store directly through the REST API. Massive deletions are still an issue.

mathiask · ‎02-20-2017

@HEC output Yeah, this would be another issue. But we currently only very selectively forward data for indexing, the volume is just one of the issues. Licencing aside, based on Splunk reference HW (100 GB/day) this would require ~200 indexers (i.e. 2400 cores), without redundancy, Splunk Enterprise might not be the 'right' solution for this. It would be totally awesome though .. Reality aside: Are the events not automatically load balanced when I configure multiple receiving indexers in the Distributed Forwarder Management? Or is it more like Round Robin, i.e. DoS on indexer 1 then on indexer 2 and so on, instead of sending events all over all the time. @fanning out incoming netflow traffic It is pretty easy thing for us to do, as we receive all the netflow traffic on one instance on our cloud first and then distribute it to different applications. We do this for several reasons, if you are interested I can elaborate We use our UDP samplicator https://github.com/sleinen/samplicator

mathiask · ‎02-20-2017

I now upgraded the instance to 16 Cores and 16 GB RAM @netflowReceiver.MM.decodingThreads = NN Thanks for documenting the parameter, I already discovered the parameter in http://docs.splunk.com/Documentation/StreamApp/7.0.1/DeployStreamApp/Performancetestresults But probably I did not use it correctly... It seems to be working, I do not see any drop messages any more from that host. Decoding occupies 10 - 14 cores estimated Indexed volume 18 TB/day I now reduced the configuration to one socket, leading to the following observations Decoding only needs 9 - 12 cores estimated Indexed volume 14 TB/day I suspect that we actually loose data, which seems to be true when I check cat /proc/net/udp I can see increasing number of UDP drops on the receiving Interface Increasing the UDP buffers sudo sysctl -w net.core.rmem_max=<buffer size> sudo sysctl -w net.core.rmem_default=<buffer size> Single Socket buffer size = 524288000, some loss buffer size = 1073741824, very little to no loss CPU utilization 800 - 1600 %, generally around 1300 % estimated indexed volume 19 TB/day, estimated over 15 min 1615 - 1630 Multiple Socket buffer size = 104857600, very little to no loss CPU utilization 1000 - 1400%, generally around 1300% estimated indexed volume 17.5 TB/day, estimated over 15 min 1705 - 1720 It looks like with a large enough buffer ... it is all the same. The less estimated indexed volume I attribute due to the time, usually it declines a bit. I will leave it running over a few days to see how it keeps up.

mathiask · ‎02-17-2017

Hi When I forward some of our netflow traffic to Splunk Stream (dedicated streamfwd). We are around 100 - 300k Flows/s I see the following Error Message NetFlowDecoder::decodeFlow Unable to decode flow set data. No template definition received with id 256 from source 2 . Dropping flow data set of size 416 I assume this is because some of the templates are missing after a few minutes I get the following messages agentMode: 1 level: ERROR message: Netflow processing queues are full for NetflowReceiver #2. Dropped 274671 packets This problem shows up when I forward some of our high volume IPFIX Netflow data. This problem does not show up when I forward the lower volume Netflow v9. These just show the template definition errors which vanish after some time. CPU does not seem to be the problem, since the load is not maxing out the cores yet ~ 700 - 750% I assume that there are some buffering issues. There were apparently similar issues when using nfsen. Is there a way to increase the buffers? My streamfwd.conf [streamfwd] ipAddr = 0.0.0.0 processingThreads = 32 dedicatedCatureMode = 0 httpRequestSenderThreads=4 httpRequestSenderConnections=40 #netflowReceiver.0.port = 3000 #netflowReceiver.0.protocol = udp #netflowReceiver.0.ip = 192.168.20.5 #netflowReceiver.0.decoder = netflow netflowReceiver.0.port = 3001 netflowReceiver.0.protocol = udp netflowReceiver.0.ip = 192.168.20.5 netflowReceiver.0.decoder = netflow netflowReceiver.1.port = 3002 netflowReceiver.1.protocol = udp netflowReceiver.1.ip = 192.168.20.5 netflowReceiver.1.decoder = netflow netflowReceiver.2.port = 3011 netflowReceiver.2.protocol = udp netflowReceiver.2.ip = 192.168.20.5 netflowReceiver.2.decoder = netflow netflowReceiver.3.port = 3012 netflowReceiver.3.protocol = udp netflowReceiver.3.ip = 192.168.20.5 netflowReceiver.3.decoder = netflow netflowReceiver.4.port = 3013 netflowReceiver.4.protocol = udp netflowReceiver.4.ip = 192.168.20.5 netflowReceiver.4.decoder = netflow netflowReceiver.5.port = 3014 netflowReceiver.5.protocol = udp netflowReceiver.5.ip = 192.168.20.5 netflowReceiver.5.decoder = netflow netflowReceiver.6.port = 3021 netflowReceiver.6.protocol = udp netflowReceiver.6.ip = 192.168.20.5 netflowReceiver.6.decoder = netflow netflowReceiver.7.port = 3022 netflowReceiver.7.protocol = udp netflowReceiver.7.ip = 192.168.20.5 netflowReceiver.7.decoder = netflow netflowReceiver.8.port = 3023 netflowReceiver.8.protocol = udp netflowReceiver.8.ip = 192.168.20.5 netflowReceiver.8.decoder = netflow

mathiask · ‎02-15-2017

Hi Thanks. What I don't understand yet is how do I get the tokens The /streams/ GET command comments The cookies and x-splunk-form-key can be obtained from the GET request's response headers. But when I run it I only get $ curl -i -k https://localhost:8000/en-US/custom/splunk_app_stream/streams/test HTTP/1.1 404 Not Found Date: Wed, 15 Feb 2017 14:35:14 GMT Content-Type: text/json;charset=utf-8 X-Content-Type-Options: nosniff Content-Length: 337 Vary: Cookie Connection: Keep-Alive X-Frame-Options: SAMEORIGIN Set-Cookie: session_id_8000=ee2f22f1c69a96be792938a883934b541b503d89; expires=Thu, 16 Feb 2017 14:35:10 GMT; httponly; Path=/; secure Server: Splunkd {"status": 404, "error": "Stream with specified id not found", "success": false} I assume that I can just take the sessionID for the cookie so splunkd_8000=ee2f22f1c69a96be792938a883934b541b503d89 But when I look at the POST command I also need splunkweb_csrf_token_8000= X-Splunk-Form-Key: Where do I get these from?

mathiask · ‎02-10-2017

Hello Splunkers There is a partial REST API documentation. There is a lot of information missing. http://docs.splunk.com/Documentation/StreamApp/7.0.1/DeployStreamApp/SplunkAppforStreamRESTAPI We are really interested in configuring our streams using the REST API. Especially, since we are unable to use the SDK, because we are using python3. Current, urgent use case We want to create, modify and delete streams Greetings Mathias

mathiask · ‎12-14-2016

Hi David We currently have a dedicated solution in place to collect the full netflows. Side note, a former team colleague actually wrote the tool/code that you are basing on for your NetFlow Add-on https://splunkbase.splunk.com/app/1658/ Obviously it would be really awesome to have all the netflow information in Splunk and to correlate it etc. etc. But as I stated the volume is far too large, and by that I mean by orders of magnitudes ... currently we are talking of an average of 100k Flows per second with much larger peaks, tendency growing ... This results in an estimated Splunk Indexed volume of about 10TB/day ... beside the license costs, this means we are talking about 100 indexers with all the maintenance, storage etc. this is way out of the current scope Even considering aggregation would not solve this. Assuming we are able to achieve a reduction of 90%, which I don't expect it to be, we are still talking about 1 TB per day, i.e. 10 servers ... This would be still out of scope and is simply not the use case we are currently pursuing. While I see the potential benefits to analyse it with Splunk instead of a dedicated solution, this is simply not the scope of our current setup. If one day we are really considering doing this, I think we rather go down the Hadoop (+Splunk?) path.

mathiask · ‎12-14-2016

Just to state this clearly Stream REST API would be the best way to go. Python is fine, fyi we are on Python 3, therefore not Splunk SDK, but you can give us the Python 2 Code and we will convert it. We can then share it back with you. If you want to share this "offline" then you can send me this at mathias.karlsson@switch.ch With the REST API we can then directly access Stream through our "processing engine" / "messaging system" where this information is being processed anyways, which is perfect. FYI We are using the Splunk KV Store as backend for that processing engine instead of a traditional DB or running a separate mongoDB. This allows us to monitor the messages that are being processed and/or are ready to be delivered directly in Splunk.

mathiask · ‎12-13-2016

Yeah I thought of that ... it would be like "hardcoding" the list(s) as REGEX rather than defining a config, i.e. lookup, and updating the list, but it could do the job ... ... given that it scales to several hundred entries and way beyond 100k events/s

mathiask · ‎12-13-2016

Thanks for that input These solutions would work well if I can first dump everything into splunk and then query it. This usually works within one enterprise/organisation and/or if you have sufficient storage. But this does not work for us. Depending on the use case we very fast talk about multiple TB/day, just to later drop 99.999% of it ... While this would be really fun to do ... the cost benefit ratio is not in my favor 🙂 Therefore, I want to lookup some information based on the information in the event, and then decide upon this to keep or drop the event. DHCP is just a simple example because everyone knows the problem ...

mathiask · ‎12-13-2016

In my case the issue "vanished" After I rolled out 6.5 and made a rolling restart of the cluster it did not show up any more.

mathiask · ‎12-13-2016

Okay I have to explain the use case a bit in more detail I do NOT want to limit the devices able to send the flows. As you mentioned this is done on routing and/or FW. I want to filter the NetFlow "events" pre-indexing. This is because we have a ridiculous amount of NetFlow (estimating 5 - 20TB/day in Splunk) ... and because of that we have currently have a dedicated solution. This might maybe change eventually .. but I doubt that running 50 - 200 Indexers (100 GB/day) is the right way for us 😉 But some select set of NetFlows are of special interest and these we would like to have in Splunk today. And the most convenient way could be Splunk Streams, given we can limit the indexed events somehow. This would point to Stream Filters. I have several "lists" with different patterns and context information (for later enrichment). The patterns can be simple IP, IP ranges or more complex like IP-port matches etc. These lists can have easily more than hundred entries We therefore would have many streams like "netflow-IP" with a large list of interesting IPs or one stream for each pattern like IP-port Is there a way to bulk configure streams through the CLI or config files? or a REST API? Is it possible to configure more "complex" filters like (ip=1.2.3.4 OR ip=2.3.4.5 OR 2001:1:2:3::4) AND (port=23 OR port=2323) Otherwise might have to create a few hundred streams Greetings Mathias So back to the first question While "local config" / CLI or lookup would sometimes be helpful since it is usually easier to get around with. But a REST API would be really helpful to manage the configuration and get it automated.

mathiask · ‎12-12-2016

Hi Splunkers This is during parsing time .. not search time. Is there a way that I can use a lookup during parsing phase, and directly write the data into the log? i.e. DHCP information, since this is volatile. so lookup "src_ip" in the CSV/etc. and add this to the event as "src_name" And can I subsequently filter out events that based on that e.g. "src_name" = "unknown" drop event

mathiask · ‎12-12-2016

How can I manage the Splunk Stream Black and Whitelist? we have large white and black lists, with several hundred entries, which are also changing very frequently. Therefore the GUI is not really an option. I could not figure out where this information is "stored" and how to access it. How can I apply the black-/whitelist also to the NetFlows? As far as I could tell the whitelist was only applied to the traffic capturing but not to the parsed NetFlow events.

mathiask · ‎12-12-2016

I don't think this will work for us We cannot simply use a "where" statement, because we have to assume that the independent input process was adding new data matching the initial query. Also not all the fetched objects might be processed for various reasons. Therefore we need to do a bulk delete like "delete the following list of objects ..." Greetings Mathias

mathiask · ‎12-08-2016

Just for clarification and summary Use Case Shared Splunk Indexer Cluster Infrastructure Certain user groups get their own SH, they can do whatever they want with it, they own it, i.e. full system root access, they are Splunk Admin on that box, they can change any settings they want. Therefore local role and app restrictions will not do anything. Answer The splunk architecture is not designed for this. Access control and enforcement is done in the GUI and CLI on the SH. Therefore Anybody with Admin permissions on the SH can access and change permissions to all the data which the SH is connected to, because the SH has to connect with Splunk Admin credentials to the Search Peer. This is true for any kind of Search Head, whether in a cluster or not. The Indexer cannot/does not enforce access to indexes. Anybody with System root permission on the SH is able to get these permissions, somehow Follow up Question Is this also true when the SH connects not to a Indexer Cluster but as a normal Distributed Search Peer using credentials with the Splunk Admin role, but limited index access. E.g. splunk-adm-restricted has the admin role, but only access to the index main Possible "Solutions" The currently viable solution is to run independent infrastructures for each special user group. Directly send the corresponding events to the infrastructures. This allows the special user groups to access the data "locally" while some other users that are hooked up to all the infrastructures can access all the events. But this also creates some problems administration nightmare forwarder administration nightmare forwarder configuration nightmare, i.e. send system log to central and application to local infrastructure OR you have a central infrastructure where all the data is being sent to. And then forward the select data to the infrastructure of the special user groups. Still a total nightmare regarding infrastructure management, but the forwarder management is much easier. But you store the data more than once (except if you don't store it on the central one). The other option is to only give these user groups limited GUI/CLI access without Splunk admin permissions and no System root access. This limits somewhat the usage andr advanced features and configurations, but in general might be good enough.

mathiask · ‎12-07-2016

To clarify Setting up the config is not the problem. Splitting it into different apps etc was also not the problem. I dont know the underlying cause, but the moment i used useACK=true together with the indexer discovery feature I after some time received lots of duplicate events. The forwarder sent sometimes the event more than 10 times. This creates false statistics and eats up the license. This did not happen when the indexers were "hard" configured. Since 6.5 or maybe even minor update before, I did not encounter the problem anymore.

mathiask · ‎12-07-2016

This problem extends to lookups It is possible to make IPv4 CIDR lookups, but for IPv6 there are only string matches. https://answers.splunk.com/answers/407153/ipv6-lookup.html It would be nice to improve on this.

Posts	73
Solutions	3
Karma Given	16
Karma Received	33
Member Since	‎06-18-2014

Online Status	Offline
Date Last Visited	‎09-16-2020 06:52 AM

KV Store field type cidr

Apache Addon upgrade to Splunk 7.x

Splunk Stream: REST API tokens

Splunk Stream: Why do I receive the error "Unable ...

Splunk Stream: How to configure streams using REST...

Is it possible to use a lookup at index time, not ...

Splunk Stream: How to manage whitelists and blackl...

Is it possible limit user access to an index on a ...

Splunk Stream: How to configure forwarder and inde...

How to connect Splunk to MongoDB, or is there a KV...

Re: KV Store field type cidr

Re: KV Store field type cidr

KV Store field type cidr

Re: How to ignore case sensitive input in lookup f...

Re: How to ignore case sensitive input in lookup f...

Apache Addon upgrade to Splunk 7.x

Splunk Stream: REST API tokens

Re: How to connect Splunk to MongoDB, or is there ...

Re: Splunk Stream: Why do I receive the error "Una...

Re: Splunk Stream: Why do I receive the error "Una...

Splunk Stream: Why do I receive the error "Unable ...

Re: Splunk Stream: How to configure streams using ...

Splunk Stream: How to configure streams using REST...

Re: Splunk Stream: How to manage whitelists and bl...

Re: Splunk Stream: How to manage whitelists and bl...

Re: Is it possible to use a lookup at index time, ...

Re: Is it possible to use a lookup at index time, ...

Re: Indexer discovery where the client is the mast...

Re: Splunk Stream: How to manage whitelists and bl...

Is it possible to use a lookup at index time, not ...

Splunk Stream: How to manage whitelists and blackl...

Re: How to perform a bulk/batch delete of KV Store...

Re: Is it possible limit user access to an index o...

Re: Indexer discovery where the client is the mast...

Re: Feature Request: CIDR matching for IPv6 in sea...