Hello, I'm hoping someone is able to help me find out what's going on with Splunk Stream and Netflow because I'm tearing my hair out trying to get it working. I have a separate indexer and search head and am trying to use the independent stream forwarder. The forwarder host also has UF installed but not Splunk_TA_stream, incidentally I tried getting it working with the Splunk_TA_stream app and was also seeing similar results. SH configuration: Splunk app for stream installed and configured as per https://docs.splunk.com/Documentation/StreamApp/7.3.0/DeployStreamApp/UseStreamtoingestNetflowandIPFIXdata#Configure_search_heads Indexer configuration: $SPLUNK_HOME/etc/apps/splunk_httpinput/local/inputs.conf   [http] disabled = 0 port = 8088 dedicatedIoThreads = 8 [http://streamfwd] description = Splunk Stream HEC disabled = 0 index = main token = <hec_token> indexes = _internal,main [splunk@<indexer> ~]$ netstat -antup | grep 8088 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 11580/splunkd   Independent forwarder setup: /opt/streamfwd/local/inputs.conf   [streamfwd://streamfwd] splunk_stream_app_location = https://<search_head>:8000/en-us/custom/splunk_app_stream/ stream_forwarder_id = disabled = 0    /opt/streamfwd/local/streamfwd.conf   [streamfwd://streamfwd] authToken = <auth_token_generated_by_curl_config> [streamfwd] httpEventCollectorToken = <HEC_TOKEN> processingThreads = 4 indexer.0.uri = https://<indexer>:8088 netflowReceiver.0.port = 9996 netflowReceiver.0.decoder = netflow netflowReceiver.0.ip = <forwarder_ip>   If i run the search index=main sourcetype="stream:*" the only events I see are:   { [-] endtime: 2020-12-22T12:18:36Z event_name: netFlowOptions exporter_ip: <router_ip> exporter_time: 2020-Dec-22 12:18:36 exporter_uptime: 4273621448 netflow_version: 9 observation_domain_id: 0 seqnumber: 340894 timestamp: 2020-12-22T12:18:36Z }   and running index=_internal sourcetype="stream:*" host="<forwarder>" gives me two sourcetypes, stream:log and stream:stats. stream:log gives me nothing of interest, just decode errors until the template is received, then these errors stop. stream:stats shows me:   { [-] agentMode: 1 ipAddress: <stream_forwarder_ip> netflow: { [-] NetflowDataHandlers: [ [-] { [-] NetflowDecoders: [ [-] { [-] name: Netflow processedRecords: 210991 } ] droppedPackets: 0 id: 0 } ] NetflowReceivers: [ [-] { [-] id: 0 recvdBytes: 8861500 running: true } ] eventsIn: 210964 eventsOut: 210964 id: NetflowManager running: true } osName: Linux senders: [ [-] { [-] busyConnections: 0 configTemplateName: connections: [ [-] { [-] endpoint: 0.0.0.0:0 id: 0 lastConnect: 2020-12-22T12:15:55.118285Z numErrors: 5 numSent: 20 queueSize: 0 status: closed workStatus: idle } { [-] endpoint: 0.0.0.0:0 id: 1 lastConnect: 2020-12-22T12:14:54.193007Z numErrors: 4 numSent: 27 queueSize: 0 status: closed workStatus: idle } { [-] endpoint: 0.0.0.0:0 id: 2 lastConnect: 2020-12-22T12:14:54.200473Z numErrors: 3 numSent: 20 queueSize: 0 status: closed workStatus: idle } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } { [+] } ] dateLastUpdated: 1608637900306 encrypted: true host: <search_head> id: <some_id> key: lastErrorCode: 0 name: numBytes: 4367915 numErrors: 41 numStreams: 1 openConnections: 0 port: 8000 requestsQueued: 0 requestsSent: 229 running: true streamForwarderGroups: [ [+] ] streamForwarderId: <forwarder_fqdn> streams: [ [-] { [-] bytes: 8016506 bytes_in: 8016506 bytes_out: 0 delta_bytes: 339112 delta_bytes_in: 339112 delta_bytes_out: 0 delta_events: 8924 delta_raw_bytes: 5889905 events: 210964 id: TEST_NETFLOW raw_bytes: 130470120 stats_only: 0 } ] } ] sniffer: { [+] } systemType: x86_64 versionNumber: 7.3.0 }   which suggests that netflow receivers are working as expected. Running a tcpdump on the receiver host I can see that I am receiving genuine netflow v9 which is readable using wireshark. I've looked at splunkd.log on the indexer and I'm not seeing anything that relates to the stream forwarder. I'm at a loss where to look next. I have gone through the documentation countless times over the last few days to make sure I'm not missing anything. Any help would be greatly appreciated! Thanks ... View more

Hi, I'm having an issue with a splunk lookup and I can't work out what the issue is. I have a lookup file, that among other things contains a mac address field and a hostname field mac, nt_host aabbccddeeff, machine1 a1b1c1d1e1f1, machine2 etc. I then have a search which returns data like the following: MAC: AA:BB:CC:DD:EE:FF interface: ge-X/X/XX switch: SWITCHNAME timestamp: 2019-10-14T09:02:02+00:00 I'm trying to match the mac in the lookup table and return the nt_host: index=os source="logfile" host="logfilehost" | eval mac = lower(trim(replace(MAC, ":", ""))) | lookup lookupfile mac OUTPUT nt_host But I get the following error: Streamed search execute failed because: Error in 'lookup' command: Could not construct lookup 'lookupfile, mac, OUTPUT, dns'. See search.log for more details.. If i run this search it works: index=os source="logfile" host="logfilehost" | eval mac = lower(trim(replace(MAC, ":", ""))) | table mac | lookup lookupfile mac OUTPUT nt_host but a subsequent stats command then returns the same error again. I'm unsure whats happening here and I see no errors in the search log. No idea what I'm missing here. Thanks ... View more

I'm trying to create a dashboard based on a number of Windows events and I have been banging my head up against this one all day. The event structure is always similar to: LogName=Security SourceName=Microsoft Windows security auditing. EventCode=4624 EventType=0 Type=Information ComputerName=$computername TaskCategory=Logon OpCode=Info RecordNumber=299255 Keywords=Audit Success Message=An account was successfully logged on. Subject: Security ID: NULL SID Account Name: - Account Domain: - Logon ID: 0x0 Logon Information: Logon Type: 3 Restricted Admin Mode: - Virtual Account: No Elevated Token: No Impersonation Level: Impersonation New Logon: Security ID: $DOMAIN\$username Account Name: $username Account Domain: $FQDN Logon ID: $LOGON_ID Linked Logon ID: 0x0 Network Account Name: - Network Account Domain: - Logon GUID: $LOGON_GUID I've been trying to extract the Security IDs as two separate fields and so far, in this particular context, I have failed. I've tried a number of things. The closest i've gotten is using a field transformation to combine the two and then splitting. However, the split is not always on the same delimiter, e.g. some Security IDs have a space between, so I can't split on that. I actually have this working for a different check using the following: index=wineventlog EventCode=4740 | eval temp=split(member_id,"$DOMAIN\\") | eval "Locked Account"=mvindex(temp,1) | rename Caller_Computer_Name as "Lockout Source",_time as "Lockout Event Time" | convert ctime("Lockout Event Time") | table "Lockout Event Time", "Locked Account", "Lockout Source" (member_id is a concatenation of both Security_IDs) However, this doesn't work for the above example because the 2nd Security ID is not always $DOMAIN\$username. Hope this makes sense! ... View more