All Apps and Add-ons

Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah
Communicator

Are there any specific ports or specific permissions this add-on requires/uses, so that I can inform the team, so if any modifications are made data flow is not interrupted.

I have configured Microsoft Log Analytics Add-on in Heavy Forwarder and forwarding the logs received to indexer. There is no clustering. I would like to hear from @jkat54 and @dpanych. Any ideas, why this keep on happening.

I used

index=_internal log_level=err* OR log_level=warn loganalytics*

The latest event I am getting some results using this query is

09-05-2018 18:24:24.168 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERROR('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
1 Solution

jkat54
SplunkTrust
SplunkTrust

It connects to the log analytics API on TCP port 443 aka HTTPS.

Nothing else is needed.

View solution in original post

0 Karma

jkat54
SplunkTrust
SplunkTrust

Can you try changing the following in input_module_log_analytics.py?

Please change:

    #Delta
    state = now_dt.strftime("%d/%m/%Y %H:%M:%S")

To:

    #Delta
    state = start_datetime.strftime("%d/%m/%Y %H:%M:%S")

And let me know if that fixes this bug please.

0 Karma

phularah
Communicator

Hi @jkat54. I was just waiting for the problem to arise again but looks like we are not getting same problem of data stoppage now in almost a week. I did not made this change

state = start_datetime.strftime("%d/%m/%Y %H:%M:%S")

The only change that I had made after last problem was to copy props.conf from Heavy forwarder and pasting it in apps\search\local local in Search Head. No changes were made in OMS as per my knowledge. I am still not sure what was the problem initially, but everything is working fine now.

Thanks for writing this app. Keep up the good work.

jkat54
SplunkTrust
SplunkTrust

Ok, thanks and you’re welcome!

0 Karma

jkat54
SplunkTrust
SplunkTrust

It connects to the log analytics API on TCP port 443 aka HTTPS.

Nothing else is needed.

0 Karma

phularah
Communicator

I am not sure why is it happening then. I tried making a new input but still I am unable to see any data. Any idea why it might be happening?

0 Karma

phularah
Communicator

After making new inputs and deleting old inputs, I am getting data now. But, I don't know why it stopped in the first place and now after first making a new input and disabling previous input, data didn't come. I again made a new input after deleting all disabled inputs, now I am getting data. It is really frustrating, and I am unable to pinpoint the source.

Now, new results are coming where log_level=warn,

09-06-2018 10:50:34.492 +0200 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 14261 - data_source="log_analytics://analytics", data_host="Hostname", data_sourcetype="loganalytics"

but it is of no concern, my concern is this error.
Could you please tell why this error might show up.

09-05-2018 18:24:24.168 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERROR('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

0 Karma

jkat54
SplunkTrust
SplunkTrust

Sounds like the checkpoints were messed up. Did you upgrade from a previous version of the app or install fresh/new?

0 Karma

phularah
Communicator

Yes, I had upgraded the add-on. I was using the latest add-on before this problem surfaced.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Ok looks like the process for upgrading should have been to delete the inputs, upgrade the app, add the inputs.

this is due to the way the checkpoints changed between versions. My apologies for the inconvenience.

Is every input you’ve re-added working now?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Was everything working after removing and adding the inputs after the upgrade?

0 Karma

493669
Super Champion

@jkat54,
is there any retry attempts for input in OMS?
like due to some reason in oms, splunk unable to collect data for 5 min. but after 5 min. everything fine at OMS side...but then splunk unable to receive any kind of data ..it get stopped so is there any retry attempt like it will try to connect with OMS for few attempts and then it will stop attemptting to connect with OMS?

0 Karma

jkat54
SplunkTrust
SplunkTrust

The “connection broken” error suggests a proxy or firewall or other network issue.

No one else is reporting this error so I believe it to be something with your environment only.

As for if the add on retries connections, no it only attempts one connection per interval.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Wait... this question wasn’t by you. @493669 do you work with phularah?

If not, you should create your own question.

If yes, then see this link for how to resolve the error you have: https://docs.microsoft.com/en-us/azure/active-directory/application-sign-in-problem-federated-sso-ga...

0 Karma

phularah
Communicator

Yes, @jkat me and @493669 are working together. My comments were sent to moderator, so I asked my colleague @493669 to post these comments. Now, I can see my posted comments.

phularah
Communicator

@jkat54 , but if we have any error at oms side then data will not come... but after I deleted previous input and created new input , data started flowing again...so it doesn't seems to be issue at OMS side..isn't it?

0 Karma

jkat54
SplunkTrust
SplunkTrust

When you recreate do you recreate in splunk only?

0 Karma

phularah
Communicator

Yes, we create new inputs in Splunk only.

0 Karma

phularah
Communicator

can you please share your take on this, why it might be happening? I had a chat with Azure guys in my team, they say everything is working fine at their side and they are not making any changes.

0 Karma

phularah
Communicator

Also, we are getting these messages running this query, index=_internal log_level=err* loganalytics*. Again after working for a few hours data has again stopped coming.

These are the error messages, we are getting:

9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" UnboundLocalError: local variable 'data' referenced before assignment

    host =  *****   
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"     for i in range(len(data["tables"][0]["rows"])):

    host =  ********    
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\input_module_log_analytics.py", line 86, in collect_events

    host =  ******  
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"     input_module.collect_events(self, ew)

    host =  *****
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py", line 96, in collect_events

    host =  *****
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"     self.collect_events(ew)

    host =  *******
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 
0 Karma

jkat54
SplunkTrust
SplunkTrust

This means the query didn’t return any results.

Any idea how long it takes before it breaks?

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...