I have to upgrade splunk enterprise (from 7.2.6 to 8.0.1 ) and enterprise security (from 5.3.0 to 6.0.0)
I am following the next documentation: https://docs.splunk.com/Documentation/Splunk/8.0.1/Installation/HowtoupgradeSplunk
But here does not talk about the upgrade order, I mean I have to upgrade first the splunk enterprise or first I have to upgrade the enterprise security?
You should be reading the upgrade docs for Enterprise Security. See Planning an upgrade of Splunk Enterprise Security in the Splunk Enterprise Security Installation and Upgrade Manual.