Hello,
This may be more historical than anything and then again architectural. Consider the case where you are sharing two networks on a pair of pNICs. For arguments sake we can call the first network the management network (where access to this network implies you can own the systems easily) and another network, say a DMZ. Now we have them setup as follows:
Outside (VLAN 200) <-> pNIC1 <-> vSwitch0 <-> pgDMZ (VLAN 200) <-> DMZ VMs
Inside (VLAN 100) <-> pNIC0 <-> vSwitch0 <-> pgMgmt (VLAN 100) <-> Mgmt VMs
On failure of pNIC1, traffic for DMZ would route through pNIC0 as the uplink would include VLAN 100 and 200. If we did not failback to pNIC1 when it came back online then we would run into a fairly major security issue from Availability concerns as a DMZ is a Noisy neighbor during a DoS attack, which could prevent management actions such as starting a vMotion, etc. In a degraded mode things like this are possible and hence why failback is enabled by default. If you are not using VLANs then traffic could co-mingle and you would have some other serious security issues around confidentiality and integrity of your actual hosts.
Granted, ideally you would never mix these two VLANs on the same vSwitch but if you use 10G links your port costs are much higher than 1G links and therefore you may only have a pair of 10G links to use.
Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, 2010, 2011,2012,2013,2014
Author of the books 'VMWare ESX and ESXi in the Enterprise: Planning Deployment Virtualization Servers', Copyright 2011 Pearson Education. 'VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment', Copyright 2009 Pearson Education.
Virtualization and Cloud Security Analyst: The Virtualization Practice, LLC -- vSphere Upgrade Saga -- Virtualization Security Round Table Podcast