vSphere administrators should know the components of virtual machines. There are multiple VMware file types that are associated with and make up a virtual machine. These files are located in the VM’s directory on a datastore. The following table will provide a quick reference and short description of a virtual machine’s files.
In order to view more information about the distributed switch configuration, use the net-dvs command. This is only available in the local shell. Notice that it specifies information like the UUID of the distributed switch and the name. We can also see information regarding Private VLANs if we have those set up.
If we keep scrolling down, we can see the MTU and CDP information for the distributed switch. Notice that we can set up LLDP for a distributed switch. Next we see information regarding the port groups and how they are configured, we see VLAN and security policy information here. At the bottom we see some stuff on a network resource pool if we have network i/o control enabled and are using this feature.
The last section we see on the net-dvs output we see is some information that is very useful during the troubleshooting process. We can see whether or not packets are being dropped and we can see from the amount of traffic going in and out and decide on whether we need to traffic shape.
There are many useful things to look at when in the memory view of esxtop.
Several important things to look at near the top of the esxtop.
PMEM /MB – memory for the host
VMKMEM /MB – memory for the VMkernel
PSHARE /MB – ESXi page sharing statistics
SWAP /MB – ESXi swap usage statistics
ZIP /MB – ESXi compression statistics
MEMCTL /MB – ESXi balloon statistics
Now looking at the virtual machines down below host information, you can see several counters listed that can be of use when troubleshooting an individual VM or group of VMs:
MEMSZ – amount of configured guest physical memory
GRANT – amount of guest physical memory granted
SZTGT – amount of memory to be allocated to a machine
TCHD – amount of guest physical memory recently used by the VM
TCHD_W – write working set estimate for a resource pool
SWCUR – current swap usage
SWTGT – expected swap usage
SWR/s – swap in from disk rate
SWW/s – swap out to disk rate
LLSWR/s – memory read from host cache rate
LLSWW/s – memory write to host cache rate
OVHDUW – overhead memory reserved for the vmx user world of a VM group.
OVHD – amount of overhead currently consumed by a VM
OVHDMAX – amount of reserved overhead memory for a VM
Ideally, you’ll look at esxtop and never see any kind of numbers for balloon, compression or swap activity. However if you do see this activity then the ESXi host is overcommitted and is in contention. More resources need to be added the the ESXi host, the cluster or some of the VMs need to be moved to an ESXi host with memory resources available.
The default view of esxtop is CPU, there are several useful counters in this view.
GID – group ID
NAME – virtual machine name
NWLD – number of worlds
%USED – percentage physical CPU time accounted to this world
%RUN – percentage of total scheduled time for the world to run
%SYS – percentage of time spend by system services for that world
%WAIT – percentage of time spent by the world in a wait state
%VMWAIT – derivative of %WAIT except it doesn’t include %IDLE
%RDY – percentage of time the world was ready to run
%IDLE – percentage of time the vCPU world is in idle loop
%OVRLP – percentage of time spend by system services on behalf of other worlds
%CSTP – percentage of time the world spend in ready, co-deschedule state (only relevant to SMP VMs)
%MLMTD – percentage of time world was ready to run but was not scheduled because that would violate “CPU limit” settings
%SWPWT – percentage of time the world is waiting for the VMkernel swapping memory
High CPU ready time is a major indicator of CPU performance issues, you may have excessive usage of vSMP or a limit set (check %MLMTD for that). Another metric to check is %CSTP, this will help you determine whether you can decrease the amount of vCPUs for some of the virtual machines which will help with improving scheduling opportunities.
%SYS is usually caused by high IO virtual machine. %SWPWT is usually caused by memory overcommitment.
The last post discussed navigating esxtop, now let’s get into each view a little bit more.
There are several network counters that are default when you go to the networking view, here’s a brief overview of each:
PKTTX/s – # of packets transmitted per second
MbTX/s – MegaBits transmitted per second
PKTRX/s – # of packets received per second
MbRX/s – MegaBits received per second
%DRPTX – percentage of transmit packets dropped
%DRPRX – percentage of receive packets dropped
A major indicator of potential network performance issues is dropped packets. This can be indicative of a physical device failing, queue congestion, bandwidth issues, etc.
Something else to check when having network issues is high CPU usage, the CPU Ready Time counter (%RDY) can be beneficial when diagnosing CPU issues.
If you are having these issues in your environment, consider using jumbo frames, taking advance of hardware features provided by the NIC like TSO (TCP Segmentation Offload) and TCO (TCP Checksum Offload)
Also, make sure to check out physical network trunks, interswitch links, etc for overloaded pipes.
Consider: moving the VM with high network demand to another switch, adding more uplinks to a virtual switch and check for which vNIC driver is being used.
A tool that is very useful is “esxtop.” This command-line tools allows monitoring and collecting of data for the core four resources: CPU, memory, network, and disk.
After enabling SSH on an ESXi host, open up PuTTY and connect to that ESXi host using your root account and password.
Start running esxtop by typing the command on a single line:
esxtop
From any view, I can type a “V” (shift + v) to parse the list and only view virtual machine information.
Organization Network
An organization network provides network services to one particular organization, whereas an external network is created at the provider level and supplies connectivity to multiple organizations. There are three options when creating organization networks: internal, NAT-connected, and direct-connected. An organization administrator cannot create an organization network due to the configuration of external IPs; only a system administrator can configure this.
Internal
An organization can be set up so that it does not have a connection to the Internet or a connection to any other external network, just an internal connection. An internal-only network could be set up for groups of test virtual machines; a virtual machine can be configured with multiple network interfaces so that it has a connection to the internal network as well as one of the other two types. With an internal organization network, vApps can connect, but there is no traffic outside the organization.
Network Address Translation (NAT)-Connected
Network Address Translation (NAT)-connected, sometimes called a “routed network,” can be connected to the external network through a vShield Edge device. The vShield Edge device provides port-forwarding services, NAT, DNS forwarding, and DHCP services to the network; the vShield Edge device gets provisioned automatically
by vCloud Director as needed. A NAT connection allows for virtual machines to communicate with each other while only having one IP seen from the Internet. Another use of NAT is to fence, which includes two sets of IP addresses: external and internal. Fencing allows for several vApps to utilize the same internal IP addresses and extremely useful for test environments.
Direct Communication
The last option for an organization network is a direct connection. The organization would use an external net- work to connect to external systems, including the Internet. Using this method, a user can connect directly to a virtual machine using remote desktop or even SSH. If a vApp configured for a direct connection then the vApp’s IP addresses must be statically assigned or a DHCP server must be connected to the external providing the vApp with those IP addresses.
For further reading, check out my vCloud Director 5.1 Networking Concepts white paper!
A VMware vCloud is made up of one or more vCloud Director servers that are integrated with underlying vSphere components. The vCloud is a new abstraction layer above vCenter Server consuming the resources that vCenter manages; this allows a user to self-provision virtual environments utilizing memory, compute, storage, and networking resources. Cloud computing has become a vague, arbitrary phrase, but there are six characteris- tics that define exactly what a cloud should consist of
A private cloud is an infrastructure whose resources are only used internally. A public cloud is an infrastructure made available to external customers for a price. A hybrid cloud combines two or more clouds with some kind of standardized technology, like VMware vCloud Connector, while each cloud maintains its own unique identity.
The foundation of the vCloud centers on the networking configuration. Networking occurs over three different layers: external, organization, and vApp; it is imperative to properly configure and manage these networks so that the vCloud can be consumed. Think of vCloud networking as an onion that will be peeled back to reveal each layer, starting with the organization’s networks that are created by an administrator with the system administrator role in vCloud Director. A system administrator is the highest role within the vCloud.
For further reading, check out my vCloud Director 5.1 Networking Concepts white paper!
The first object that is created within vCloud Director is the External Network. An External Network provides the connection from the cloud to the outside world, allowing inter-Cloud connections and is port group based. Even though this connection is called the external connection, an Internet connection is not actually required; this can be set up to provide a connection to several different internal entities, like ESXi hosts, without an actual route to the Internet. Since this connection is port group-based, then the port group needs to exist prior to attempting to establish the connection. The port group can be defined on a standard vSwitch, a distributed vSwitch, or on a Nexus 1000V. Organization virtual datacenters can use the external networks to provide Internet connectivity to the organizations and the virtual machines that reside within a vApp, given that the vApp network is configured for that. By creating an external network, vCloud Director is effectively configured to send all external traffic using the port group(s) selected. Should there be multiple external networks created then be sure to separate them by using VLANs. Only someone with the system administrator role within the vCloud can create and manage external networks.
For further reading, check out my vCloud Director 5.1 Networking Concepts white paper!
TECHNICLOUD 