Welcome!

Microsoft Cloud Authors: Dana Gardner, Yung Chou, David Bermingham, Jayaram Krishnaswamy, Pat Romanski

Blog Feed Post

Performance Tuning Windows 2012: Network Subsystem Part 2

In our previous article we discussed the hardware supported features of some of the high-end network adapters. Let’s take a look how you can use some of those settings to their best advantage. Remember that the correct settings depend on the network adapter, your workload, the resources of the host computer, and of course your performance goals.

Enabling Offload Features

Turning on network adapter offload features typically will benefit performance.However, sometimes the network adapter might not be powerful enough to handle the offload capabilities with high throughput. For example, enabling segmentation offload can reduce the maximum sustainable throughput on some network adapters because of limited hardware resources. However, if the reduced throughput is not expected to be a limitation, you should enable offload capabilities, even for those network adapters. Some network adapters require offload features to be independently enabled for send and receive paths.

Enabling RSS for Web Scenarios

RSS can improve web scalability and performance if you have fewer network adapters than logical processors in your server. When all the web traffic is going through the RSS-capable network adapters, incoming web requests from different connections can be simultaneously processed across different CPUs. Because of the logic in RSS and HTTP for load distribution, performance can be degraded if a non-RSS-capable network adapter accepts web traffic on a server that has one or more RSS-capable network adapters. It is recommended that you use RSS-capable network adapters or disable RSS from the Advanced Properties tab. To determine whether a network adapter is RSS-capable, view the RSS information on the Advanced Properties tab for the device.

RSS Profiles and RSS Queues

RSS Profiles are new in Windows Server 2012. The default profile is NUMA Static, which changes the default behavior from previous versions of Windows. We suggest reviewing the available profiles and understanding when they are beneficial. For example, you can use Task Manager to examine if your logical processors is underutilized for receive traffic, and try increasing the number of RSS queues from the default of 2 to the maximum that your network adapter supports. Changing the number of RSS queues is an option your network adapter may have as part of the driver.

Increasing Network Adapter Resources

For network adapters that allow manual configuration of resources, such as receive and send buffers, you should increase the allocated resources. Some network adapters set their receive buffers low to conserve allocated memory from the host. The low value results in dropped packets and decreased performance. Therefore, for receive-intensive scenarios, it is recommended to increase the receive buffer value to the maximum. If your adapter does not supports or exposes manual configuration, it most likely dynamically configures the resources, or it might be a fixed value that you can’t change.

Enabling Interrupt Moderation

To control interrupt moderation, some network adapters expose different interrupt moderation levels, buffer coalescing parameters (sometimes separately for send and receive buffers), or both. You should consider interrupt moderation for CPU-bound workloads, and consider the trade-off between the host CPU savings and latency versus the increased host CPU savings because of more interrupts and less latency. If the network adapter does not perform interrupt moderation, but it does expose buffer coalescing, increasing the number of coalesced buffers allows more buffers per send or receive, which improves performance.

Workload Specific Tuning

Your network adapter has a number of options to optimize latency caused by the operating system. This latency is the time between the network driver processing an incoming packet and the sending it back, and is usually measured in microseconds. As a comparison, the transmission time for packets transmitted over long distances is usually expressed in milliseconds. This tuning will not reduce the time a packet spends in transit.

Some tuning suggestions for microsecond-sensitive networks include:

· Set the computer BIOS to High Performance, with C-states disabled. However, this is system and BIOS dependent. Some systems provide higher performance if the operating system controls power management. You can check and adjust your power management settings from the Control Panel or by using the powercfg command.

· Set the operating system power management profile to High Performance System. For this to work as expected, your system’s BIOS has to be set to enable operating system control of power management.

· Enable Static Offloads, for example, UDP Checksums, TCP Checksums, and Send Large Offload (LSO)

· Enable RSS if the traffic is multi-streamed, such as high-volume multicast receive

· Disable the Interrupt Moderation setting for network drivers that require the lowest possible latency. The tradeoff is that this can use more CPU time.

· Handle network adapter interrupts and DPCs on a core processor that shares CPU cache with the core that is being used by the program that is handling the packet. CPU affinity tuning can be used to direct a process to certain logical processors in conjunction with RSS configuration to accomplish this. Using the same core for the interrupt, DPC, and user mode thread exhibits worse performance as load increases because the ISR, DPC, and thread contend for the use of the core.

System Management Interrupts

Many hardware systems use System Management Interrupts (SMI) for a variety of maintenance functions, including reporting of ECC memory errors, legacy USB compatibility, fan control, and BIOS controlled power management. The SMI is the highest priority interrupt on the system and places the CPU in a management mode, which preempts all other activity while it runs an interrupt service routine, typically contained in BIOS.

This behavior can result in latency spikes of 100 microseconds or more. If you need to achieve the lowest latency, look for a BIOS version from your hardware provider that reduces SMIs to the lowest degree possible, referred to as “low latency BIOS” or “SMI free BIOS.” It is not possible to eliminate SMI activity altogether because it is used to control some essential functions, such as fan control.

Tuning TCP

TCP Receive Window Auto-Tuning

Prior to Windows Server 2008, the network stack used a fixed-size receive-side window that limited the overall potential throughput for connections. One of the most significant changes to the TCP stack is TCP receive window auto-tuning. You can calculate the total throughput of a single connection when you use this fixed size default as:

Total achievable throughput in bytes = TCP window * (1 / connection latency)

As an example, the achievable throughput is only 51 Mbps on a 1 GB connection with 10ms latency. With auto-tuning, the receive-side window is adjustable, and it can grow to meet the demands of the sender. It is possible for a connection to achieve a full line rate of a 1 GB connection. Network usage requirements that might have been limited in the past by the total achievable throughput of TCP connections can now fully use the network.

Windows Filtering Platform

The Windows Filtering Platform (WFP) that was introduced in Windows Vista and Windows Server 2008 provides APIs to non-Microsoft independent software vendors (ISVs) to create packet processing filters. Examples include firewall and antivirus software. Be careful that a poorly written WFP filter can significantly decrease networking performance.

TCP Parameters

The following registry keywords from Windows Server 2003 are no longer supported, and they are ignored in Windows Server 2012, as well as Windows Server 2008 R2, and Windows Server 2008:

· TcpWindowSize – HKLM\System\CurrentControlSet\Services\Tcpip\Parameters

· NumTcbTablePartitions – HKLM\system\CurrentControlSet\Services\Tcpip\Parameters

· MaxHashTableSize – HKLM\system\CurrentControlSet\Services\Tcpip\Parameters

Network-Related Performance Counters

This section lists the counters that are relevant to managing network performance.

Resource Utilization

· IPv4, IPv6

  • Datagrams Received/sec
  • Datagrams Sent/sec

· TCPv4, TCPv6

  • Segments Received/sec
  • Segments Sent/sec
  • Segments Retransmitted/sec

· Network Interface(*), Network Adapter(*)

  • Bytes Received/sec
  • Bytes Sent/sec
  • Packets Received/sec
  • Packets Sent/sec
  • Output Queue Length

This counter is the length of the output packet queue (in packets). If this is longer than 2, delays occur. You should find the bottleneck and eliminate it if you can. Because NDIS queues the requests, this length should always be 0.

· Processor Information

  • % Processor Time
  • Interrupts/sec
  • DPCs Queued/sec

This counter is an average rate at which DPCs were added to the logical processor’s DPC queue. Each logical processor has its own DPC queue. This counter measures the rate at which DPCs are added to the queue, not the number of DPCs in the queue. It displays the difference between the values that were observed in the last two samples, divided by the duration of the sample interval.

Potential Network Problems

· Network Interface(*), Network Adapter(*)

  • Packets Received Discarded
  • Packets Received Errors
  • Packets Outbound Discarded
  • Packets Outbound Errors

· WFPv4, WFPv6

  • Packets Discarded/sec

· UDPv4, UDPv6

  • Datagrams Received Errors

· TCPv4, TCPv6

  • Connection Failures
  • Connections Reset

· Network QoS Policy

  • Packets dropped
  • Packets dropped/sec

· Per Processor Network Interface Card Activity

  • Low Resource Receive Indications/sec
  • Low Resource Received Packets/sec

· Microsoft Winsock BSP

  • Dropped Datagrams
  • Dropped Datagrams/sec
  • Rejected Connections
  • Rejected Connections/sec

Receive Side Coalescing (RSC) performance

· Network Adapter(*)

  • TCP Active RSC Connections
  • TCP RSC Average Packet Size
  • TCP RSC Coalesced Packets/sec
  • TCP RSC Exceptions/sec
Share Now:del.icio.usDiggFacebookLinkedInBlinkListDZoneGoogle BookmarksRedditStumbleUponTwitterRSS

Read the original blog entry...

More Stories By Hovhannes Avoyan

Hovhannes Avoyan is the CEO of PicsArt, Inc.,

@ThingsExpo Stories
SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management...
SYS-CON Events announced today that VAI, a leading ERP software provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. VAI (Vormittag Associates, Inc.) is a leading independent mid-market ERP software developer renowned for its flexible solutions and ability to automate critical business functions for the distribution, manufacturing, specialty retail and service sectors. An IBM Premier Business Part...
With an estimated 50 billion devices connected to the Internet by 2020, several industries will begin to expand their capabilities for retaining end point data at the edge to better utilize the range of data types and sheer volume of M2M data generated by the Internet of Things. In his session at @ThingsExpo, Don DeLoach, CEO and President of Infobright, will discuss the infrastructures businesses will need to implement to handle this explosion of data by providing specific use cases for filte...
Cognitive Computing is becoming the foundation for a new generation of solutions that have the potential to transform business. Unlike traditional approaches to building solutions, a cognitive computing approach allows the data to help determine the way applications are designed. This contrasts with conventional software development that begins with defining logic based on the current way a business operates. In her session at 18th Cloud Expo, Judith S. Hurwitz, President and CEO of Hurwitz & ...
Fortunately, meaningful and tangible business cases for IoT are plentiful in a broad array of industries and vertical markets. These range from simple warranty cost reduction for capital intensive assets, to minimizing downtime for vital business tools, to creating feedback loops improving product design, to improving and enhancing enterprise customer experiences. All of these business cases, which will be briefly explored in this session, hinge on cost effectively extracting relevant data from ...
SYS-CON Events announced today that Pythian, a global IT services company specializing in helping companies adopt disruptive technologies to optimize revenue-generating systems, has been named “Bronze Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. Founded in 1997, Pythian is a global IT services company that helps companies compete by adopting disruptive technologies such as cloud, Big Data, advanced analytics, and DevO...
Companies can harness IoT and predictive analytics to sustain business continuity; predict and manage site performance during emergencies; minimize expensive reactive maintenance; and forecast equipment and maintenance budgets and expenditures. Providing cost-effective, uninterrupted service is challenging, particularly for organizations with geographically dispersed operations.
SYS-CON Events announced today that Avere Systems, a leading provider of enterprise storage for the hybrid cloud, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Avere delivers a more modern architectural approach to storage that doesn’t require the overprovisioning of storage capacity to achieve performance, overspending on expensive storage media for inactive data or the overbuilding of data centers ...
With the Apple Watch making its way onto wrists all over the world, it’s only a matter of time before it becomes a staple in the workplace. In fact, Forrester reported that 68 percent of technology and business decision-makers characterize wearables as a top priority for 2015. Recognizing their business value early on, FinancialForce.com was the first to bring ERP to wearables, helping streamline communication across front and back office functions. In his session at @ThingsExpo, Kevin Roberts...
As enterprises work to take advantage of Big Data technologies, they frequently become distracted by product-level decisions. In most new Big Data builds this approach is completely counter-productive: it presupposes tools that may not be a fit for development teams, forces IT to take on the burden of evaluating and maintaining unfamiliar technology, and represents a major up-front expense. In his session at @BigDataExpo at @ThingsExpo, Andrew Warfield, CTO and Co-Founder of Coho Data, will dis...
SYS-CON Events announced today that iDevices®, the preeminent brand in the connected home industry, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. iDevices, the preeminent brand in the connected home industry, has a growing line of HomeKit-enabled products available at the largest retailers worldwide. Through the “Designed with iDevices” co-development program and its custom-built IoT Cloud Infrastruc...
Eighty percent of a data scientist’s time is spent gathering and cleaning up data, and 80% of all data is unstructured and almost never analyzed. Cognitive computing, in combination with Big Data, is changing the equation by creating data reservoirs and using natural language processing to enable analysis of unstructured data sources. This is impacting every aspect of the analytics profession from how data is mined (and by whom) to how it is delivered. This is not some futuristic vision: it's ha...
Silver Spring Networks, Inc. (NYSE: SSNI) extended its Internet of Things technology platform with performance enhancements to Gen5 – its fifth generation critical infrastructure networking platform. Already delivering nearly 23 million devices on five continents as one of the leading networking providers in the market, Silver Spring announced it is doubling the maximum speed of its Gen5 network to up to 2.4 Mbps, increasing computational performance by 10x, supporting simultaneous mesh communic...
The cloud promises new levels of agility and cost-savings for Big Data, data warehousing and analytics. But it’s challenging to understand all the options – from IaaS and PaaS to newer services like HaaS (Hadoop as a Service) and BDaaS (Big Data as a Service). In her session at @BigDataExpo at @ThingsExpo, Hannah Smalltree, a director at Cazena, will provide an educational overview of emerging “as-a-service” options for Big Data in the cloud. This is critical background for IT and data profes...
SYS-CON Events announced today that Men & Mice, the leading global provider of DNS, DHCP and IP address management overlay solutions, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. The Men & Mice Suite overlay solution is already known for its powerful application in heterogeneous operating environments, enabling enterprises to scale without fuss. Building on a solid range of diverse platform support,...
SYS-CON Events announced today that Alert Logic, Inc., the leading provider of Security-as-a-Service solutions for the cloud, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Alert Logic, Inc., provides Security-as-a-Service for on-premises, cloud, and hybrid infrastructures, delivering deep security insight and continuous protection for customers at a lower cost than traditional security solutions. Ful...
SYS-CON Events announced today that Interoute, owner-operator of one of Europe's largest networks and a global cloud services platform, has been named “Bronze Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. Interoute is the owner-operator of one of Europe's largest networks and a global cloud services platform which encompasses 12 data centers, 14 virtual data centers and 31 colocation centers, with connections to 195 ad...
One of the bewildering things about DevOps is integrating the massive toolchain including the dozens of new tools that seem to crop up every year. Part of DevOps is Continuous Delivery and having a complex toolchain can add additional integration and setup to your developer environment. In his session at @DevOpsSummit at 18th Cloud Expo, Miko Matsumura, Chief Marketing Officer of Gradle Inc., will discuss which tools to use in a developer stack, how to provision the toolchain to minimize onboa...
SYS-CON Events announced today that Fusion, a leading provider of cloud services, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Fusion, a leading provider of integrated cloud solutions to small, medium and large businesses, is the industry's single source for the cloud. Fusion's advanced, proprietary cloud service platform enables the integration of leading edge solutions in the cloud, including clou...
Most people haven’t heard the word, “gamification,” even though they probably, and perhaps unwittingly, participate in it every day. Gamification is “the process of adding games or game-like elements to something (as a task) so as to encourage participation.” Further, gamification is about bringing game mechanics – rules, constructs, processes, and methods – into the real world in an effort to engage people. In his session at @ThingsExpo, Robert Endo, owner and engagement manager of Intrepid D...