Welcome!

Microsoft Cloud Authors: Elizabeth White, Mihai Corbuleac, Pat Romanski, David Bermingham, Steven Mandel

Blog Feed Post

Performance Tuning Windows 2012: Network Subsystem Part 2

In our previous article we discussed the hardware supported features of some of the high-end network adapters. Let’s take a look how you can use some of those settings to their best advantage. Remember that the correct settings depend on the network adapter, your workload, the resources of the host computer, and of course your performance goals.

Enabling Offload Features

Turning on network adapter offload features typically will benefit performance.However, sometimes the network adapter might not be powerful enough to handle the offload capabilities with high throughput. For example, enabling segmentation offload can reduce the maximum sustainable throughput on some network adapters because of limited hardware resources. However, if the reduced throughput is not expected to be a limitation, you should enable offload capabilities, even for those network adapters. Some network adapters require offload features to be independently enabled for send and receive paths.

Enabling RSS for Web Scenarios

RSS can improve web scalability and performance if you have fewer network adapters than logical processors in your server. When all the web traffic is going through the RSS-capable network adapters, incoming web requests from different connections can be simultaneously processed across different CPUs. Because of the logic in RSS and HTTP for load distribution, performance can be degraded if a non-RSS-capable network adapter accepts web traffic on a server that has one or more RSS-capable network adapters. It is recommended that you use RSS-capable network adapters or disable RSS from the Advanced Properties tab. To determine whether a network adapter is RSS-capable, view the RSS information on the Advanced Properties tab for the device.

RSS Profiles and RSS Queues

RSS Profiles are new in Windows Server 2012. The default profile is NUMA Static, which changes the default behavior from previous versions of Windows. We suggest reviewing the available profiles and understanding when they are beneficial. For example, you can use Task Manager to examine if your logical processors is underutilized for receive traffic, and try increasing the number of RSS queues from the default of 2 to the maximum that your network adapter supports. Changing the number of RSS queues is an option your network adapter may have as part of the driver.

Increasing Network Adapter Resources

For network adapters that allow manual configuration of resources, such as receive and send buffers, you should increase the allocated resources. Some network adapters set their receive buffers low to conserve allocated memory from the host. The low value results in dropped packets and decreased performance. Therefore, for receive-intensive scenarios, it is recommended to increase the receive buffer value to the maximum. If your adapter does not supports or exposes manual configuration, it most likely dynamically configures the resources, or it might be a fixed value that you can’t change.

Enabling Interrupt Moderation

To control interrupt moderation, some network adapters expose different interrupt moderation levels, buffer coalescing parameters (sometimes separately for send and receive buffers), or both. You should consider interrupt moderation for CPU-bound workloads, and consider the trade-off between the host CPU savings and latency versus the increased host CPU savings because of more interrupts and less latency. If the network adapter does not perform interrupt moderation, but it does expose buffer coalescing, increasing the number of coalesced buffers allows more buffers per send or receive, which improves performance.

Workload Specific Tuning

Your network adapter has a number of options to optimize latency caused by the operating system. This latency is the time between the network driver processing an incoming packet and the sending it back, and is usually measured in microseconds. As a comparison, the transmission time for packets transmitted over long distances is usually expressed in milliseconds. This tuning will not reduce the time a packet spends in transit.

Some tuning suggestions for microsecond-sensitive networks include:

· Set the computer BIOS to High Performance, with C-states disabled. However, this is system and BIOS dependent. Some systems provide higher performance if the operating system controls power management. You can check and adjust your power management settings from the Control Panel or by using the powercfg command.

· Set the operating system power management profile to High Performance System. For this to work as expected, your system’s BIOS has to be set to enable operating system control of power management.

· Enable Static Offloads, for example, UDP Checksums, TCP Checksums, and Send Large Offload (LSO)

· Enable RSS if the traffic is multi-streamed, such as high-volume multicast receive

· Disable the Interrupt Moderation setting for network drivers that require the lowest possible latency. The tradeoff is that this can use more CPU time.

· Handle network adapter interrupts and DPCs on a core processor that shares CPU cache with the core that is being used by the program that is handling the packet. CPU affinity tuning can be used to direct a process to certain logical processors in conjunction with RSS configuration to accomplish this. Using the same core for the interrupt, DPC, and user mode thread exhibits worse performance as load increases because the ISR, DPC, and thread contend for the use of the core.

System Management Interrupts

Many hardware systems use System Management Interrupts (SMI) for a variety of maintenance functions, including reporting of ECC memory errors, legacy USB compatibility, fan control, and BIOS controlled power management. The SMI is the highest priority interrupt on the system and places the CPU in a management mode, which preempts all other activity while it runs an interrupt service routine, typically contained in BIOS.

This behavior can result in latency spikes of 100 microseconds or more. If you need to achieve the lowest latency, look for a BIOS version from your hardware provider that reduces SMIs to the lowest degree possible, referred to as “low latency BIOS” or “SMI free BIOS.” It is not possible to eliminate SMI activity altogether because it is used to control some essential functions, such as fan control.

Tuning TCP

TCP Receive Window Auto-Tuning

Prior to Windows Server 2008, the network stack used a fixed-size receive-side window that limited the overall potential throughput for connections. One of the most significant changes to the TCP stack is TCP receive window auto-tuning. You can calculate the total throughput of a single connection when you use this fixed size default as:

Total achievable throughput in bytes = TCP window * (1 / connection latency)

As an example, the achievable throughput is only 51 Mbps on a 1 GB connection with 10ms latency. With auto-tuning, the receive-side window is adjustable, and it can grow to meet the demands of the sender. It is possible for a connection to achieve a full line rate of a 1 GB connection. Network usage requirements that might have been limited in the past by the total achievable throughput of TCP connections can now fully use the network.

Windows Filtering Platform

The Windows Filtering Platform (WFP) that was introduced in Windows Vista and Windows Server 2008 provides APIs to non-Microsoft independent software vendors (ISVs) to create packet processing filters. Examples include firewall and antivirus software. Be careful that a poorly written WFP filter can significantly decrease networking performance.

TCP Parameters

The following registry keywords from Windows Server 2003 are no longer supported, and they are ignored in Windows Server 2012, as well as Windows Server 2008 R2, and Windows Server 2008:

· TcpWindowSize – HKLM\System\CurrentControlSet\Services\Tcpip\Parameters

· NumTcbTablePartitions – HKLM\system\CurrentControlSet\Services\Tcpip\Parameters

· MaxHashTableSize – HKLM\system\CurrentControlSet\Services\Tcpip\Parameters

Network-Related Performance Counters

This section lists the counters that are relevant to managing network performance.

Resource Utilization

· IPv4, IPv6

  • Datagrams Received/sec
  • Datagrams Sent/sec

· TCPv4, TCPv6

  • Segments Received/sec
  • Segments Sent/sec
  • Segments Retransmitted/sec

· Network Interface(*), Network Adapter(*)

  • Bytes Received/sec
  • Bytes Sent/sec
  • Packets Received/sec
  • Packets Sent/sec
  • Output Queue Length

This counter is the length of the output packet queue (in packets). If this is longer than 2, delays occur. You should find the bottleneck and eliminate it if you can. Because NDIS queues the requests, this length should always be 0.

· Processor Information

  • % Processor Time
  • Interrupts/sec
  • DPCs Queued/sec

This counter is an average rate at which DPCs were added to the logical processor’s DPC queue. Each logical processor has its own DPC queue. This counter measures the rate at which DPCs are added to the queue, not the number of DPCs in the queue. It displays the difference between the values that were observed in the last two samples, divided by the duration of the sample interval.

Potential Network Problems

· Network Interface(*), Network Adapter(*)

  • Packets Received Discarded
  • Packets Received Errors
  • Packets Outbound Discarded
  • Packets Outbound Errors

· WFPv4, WFPv6

  • Packets Discarded/sec

· UDPv4, UDPv6

  • Datagrams Received Errors

· TCPv4, TCPv6

  • Connection Failures
  • Connections Reset

· Network QoS Policy

  • Packets dropped
  • Packets dropped/sec

· Per Processor Network Interface Card Activity

  • Low Resource Receive Indications/sec
  • Low Resource Received Packets/sec

· Microsoft Winsock BSP

  • Dropped Datagrams
  • Dropped Datagrams/sec
  • Rejected Connections
  • Rejected Connections/sec

Receive Side Coalescing (RSC) performance

· Network Adapter(*)

  • TCP Active RSC Connections
  • TCP RSC Average Packet Size
  • TCP RSC Coalesced Packets/sec
  • TCP RSC Exceptions/sec
Share Now:del.icio.usDiggFacebookLinkedInBlinkListDZoneGoogle BookmarksRedditStumbleUponTwitterRSS

Read the original blog entry...

More Stories By Hovhannes Avoyan

Hovhannes Avoyan is the CEO of PicsArt, Inc.,

@ThingsExpo Stories
Digital payments using wearable devices such as smart watches, fitness trackers, and payment wristbands are an increasing area of focus for industry participants, and consumer acceptance from early trials and deployments has encouraged some of the biggest names in technology and banking to continue their push to drive growth in this nascent market. Wearable payment systems may utilize near field communication (NFC), radio frequency identification (RFID), or quick response (QR) codes and barcodes...
Artificial Intelligence has the potential to massively disrupt IoT. In his session at 18th Cloud Expo, AJ Abdallat, CEO of Beyond AI, will discuss what the five main drivers are in Artificial Intelligence that could shape the future of the Internet of Things. AJ Abdallat is CEO of Beyond AI. He has over 20 years of management experience in the fields of artificial intelligence, sensors, instruments, devices and software for telecommunications, life sciences, environmental monitoring, process...
You think you know what’s in your data. But do you? Most organizations are now aware of the business intelligence represented by their data. Data science stands to take this to a level you never thought of – literally. The techniques of data science, when used with the capabilities of Big Data technologies, can make connections you had not yet imagined, helping you discover new insights and ask new questions of your data. In his session at @ThingsExpo, Sarbjit Sarkaria, data science team lead ...
The IETF draft standard for M2M certificates is a security solution specifically designed for the demanding needs of IoT/M2M applications. In his session at @ThingsExpo, Brian Romansky, VP of Strategic Technology at TrustPoint Innovation, will explain how M2M certificates can efficiently enable confidentiality, integrity, and authenticity on highly constrained devices.
There is an ever-growing explosion of new devices that are connected to the Internet using “cloud” solutions. This rapid growth is creating a massive new demand for efficient access to data. And it’s not just about connecting to that data anymore. This new demand is bringing new issues and challenges and it is important for companies to scale for the coming growth. And with that scaling comes the need for greater security, gathering and data analysis, storage, connectivity and, of course, the...
A critical component of any IoT project is the back-end systems that capture data from remote IoT devices and structure it in a way to answer useful questions. Traditional data warehouse and analytical systems are mature technologies that can be used to handle large data sets, but they are not well suited to many IoT-scale products and the need for real-time insights. At Fuze, we have developed a backend platform as part of our mobility-oriented cloud service that uses Big Data-based approache...
The IoTs will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, will demonstrate how to move beyond today's coding paradigm and share the must-have mindsets for removing complexity from the development proc...
trust and privacy in their ecosystem. Assurance and protection of device identity, secure data encryption and authentication are the key security challenges organizations are trying to address when integrating IoT devices. This holds true for IoT applications in a wide range of industries, for example, healthcare, consumer devices, and manufacturing. In his session at @ThingsExpo, Lancen LaChance, vice president of product management, IoT solutions at GlobalSign, will teach IoT developers how t...
Manufacturers are embracing the Industrial Internet the same way consumers are leveraging Fitbits – to improve overall health and wellness. Both can provide consistent measurement, visibility, and suggest performance improvements customized to help reach goals. Fitbit users can view real-time data and make adjustments to increase their activity. In his session at @ThingsExpo, Mark Bernardo Professional Services Leader, Americas, at GE Digital, will discuss how leveraging the Industrial Interne...
Increasing IoT connectivity is forcing enterprises to find elegant solutions to organize and visualize all incoming data from these connected devices with re-configurable dashboard widgets to effectively allow rapid decision-making for everything from immediate actions in tactical situations to strategic analysis and reporting. In his session at 18th Cloud Expo, Shikhir Singh, Senior Developer Relations Manager at Sencha, will discuss how to create HTML5 dashboards that interact with IoT devic...
SYS-CON Events announced today that Ericsson has been named “Gold Sponsor” of SYS-CON's @ThingsExpo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. Ericsson is a world leader in the rapidly changing environment of communications technology – providing equipment, software and services to enable transformation through mobility. Some 40 percent of global mobile traffic runs through networks we have supplied. More than 1 billion subscribers around the world re...
We’ve worked with dozens of early adopters across numerous industries and will debunk common misperceptions, which starts with understanding that many of the connected products we’ll use over the next 5 years are already products, they’re just not yet connected. With an IoT product, time-in-market provides much more essential feedback than ever before. Innovation comes from what you do with the data that the connected product provides in order to enhance the customer experience and optimize busi...
The increasing popularity of the Internet of Things necessitates that our physical and cognitive relationship with wearable technology will change rapidly in the near future. This advent means logging has become a thing of the past. Before, it was on us to track our own data, but now that data is automatically available. What does this mean for mHealth and the "connected" body? In her session at @ThingsExpo, Lisa Calkins, CEO and co-founder of Amadeus Consulting, will discuss the impact of wea...
In his session at @ThingsExpo, Chris Klein, CEO and Co-founder of Rachio, will discuss next generation communities that are using IoT to create more sustainable, intelligent communities. One example is Sterling Ranch, a 10,000 home development that – with the help of Siemens – will integrate IoT technology into the community to provide residents with energy and water savings as well as intelligent security. Everything from stop lights to sprinkler systems to building infrastructures will run ef...
Whether your IoT service is connecting cars, homes, appliances, wearable, cameras or other devices, one question hangs in the balance – how do you actually make money from this service? The ability to turn your IoT service into profit requires the ability to create a monetization strategy that is flexible, scalable and working for you in real-time. It must be a transparent, smoothly implemented strategy that all stakeholders – from customers to the board – will be able to understand and comprehe...
You deployed your app with the Bluemix PaaS and it's gaining some serious traction, so it's time to make some tweaks. Did you design your application in a way that it can scale in the cloud? Were you even thinking about the cloud when you built the app? If not, chances are your app is going to break. Check out this webcast to learn various techniques for designing applications that will scale successfully in Bluemix, for the confidence you need to take your apps to the next level and beyond.
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus inter...
So, you bought into the current machine learning craze and went on to collect millions/billions of records from this promising new data source. Now, what do you do with them? Too often, the abundance of data quickly turns into an abundance of problems. How do you extract that "magic essence" from your data without falling into the common pitfalls? In her session at @ThingsExpo, Natalia Ponomareva, Software Engineer at Google, will provide tips on how to be successful in large scale machine lear...
SYS-CON Events announced today that Fusion, a leading provider of cloud services, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Fusion, a leading provider of integrated cloud solutions to small, medium and large businesses, is the industry's single source for the cloud. Fusion's advanced, proprietary cloud service platform enables the integration of leading edge solutions in the cloud, including cloud...
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.