Microsoft® Windows® 2000
Server with Internet Information Services (IIS) 5.0 offers performance gains
and higher availability for your Web servers and sites. With tighter
integration between the operating system and IIS, you can now tune your servers
to perform much faster and more efficiently than in previous versions.
This document is intended for Web
server administrators charged with monitoring and tuning Web sites running on
Windows 2000 and IIS. Although there is some discussion of Web application
testing and tuning here, this document's primary audience does not include Web
application developers.
This document
discusses an approach to tuning the performance of your Web servers. It also
addresses why performance tuning is important and discusses hardware, software,
and testing issues involved in tuning your IIS 5.0 Web servers. Finally, it
includes a short discussion of tools you can use to monitor and test server
performance. While there is a discussion of some of the more prominent issues
in tuning Web applications, this document does not delve into this issue
exhaustively. For links and references to this and other topics, see the
Resources section of this document.
There are as many ways to tune a Web
server’s performance as there are Web sites on the Internet. Depending upon the
choices your company has made about its presence on the Web, you may be
responsible for tuning your Web servers to best serve static Web pages or
dynamically compiled application pages. Each type of site demands different hardware,
application, and Windows 2000 and IIS performance tuning options. Another
consideration is the amount of traffic that you may realistically expect your
Web site to handle, particularly during peak load periods. Load will affect Web
server performance, and varying business choices, such as the breadth of your
company’s advertising campaign, can determine the number of user requests your
Web sites will be required to handle. You should have a good idea of what those
loads will be and simulate them on your servers before you put them on line.
These are a few of the reasons why there is no silver bullet recommendation on
how to tune your Web server.
Performance tuning a Web server should
be viewed as an art as much as a science: trial and error can be an important
technique in determining what settings and hardware work best for your Web
site’s requirements. While it's crucial that you understand the technical
settings discussed in this document, it is equally important that you
understand the profile of your applications or Web sites and how they behave
under different conditions. Like a painter who sketches in charcoal to develop
a sense of how he wants to create a painting, you need to have a plan for
evaluating your Web server performance. The first step is to set up a
controlled environment in which to test your Web site, conduct performance
analysis of predicted loads, and then measure performance in that environment
before you expose your Web server to the Internet. Since the performance of the
server can vary greatly with the amount of browser traffic hitting your site
during different periods, be sure to monitor your test site under a number of
different loads to capture a true picture of activity on the server. During
this period, you can develop backup plans to help prevent your site from going
down due to any problems during or after deployment.
To improve server performance, examine
every part of the system for potential bottlenecks. Bottlenecks can be caused
by inadequate or improperly configured hardware or by software settings in
either IIS or Windows 2000. A good monitoring plan checks performance in all
areas.
Once you know how your server is
performing, you can make changes aimed at improving performance. Changes should
be made one at a time, with a tested rollback plan, or it becomes difficult to
assess the impact of individual changes.
After each change is made, continue to
monitor to see if the change has had the desired effect. If unwanted side
effects are observed, the rollback procedure can be performed, returning the
server to its previous state. Because changes to one resource can cause
bottlenecks to appear in other areas, it is important to check on the
performance of all resources after you make a change. Once you have assessed the
impact of a change, you can determine whether further changes are necessary.
There
are a number of good business reasons for you to tune your Web servers. First,
the improved performance that tuning provides will create a better user
experience with less waiting while the server responds to requests. Tuning will
help avoid usage bottlenecks, and can help extend the time between hardware
upgrades or the purchase of new servers for your Web farm. This allows you to
budget for when you really need to purchase new components such as RAM,
processors, disks, and network cards.
Along
with these business reasons, there are a number of technical reasons to tune
your Web server’s performance. Tuning allows you to best take advantage of the
hardware that you do have and to determine what upgrades you need to make now
or in the future. By tuning your Web server you can maximize throughput and
minimize response times of your Web applications, and determine which of these
applications are handling peak loads as required.
One of
the difficulties in tuning your Web server is knowing exactly what to tune. For
this reason it is vital that you monitor your Web servers before you make any
adjustments to settings, hardware, Web applications, or even upgrade to Windows
2000 and IIS 5.0. This point cannot be emphasized enough: gathering baseline
information about your servers will allow you to understand their behavior and
refine your Web server performance goals. You can use the Performance Monitor
and the Performance Counters built in to the operating system and IIS to
establish this baseline. Once you have gathered your baseline data, analyze it
to determine what the underlying reasons for performance problems may be before
making a change, whether it be adding RAM or adjusting internal IIS settings.
Once you've made a change, remember to monitor the servers again. Any change
you make may have unforeseen effects on other components in your system.
This
topic is broken into five sections: hardware tuning issues, Web-application
tuning issues, tools you can use to monitor and stress test your system,
security considerations, and settings internal to Windows 2000 and IIS 5.0 that
affect Web server performance.
Note that all of these issues are interrelated. From upgrading
hardware to modifying internal settings, tuning your Web server will require
you to carefully monitor how any changes affect the performance of your Web
server.
Problems
caused by memory shortages can often appear to be problems in other parts of
the system. You should monitor memory first to verify that your server has
enough, and then move on to other components. To run Windows 2000 and IIS 5.0,
the minimum amount of RAM a dedicated Web server needs is 128MB, but 256 MB to
1GB is often better. Additional memory is particularly beneficial to e-commerce
sites, sites with a lot of content, and sites that experience a high volume of
traffic. Since the IIS File Cache is set to use up to half of available memory
by default, the more memory you have, the larger the IIS File Cache can be.
Note: Windows 2000 Advanced Server can support up to 8GB of RAM
and Windows 2000 DataCenter can support up to 32GB, but the IIS File Cache
cannot take full advantage of this unless you partition the system.
Partitioning the system in this way may be useful in server consolidation
scenarios. With 32 processors, you can partition Windows 2000 DataCenter into
eight autonomous machines, each using 1.5 gigabytes of RAM for system cache.
Furthermore, if you add the /3GB switch in c:\boot.ini, then inetinfo.exe can
address up to 3GB of memory; otherwise, it's limited to 2GB of address space.
In addition, every instance of dllhost.exe can address up to 2GB, so if you had
a sufficiently large system, the cumulative memory used by all the IIS-related
processes could go beyond 4GB. As ISAPIs and ASPs execute at medium isolation
by default, they are in an instance of dllhost separate from inetinfo.exe. In
addition, high-isolation applications each have their own dllhost.
The
static file cache lives in the inetinfo process and can store up to 1.5GB of
content. The ASP caches live in each
process hosting asp.dll. The caches draw upon the total memory available to the
process hosting them.
To
determine if the current amount of memory on your server will be sufficient for
your needs, use the Performance tool (formerly known as PerfMon) that is built
in to Windows 2000. The System Monitor, which is part of the Performance tool,
graphically displays counter readings as they change over time.
Also,
keep an eye on your cache settings—adding memory alone won't necessarily solve
performance problems. You need to be aware of IIS cache settings and how they
affect your server's performance. If these settings are inappropriate for the
loads placed on your server, they, rather than a lack of memory, may cause
performance bottlenecks. For more information about these cache settings, see
the IIS Settings section and Appendix 1: Performance Settings of this document.
For a discussion about caching with ASP and IIS, see Appendix 3: ASP Caching.
Note: When using Performance
counters to monitor performance, you can see a description of any counter by
selecting that counter in the Add Counters dialog and clicking Explain.
Log the following counters to determine if there are performance
bottlenecks associated with memory:
·
Memory: Available Bytes. Try to reserve
at least ten percent of memory available for peak use. Keep in mind that IIS
5.0 uses up to 50 percent of available memory for its file cache by default.
·
Memory: Page Faults/sec, Memory: Pages Input/sec, Memory: Page Reads/sec, and Memory: Transition Faults/sec. If a process requests a page in
memory and the system cannot find it at the requested location, this
constitutes a page fault. If the page is elsewhere in memory, the fault is
called a soft page fault (measured by Transition Faults/sec). If the page must
be retrieved from disk, the fault is called a hard page fault. Most processors
can handle large numbers of soft faults without consequence. However, hard
faults can cause significant delays. Page Faults/sec is the overall rate at
which the processor handles faulted pages, including both hard and soft page
faults. Pages Input/sec is the total number of pages read from disk to resolve
hard page faults. Page Reads/sec is the number of times the disk was read to
resolve hard page faults. Pages Input/sec will be greater than or equal to Page
Reads/sec and can give you a good idea of your hard page fault rate. If these
numbers are low, your server should be responding to requests quickly. If they
are high, it may be because you've dedicated too much memory to the caches, not
leaving enough memory for the rest of the system. You may need to increase the
amount of RAM on your server, though lowering cache sizes can also be
effective.
·
Memory: Cache Bytes, Internet Information Services Global: File
Cache Hits %, Internet Information
Services Global: File Cache Flushes,
and Internet Information Services Global:
File Cache Hits. The first counter, Memory: Cache Bytes, reveals the size
of the File System Cache, which is set to use up to 50 percent of available
physical memory by default. Since IIS automatically trims the cache if it is
running out of memory, keep an eye on the direction in which this counter
trends. The second counter is the ratio of cache hits to total cache requests
and reflects how well the settings for the IIS File Cache are working. For a
site largely made up of static files, 80 percent or more cache hits is
considered a good number. Compare logs for the last two counters, IIS Global:
File Cache Flushes and IIS Global: File Cache Hits, to determine if you are
flushing objects out of your cache at an appropriate rate. If flushes are
occurring too quickly, objects may be flushed from cache more often than they
need to be. If flushes are occurring too slowly, memory may be wasted. See the ObjectCacheTTL, MemCacheSize, and MaxCachedFileSize
objects in Appendix 1: Performance
Settings.
·
Page File Bytes: Total. This counter
reflects the size of the paging file(s) on the system. The larger the paging
file, the more memory the system commits to it. Windows 2000 itself creates a
paging file on the system drive; you can create a paging file on each logical
disk, and you can change the sizes of the existing files. In fact, striping a
paging file across separate physical drives improves paging file performance
(use drives that do not contain your site’s content or log files). Remember
that the paging file on the system drive should be at least twice the size of
physical memory, so that the system can write the entire contents of RAM to
disk if a crash occurs.
·
Memory: Pool Paged Bytes, Memory: Pool Nonpaged Bytes, Process (inetinfo): Virtual Bytes, Process
(dllhost#n) : Virtual Bytes, Process (inetinfo): Working Set, and Process (dllhost#n): Working Set.. Memory: Pool Paged Bytes
and Memory: Pool Nonpaged Bytes monitor the pool space for all processes on the
server. The Virtual Bytes counters monitor the amount of virtual address space
reserved directly by IIS 5.0, either by the Inetinfo process (in which the core
of IIS runs) or by the Dllhost processes (in which isolated or pooled
applications run) instantiated on your server. The Working Set counters measure
the number of memory pages used by each process. Be sure that you monitor
counters for all instances of Dllhost on your server; otherwise, you will
not get an accurate reading of pool space used by IIS. The system’s memory
pools hold objects created and used by applications and the operating system.
The contents of the memory pools are accessible only in privileged mode. That
is, only the kernel of the operating system can directly use the memory pools;
user processes cannot. On servers running IIS 5.0, threads that service
connections are stored in the nonpaged pool along with other objects used by
the service, such as file handles and sockets.
Besides
adding more RAM, try the following techniques to enhance memory performance:
improve data organization, try disk mirroring or striping, replace CGI
applications with ISAPI or ASP applications, enlarge paging files, change the
frequency of the IIS File Cache Scavenger, disable unnecessary features or
services, and change the balance of the File System Cache to the IIS 5.0
Working Set. The last of these techniques is detailed later in this document.
For a
detailed discussion list of Windows 2000 and IIS 5.0 settings that will affect
these counter numbers, see Appendix 1:
Performance Settings.
With
users demanding quick response time from Web sites and the increasing amount of
dynamically generated content on these sites, a premium is placed on fast and
efficient processor usage. Bottlenecks occur when one or more processes consume
practically all of the processor time. This forces threads that are ready to be
executed to wait in a queue for processor time. Adding other hardware, whether
memory, disks or network connections, to try to overcome a processor bottleneck
will not be effective and will frequently only make matters worse.
IIS 5.0
on Windows 2000 Server scales effectively across two to four processors, and
with a little additional tuning scales well on eight processors (see
Appendix 4). Consider the business needs of your Web sites if you're
thinking of adding more processors. For example, if you host primarily static
content on your server, a two-processor computer is likely to be sufficient. If
you host dynamically generated content, a four-processor setup may solve your
problems. However, if the workload on your site is sufficiently CPU-intensive,
no single computer will be able to keep up with requests. If this is the case
for your site, you should scale it out across multiple servers. If you already
run your site on multiple servers, consider adding more.
You
should be aware, however, that the biggest performance gains with Windows 2000
and IIS 5.0 result from resolving inadequate memory issues. Before you make any
decisions about changing the number of processors on your Web servers, rule out
memory problems and then monitor the following Performance Counters.
·
System: Processor Queue Length. This counter displays the
number of threads waiting to be executed in the queue that is shared by all
processors on the system. If this counter has a sustained value of two or more
threads, you have a processor bottleneck on your hands.
·
Processor: %Processor Time.
Processor
bottlenecks are characterized by situations in which Processor: % Processor
Time numbers are high while the network adapter card and disk I/O remain well
below capacity. On a multi-processor computer, it's a good idea to examine the
Processor: % Processor Time counter to pick up any imbalance.
·
Thread (Inetinfo/thread-number): Context Switches/sec, Thread (Dllhost/thread-number#process-instance): Context Switches/sec, and System: Context Switches/sec. If you
decide to increase the size of any of the thread pools, you should monitor the
three counters listed here. Increasing the number of threads may increase the
number of context switches to the point where performance decreases instead of
increases. Ten context switches or more per request is quite high; if these numbers
appear, consider reducing thread pool size. Balancing threads against overall
performance as measured by connections and requests can be difficult. Any time
you tune threads, follow-up with overall performance monitoring to see if
performance increases or decreases. To determine if you should adjust the
thread count, compare the number of threads and the processor time for each
thread in the process to the total processor time. If the threads are
constantly busy, but are not fully using the processor time, performance may
benefit from creating more threads. However, if all the threads are busy and
the processors are close to their maximum capacity, you are better off
distributing the load across more servers rather than increasing the number of
threads. See also the AspThreadGateEnabled and AspProcessorThreadMax
metabase properties in Appendix 1: Performance Settings in this
document.
·
Processor: Interrupts/sec and Processor: % DPC Time. Use these
counters to determine how much time the processor is spending on interrupts and
deferred procedure calls (DPCs). These two factors can be another source of
load on the processor. Client requests can be a major source of each. Some new
network adapter cards include interrupt moderation, which accumulates
interrupts in a buffer when the level of interrupts becomes too high.
Scaling Out Across Multiple Computers
If
processor problems persist, try scaling your site out across multiple computers
using Network Load Balancing (NLB) or a hardware load balancer such as
Microsoft’s deployment and management tool, Application Center. While
setting up a Web farm using one of these methods adds a layer of complexity and
introduces a number of other issues, this action is likely to solve a number of
your performance issues if your site is large enough. For more information
about NLB, see the Network Load
Balancing Technical Overview148369.
Essentially,
the network is the line through which clients send requests to your server. The
time it takes for those requests and responses to travel to and from your
server is one of the largest limiting factors in user-perceived server
performance. This request-response cycle time is called latency, and latency is
almost exclusively out of your control as a Web server administrator. For
example, there is little you can do about a slow router on the Internet, or the
physical distance between a client and your server.
On a
site consisting primarily of static content, network bandwidth is the most
likely source of a performance bottleneck. Even a fairly modest server can
completely saturate a T3 connection (45Mbps) or a 100Mbps Fast Ethernet
connection. You can mitigate some of these issues by tuning the connection you
have to the network and maximizing your effective bandwidth as best you can.
The
simplest way to measure effective bandwidth is to determine the rate at which
your server sends and receives data. There are a number of Performance counters
that measure data transmission in many components of your server. These include
counters on the Web, FTP, and STMP services, the TCP object, the IP object, and
the Network Interface object. Each of these reflects different Open System
Interconnectivity (OSI) layers. For a detailed list of these counters and their
analysis, see the Internet Information
Services 5.0 Resource Guide, released with the Windows 2000 Server Resource Kit. In particular, see the Network I/O section of the Monitoring and Tuning Your Server
chapter. To start, however, use the following counters:
·
Network Interface: Bytes Total/sec. To
determine if your network connection is creating a bottleneck, compare the
Network Interface: Bytes Total/sec counter to the total bandwidth of your
network adapter card. To allow headroom for spikes in traffic, you should
usually be using no more than 50 percent of capacity. If this number is very
close to the capacity of the connection, and processor and memory use are
moderate, then the connection may well be a problem.
·
Web Service: Maximum Connections and Web Service: Total Connection Attempts.
If you are running other services on the computer that also use the network
connection, you should monitor the Web Service: Maximum Connections and Web
Service: Total Connection Attempts counters to see if your Web server can use
as much of the connection as it needs. Remember to compare these numbers to
memory and processor usage figures so that you can be sure that the connection
is the problem, not one of the other components.
See
the Tuning and Troubleshooting Suggestions section of this document for
suggestions on how to reduce bandwidth usage by reducing file sizes and by
enabling proxy and client caching.
Since IIS 5.0 writes logs to disk, there is regular disk
activity even with 100 percent client cache hits. Generally speaking, if there
is high disk read or write activity other than logging, this means that other
areas of your system need to be tuned. For example, hard page faults cause
large amounts of disk activity, but they are indicative of insufficient RAM.
Accessing
memory is faster than disk seeks by a factor of roughly 1 million (nanoseconds
versus milliseconds); clearly, searching the hard disk to fill requests will
degrade performance. The type of site you host can have a significant impact on
the frequency of disk seeks. If your site has a very large file set that is
accessed randomly, if the files on your site tend to be very large, or if you
have a very small amount of RAM, then IIS is unable to maintain copies of the
files in RAM for faster access.
Typically,
you will use the Physical Disk counters to watch for spikes in the number of
disk reads when your server is busy. If you have enough RAM, most connections
will result in cache hits unless you have a database stored on the same server,
and clients are making dissimilar queries. This situation precludes caching. Be
aware that logging can also cause disk bottlenecks. If there are no obvious
disk-intensive issues on your server, but you see lots of disk activity anyway,
you should check the amount of RAM on your server immediately to make sure you
have enough memory.
To
determine the frequency of disk access, log the following counters:
·
Processor: % Processor Time, Network Interface Connection: Bytes
Total/sec, and PhysicalDisk: % Disk
Time. If all three of these counters have high values, then the hard disk
is not causing a bottleneck for your site. However, if the % Disk Time is high
and the processor and network connection are not saturated, then the hard disk
may be creating a bottleneck. If the Physical Disk performance counters are not
enabled on your server, open a command line and use the diskperf -yd command.
See
also the Tuning and Troubleshooting Suggestions section.
Balancing
performance with users' concerns about the security of your Web applications is
one of the most important issues you will face, particularly if you run an
e-commerce Web site. Since secure Web communication requires more resources
than non-secure Web communications, it is important that you know when to use
various security techniques, such as the SSL protocol or IP address checking,
and when not to use them. For example, your home page or a search results page
most likely doesn’t need to be run through SSL. However, when a user goes to a
checkout or purchase page, you will want to make sure that page is secure.
If you do use SSL, be aware that establishing the
initial connection is five times as expensive as reconnecting using security
information in the SSL session cache. The default timeout for the SSL session
cache has been changed from two minutes in Windows NT 4.0 to five minutes in
Windows 2000. Once this data is flushed, the client and server must establish a
completely new connection. If you plan on supporting long SSL sessions,
consider lengthening this timeout with the ServerCacheTime
registry setting (described in Appendix 1). If you expect thousands of
users to connect to your site using SSL, a safer approach is to estimate how
long you expect SSL sessions to last, then set the ServerCacheTime parameter to
slightly longer than your estimate. Do not set the timeout much longer than
this or else your server may leave stale data in the cache. Also, make sure that HTTP Keep-Alives are enabled (on by default). SSL
sessions do not expire when used in conjunction with HTTP Keep-Alives unless
the browser explicitly closes the connection.
In
addition to all security techniques having performance costs, Windows 2000 and
IIS 5.0 security services are integrated into a number of operating system
services. This means that you can't monitor security features separately from
other aspects of those services. Instead, the most common way to measure
security overhead is to run tests comparing server performance with and without
a security feature. The tests should be run with fixed workloads and a fixed
server configuration, so that the security feature is the only variable. During
the
