Under high load or blocked conditions, we have found that significant
queueing can occur when servicing .ASP Server.CreateObject requests.
This occurs when a process on which IIS is waiting is called more
frequently than the number of transaction/second the component can
satisfy. When blocking occurs, incoming requests are placed in a queue
and processed in First In-First Out. If the blocking only occurs for
several seconds the queue smooths out short fluctuations in load and all
incoming requests are serviced in a timely fashion and life is good.
However, when the spikes last for a long period of time (say 30 seconds)
the queue builds and will peak at its default 500 value. At queue
saturation, what happens next is determined by whether the exact queue
size is at RequestQueueMax (in this case 500) or something less. When a
request comes in and the number of queued requests exactly queue equals
RequestQueueMax IIS returns a Server Too Busy. This is an accurate
reflection of the current condition, because incoming requests cannot be
serviced in a timely fashion. If the number of queued requests is under
500 (say 498) the user with the 499th request will see an hourglass in
their browser as their request sits in the queue waiting for all the
other requests in the queue to be satisfied before their request is
processed. This is an acceptable condition if the queue clears very
rapidly (say 15 seconds.) It turns out on a site as busy as MS.COM the
wait is closer to 60 seconds and is probably intolerable for all but the
most patient users. After about 15 seconds or so, most users probably
get impatient and hit the stop button thinking the server is hung. At
this point they probably try again which worsens the condition and get
the same or different results based on which server picks up their next
request in the DNS Round-Robin.
Queueing Philosophy
Any effort spent processing requests which have aged more than the
amount of time the average user is willing to wait around for the page
is a waste of capacity (which is already scarce during high-queue
conditions). The client making the original request will not be around
to see the results of the Server's work anyway as they will have given
up and hit the Stop button. High Queue latency also has the side effect
of yielding poor performance for all non-.HTM request for the entire
time the queue is saturated as all .ASP requests share the same Queue.
Further exacerbating the condition is that processor utilization tends
to spike lengthening the duration of the poor response time - probably
due to managing the large queue.
Registry Settings to Focus On
There are 2 Registry setting for .ASP which can have a significant
impact on the performance and queueing of your site. They are
ProcessorThreadMax and RequestQueueMax. To get a mental picture of what
these do, think of Traffic on a major highway that gets congested during
rush hour(s). ProcessorThreadMax essentially increases the total number
of lanes and RequestQueueMax is like a flow control system which limits
the number of cars who can enter the highway at any one time.
These settings are found in the Registry at the following location:
Goal: get processor utilization above 50% (if possible) during peak load
To get your money's worth out of your machines, you should be seeing
better than 50% processor utilization during peak load or you may
actually have more processing capability than you need given other
bottlenecks in your system.
It is not possible to calculate what the appropriate number of threads
should be as live sites are too dynamic. You should gather statistics
using Performance Monitor (PerfMon) including at least:
Processor: %Processor Time for each processors
ActiveServerPages: Requests/Sec, Requests Rejected, Total Queue Length
HttpService: Connections/Second
Watch PerfMon counters on one of your live machines during peak load
time as it is running (use a 1 second Chart interval). Typically if you
have a busy site with a variety of work going on and some minimal
blocking, you will see the Total Queue Length counter go up and down. If
the Total Queue Length never goes up and you are running at low
processor utilization, this probably indicates you have a smooth running
site with more capacity (machines) than you need. Tuning in this
scenario will not help you much unless you want to increase the
efficiency of your machines in an effort to reduce the total number of
machines you are using.
If your queue is going up and down and your processors are running below
50% this is an indication that some of your requests are blocking and
you can probably benefit by increasing your threads. Start by doubling
the number of threads by changing ProcessorThreadMax from 10 (default)
to 20. Restart the machine (although you may be able to get away with
just starting and stopping the service, this is not practical on a busy
site such as MS.COM). You should expect to see some increase in
processor utilization and the queue tend to go up and down more quickly.
Increasing the number of threads improves response time for non-blocking
operations as available processing power can be put to use on satisfying
more requests concurrently improving overall response time.
After increasing the threads, If you see the queue climb and processor
utilization actually go down this is typically a bad sign and an
indication that you may have some serious blocking problems and that a
significant percentage of your requests are blocking. Another side
effect of increasing threads if putting more concurrent pressure on
blocking problems as more threads tend to be blocked on the same
external resource. If there are any bugs in any of the components, they
will tend to surface more quickly as you increase the number of threads.
If you have a single point of entry for your site that has serious
blocking problems (such as your home page) you need to do some redesign.
If your queue stays down and processor utilization increases continue
increasing Processor ThreadMax until you hit your target CPU utilization
(stay under 70% and keep ProcessorThreadMax below 100). You may find
that other things such as network I/O is throttling you, you have more
machines than you really need, or you do not have any blocking and high
CPU utilization cannot be achieved as IIS is extremely efficient for
.HTM and non-blocking .ASP.
Tuning RequestQueueMax
Goal: insure good response time while
minimizing Server Too Busy condition
Now that you have your threads dialed in, you can focus attention on
dialing in your RequestQueueMax. The goal here is simple: Determine your
target response time (say 10 seconds) and keep the Queue at a size which
will never queue more than that many seconds of work - this is your
upper queue limit. The ideal queue size would be below your response
time limit, and still above your typical peak load queue size.
A queue size which is too small will yield too many Server Too Busy
messages under load spikes, if it is too large the site looks hung and
the user will give up anyway as the site is not meeting response time
requirements. Watch the queue at peak time you should see a natural
pattern of it going up and down. Make a note of the peaks and put the
queue just above the peaks but under your Response time threshhold. The
point is to use the queue to handle short-term spikes, insure response
time, and throttle the system to avoid overload with sustained
unexpected spikes. You can calculate the response time by timing how
long it takes for the queue to return to 0 after it goes up to the max
value. On a smooth running non-blocking site, the queue will typically
stay near 0 most of the time as requests are coming in and getting
satisfied in a timely fashion.
If you have no data, A good starting setting seems to be a 1 to 1 ratio
of queue to total threads. Example: If you have ProcessorThreadMax set
at 25 and 4 processors (100 threads) then start your RequestQueueMax at
100 and tune from there.
How to Minimize Queueing and improve response time for your site
Use .HTM files whenever possible
Minimize use of Server.CreateObject in your .ASP's
whenever possible
Severely Minimize external dependencies - anything which
is not on the same machine as your WebServer.
Maximize network performance/reliability to external
dependencies
Load test and rate all custom objects before deploying
them on your site
Insure that every component will execute faster than the
rate at which it is called. If a component is called 20 times/second it
must complete each cycle in much less than 1/20 of a second or it will
block. A single blocking component can ruin the performance of all .ASP
on your site.
Monitor PerfMon stats for your site each week and keep a
close eye on it - a single blocking object can impact your whole site.
How to tell if your site is running smoothly On a smoothly running site, you should see very little queueing with all spikes staying well below your RequestQueueMax throughout the day as all components are able to satisfy the load placed on them. Processor utilization hangs around your target rate during peak load.
Credits
This paper is the result of research done by the Customer System Group
in tuning/managing www.Microsoft.Com and Home.Microsoft.Com.
This article is a reprint of chapter 15, by Nelson Howell , in a new book called 'Using Microsoft Internet Information Server 4' from Que Education & Training (ISBN 0789712636) due for publication in early March 1998. The chapter covers performance tuning of the Internet Information Server version 4.0. Including: What is performance, building web sites for speed, and the performance monitor. [Read This Article][Top]
This article is a reprint of chapter 16, by Nelson Howell , in a new book called 'Using Microsoft Internet Information Server 4' from Que Education & Training (ISBN 0789712636) due for publication in early March 1998. This chapter covers understanding and planning for server-side loading with the Internet Information Server version 4.0. Including determining load using performance monitor, using event viewer to discover errors, using TCP/IP troubleshooting utilities, configuring ODBC loads and understanding IIS logging. [Read This Article][Top]
This article by Jan Hein de Waal Malefijt and Bart van Langen describes how to creating a Active Server Page caching system in JavaScript and SQL Insight. [Read This Article][Top]
Srinivasa Sivakumar offers lengthy tips and sample code for improving ASP application performance. Keys to improving HTML page performance reach beyond the usual client hardware/bandwidth issues and into 10 detailed suggestions about how to handle images, frames, and redundant tags. Response-time problems are carved into individual discussions about ASP page performance, network bandwidth, and database issues. He offers twofold examples offering both the "slower" code and the much-preferred "faster" code. [Read This Article][Top]
Everyone seems to want the speed-to-market that Windows NT can provide when developing a Web site, but too many people have heard about NT's track record of stability and stability troubles. This paper attempts to prove NT can be stable and reliable, and that there are many good reasons to
consider NT when building your site. [Read This Article][Top]
This article covers three ways to cache your dynamic Web site written
in ASPs, including Microsoft Acceleration Server, Microsoft's ASP+, and Post Point Software's XCache, and offer the pros and cons of
using each product. [Read This Article][Top]
In this comparison of ASP, VB, and C++ database access for Internet applications, Drew Seale
examines whether components are all they are cracked up to be. The results are surprising! [Read This Article][Top]
Active Server Pages 3.0 offers two new methods to the Server object: Transfer and Execute . However, as Paul Litwin explains, there will be times when it's better to use good old Response.Redirect. [Read This Article][Top]
Learn how to overcome the limitations of WAST, Microsoft's Web Application Stress Tool, and retrieve dynamic variables from your ASP pages with the new Visual Studio.NET load-testing tool.
[Read This Article][Top]
Paging is basically querying a database and presenting a page full of the query's results. In ASP and SQL Server programming, there are three approaches to paging. This article compares the approaches and explains why the third is the most efficient when dealing with very large sets of data.
[Read This Article][Top]
