asp tutorials, asp.net tutorials, sample code, and Microsoft news from 15Seconds
Data Access  |   Troubleshooting  |   Security  |   Performance  |   ADSI  |   Upload  |   Email  |   Control Building  |   Component Building  |   Forms  |   XML  |   Web Services  |   ASP.NET  |   .NET Features  |   .NET 2.0  |   App Development  |   App Architecture  |   IIS  |   Wireless
 
Pioneering Active Server
 Power Search








Active News
15 Seconds Weekly Newsletter
• Complete Coverage
• Site Updates
• Upcoming Features

More Free Newsletters
Reference
News
Articles
Archive
Writers
Code Samples
Components
Tools
FAQ
Feedback
Books
Links
DL Archives
Community
Messageboard
List Servers
Mailing List
WebHosts
Consultants
Tech Jobs
15 Seconds
Home
Site Map
Press
Legal
Privacy Policy
internet.commerce














internet.com
IT
Developer
Internet News
Small Business
Personal Technology
International

Search internet.com
Advertise
Corporate Info
Newsletters
Tech Jobs
E-mail Offers

HardwareCentral
Compare products, prices, and stores at Hardware Central!

Efficiently Uploading Large Files via Streaming
By Dina Fleet Berry
Rating: 4.0 out of 5
Rate this article


  • email this article to a colleague
  • suggest an article

  • download source code
  • Introduction

    When uploading a file to a web server, the upload process generally requires the incoming file to be stored in memory until the upload is complete. If an uploaded file is larger than the available memory, memory usage in general and performance in particular will suffer.

    Typically, the uploaded file is either held in memory or the length of the posted file is checked and refused the file if it is too large. The streaming solution presented in this article allows you to grab an uploaded file of any size without swamping your web server by grabbing it in chunks and writing each chunk to the database. Only the specific chunk size of memory is needed at a time. This allows your memory to not get swamped on one upload but instead handle many uploads of large file sizes because the memory used by any upload at a time is only the chunk size. In this example, the chunk size is 1024 bytes.

    Instead of having the web server or operating system manage the memory of large file uploads, this article will allow you to manage large file uploads by streaming the upload right into the database.

    Associated Article

    This article is provided in conjunction with another article about caching large files (How to Create a Web Server Farm Cache). The caching article's portion of this web site is in the Cache.aspx file. This article's code for the upload portion of that web site and the associated file is called UploadBlob.aspx.

    Requirements

    You need an IIS web server and a SQL server. The SQL tables, and stored procedures are provided in the Upload.sql file. You will also need files to upload. The /images directory of the associated web site has three images that you can use. A small view of each of the three images is below.

    The SQL statement for creating the table is:

    CREATE TABLE [dbo].[UploadBytesOfBlob](
        [FileGuid] [uniqueidentifier] NOT NULL,
        [Name] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
        [Content] [varbinary](max) NULL,
        [ContentType] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
        [Size] [numeric](18, 0) NULL,
        [NumberOfChunks] [numeric](18, 0) NULL,
        [FirstModifiedDate] [datetime] NULL,
        [LastModifiedDate] [datetime] NULL,
    CONSTRAINT [PK_UploadBytesOfBlob] PRIMARY KEY CLUSTERED
    (
        [FileGuid] ASC
    )WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
    ) ON [PRIMARY]

    The primary key for the table in the sql database is called FileGuid and is of type UniqueIdentifier. This is a Guid in terms of ASP.NET.

    How the System Works

    The web site file UploadBlob.aspx uses the Upload control that is provided as part of ASP.NET. The uploaded file is read a chunk at a time. As each chunk is read, the chunk is then appended to the end of the database row's content column. In this way, each chunk is handled at a time. Along with appending the data for each chunk, several columns are modified: the size, count of chunks and the last modified date.

    The Sample Web Site

    The sample web site has three main files: default.aspx, cache.aspx, and uploadblob.aspx. Download, compile and run the web site. Make sure to create the SQL table and stored procedures using the Upload.sql file. Put a break point at the top of the Page_Load method of the uploadblob.aspx.cs file. This is the area of code this article focuses on.

    The first time you load the file, nothing will appear except a link to the upload page on the default.aspx. As you upload images, this page will show a GridView listing those images.

    Click on the Upload File link, and you will go to the upload page. The uploadblob.aspx file is short with nothing specific to this solution:

        <form id="form1" runat="server" >
        <div>
            Large File to Upload<br />
            <asp:FileUpload ID="FileUpload1" runat="server" /></div>
            <asp:Button ID="Button1" runat="server" Text="Upload" />
        </form>

    Upload the ascent.jpg file from the /images directory. Your breakpoint should now be active in Visual Studio at the top of the Page_Load method of uploadblob.aspx.cs.

    The if statement is there so that a previously uploaded file can be modified/overwritten with a new file. This is part of the caching article code.

    In this first iteration of the code, the if statement doesn't apply so the else statement is hit.

    The code grabs the file as a Stream object, along with the file name and the content or mime type. The mime type is important for sending the file back to the browser so that the appropriate plug-in or application can render the file.

            // DFB: Upload goes into stream
            Stream myStream = FileUpload1.PostedFile.InputStream;
            _name = FileUpload1.FileName;
            _contentType = FileUpload1.PostedFile.ContentType;

    Next a Byte array is created of 1024 bytes. The size of 1024 can be considered artibrary in this example. You will want to determine the best size for your chunks given your memory on the web server and the sql server as well as general size of files uploaded and the frequency with which they are uploaded.

            // DFB: Create buffer for stream
            Byte[] myBuffer;
            myBuffer = new byte[1024];

    The code then checks the state for the FileGuid value. If the value is not found, a new guid is created and a row is inserted into the SQL Table:

            // DFB: If new blob, create guid and row
            if (ViewState["FileGuid"] == null)
            {
                // DFB: Create new guid/file
                _guid = System.Guid.NewGuid();

                // DFB: Create new row for this data
                CreateRow();
            }
            else
            {
                // DFB: This is the postback of an update (not a new file)
                _guid = new Guid(ViewState["FileGuid"].ToString());
            }

    The SQL for the CreateRow function is:

    INSERT INTO UploadBytesOfBlob (FileGuid, Name, ContentType, LastModifiedDate, FirstModifiedDate)
    VALUES (@FileGuid, @Name, @ContentType, GetDate(), GetDate())

    Next, the row needs to be cleared. You must clear because you can not append to a null column. The clear is accomplished by adding zero data.

            // DFB: Can't append to null column so use clear first
            Clear();

    The cooresponding SQL for the clear funtion is

    --DFB nulls the column
    UPDATE    UploadBytesOfBlob
    SET       Content = CONVERT(varbinary(max),0)
    WHERE     FileGuid = @FileGuid

    --DFB writes
    UPDATE    UploadBytesOfBlob
    SET       LastModifiedDate = GetDate(),
              Content.WRITE(NULL,0,0),
              Size = 0,
              NumberOfChunks=1
    WHERE     FileGuid = @FileGuid

    The Content.Write(Null,0,0) clause uses the .Write function that is only available to varchar(max), nvarchar(max), or varbinary(max) column types. The .Write function specifies what part of the column data to modify. The three parameters for the .Write function are the content of the column, the offset into the column, and the length of the existing content to replace. By using the above parameter values of (Null,0,0), the column content is truncated to the beginning of the column.

    Now the Stream of the Upload control is fed right into the database one chunk at a time.

            // DFB: Read upload and stream into database in chunks
            while(myStream.Read(myBuffer, 0, myBuffer.Length)>0)
            {
                //DFB: Append stream chunk to end of database row/column
                AppendChunk(myBuffer);
            }

    The associated AppendChunk sql stored procedure is

    UPDATE    UploadBytesOfBlob
    SET       Content.WRITE (@Data, NULL, NULL)
    WHERE     FileGuid = @FileGuid

    UPDATE    UploadBytesOfBlob
    SET       Size = LEN(Content),
              LastModifiedDate = GetDate(),
              NumberOfChunks = NumberOfChunks + 1
    WHERE     FileGuid = @FileGuid

    The new parameters for the .Write function indicate that the data should be appended to the current column's content. The second update statement modifies the Size, LastModifiedDate, and the NumberOfChunks columns to reflect the updated Content value.

    The stream is closed and the page redirects to the default.aspx page.

            myStream.Close();

            Response.Redirect("~/default.aspx");

    The default.aspx page now shows the ascent.jpg information in the GridView.

    You can click on the FileGuid link to see the content loaded from the database via the Cache.aspx file. The file is streamed down from the database using the same chunk method to read the content and write to a file. The web server then returns the file to the browser. The Upload link on the row allows you to change the content of the row. The content is completely cleared every time the file is uploaded via this chunking method.

    Summary

    Uploading large files can have a tendency to swamp your web server. This solution allows for many uploads of large files because each upload is only handled a chunk at a time. As each chunk is read in, it is immediately put in the database. The database keeps track of the chunk count and size of the content.

  • Rate This Article
    Not HelpfulMost Helpful
    1 2 3 4 5
    Other Articles
    Jul 21, 2005 - N-Tier Web Applications using ASP.NET 2.0 and SQL Server 2005 - Part 1
    While the .NET Framework made building ASP.NET applications easier then it had ever been in the past, .NET 2.0 builds on that foundation in order to take things to the next level. This article shows you to how to construct an N-Tier ASP.NET 2.0 Web application by leveraging the new features of ASP.NET 2.0 and SQL Server 2005.
    [Read This Article]  [Top]
    Apr 28, 2005 - New Files and Folders in ASP.NET 2.0
    With the release of ASP.NET 2.0, Microsoft has greatly increased the power of ASP.NET by introducing a suite of new features and functionalities. As part of this release, ASP.NET 2.0 also comes with a host of new special files and folders that are meant to be used to implement a specific functionality. This article examines these new files and folders in detail and provides examples that demonstrate how to utilize them to create ASP.NET 2.0 applications.
    [Read This Article]  [Top]
    Mar 10, 2005 - The DataSet Grows Up in ADO.NET 2.0 - Part 2, Cont'd
    Alex Homer continues his detailed look at the major changes to the DataSet class. In this part, he looks at two features that allow developers to work with data in a more structured and efficient way when using the DataSet with a SQL Server 2005 database server.
    [Read This Article]  [Top]
    Mar 9, 2005 - The DataSet Grows Up in ADO.NET 2.0 - Part 2
    Alex Homer continues his detailed look at the major changes to the DataSet class. In this part, he looks at two features that allow developers to work with data in a more structured and efficient way when using the DataSet with a SQL Server 2005 database server.
    [Read This Article]  [Top]
    Mar 3, 2005 - The DataSet Grows Up in ADO.NET 2.0 - Part 1, Cont'd
    In this article, Alex Homer looks at the changes between the version 1.x and version 2.0 DataSet and their associated classes, showing you how you can take advantage of the new features to improve your applications' capabilities and performance.
    [Read This Article]  [Top]
    Mar 2, 2005 - The DataSet Grows Up in ADO.NET 2.0 - Part 1
    In this article, Alex Homer looks at the changes between the version 1.x and version 2.0 DataSet and their associated classes, showing you how you can take advantage of the new features to improve your applications' capabilities and performance.
    [Read This Article]  [Top]
    Feb 16, 2005 - Writing a Custom Membership Provider for the Login Control in ASP.NET 2.0
    In ASP.NET 2.0 and Visual Studio 2005, you can quickly program custom authentication pages with the provided Membership Login controls. In this article, Dina Fleet Berry examines the steps involved in using the Login control with a custom SQL Server membership database.
    [Read This Article]  [Top]
    Dec 29, 2004 - ClickOnce Deployment in .NET Framework 2.0
    In this article, Thiru Thangarathinam examines .NET 2.0's new ClickOnce deployment technology that is designed to ease deployment of Windows forms applications. This new technology not only provides an easy application installation mechanism, it also eases deployment of upgrades to existing applications.
    [Read This Article]  [Top]
    Dec 15, 2004 - A Sneak Peek at ASP.NET 2.0's Administrative Tools
    With ASP.NET 2.0, Microsoft has made great strides in increasing developer productivity and has made implementing previously complex solutions relatively easy. Where this version of ASP.NET really shines, however, is in its new administrative tools that allow developers to spend less time managing the configuration of the servers and software and more time developing great code.
    [Read This Article]  [Top]
    Nov 17, 2004 - The ASP.NET 2.0 TreeView Control
    Thiru Thangarathinam introduces ASP.NET 2.0's new TreeView control which provides a seamless way to consume and display information from hierarchical data sources. The article discusses this new control in depth and explains how to use this feature rich control in your ASP.NET applications.
    [Read This Article]  [Top]
    Mailing List
    Want to receive email when the next article is published? Just Click Here to sign up.

    Support the Active Server Industry



    JupiterOnlineMedia

    internet.comearthweb.comDevx.commediabistro.comGraphics.com

    Search:

    Jupitermedia Corporation has two divisions: Jupiterimages and JupiterOnlineMedia

    Jupitermedia Corporate Info


    Legal Notices, Licensing, Reprints, & Permissions, Privacy Policy.

    Advertise | Newsletters | Tech Jobs | Shopping | E-mail Offers

    Solutions
    Whitepapers and eBooks
    Microsoft Article: Will Hyper-V Make VMware This Decade's Netscape?
    Microsoft Article: 7.0, Microsoft's Lucky Version?
    Microsoft Article: Hyper-V--The Killer Feature in Windows Server 2008
    Avaya Article: How to Feed Data into the Avaya Event Processor
    Microsoft Article: Install What You Need with Windows Server 2008
    HP eBook: Putting the Green into IT
    Whitepaper: HP Integrated Citrix XenServer for HP ProLiant Servers
    Intel Go Parallel Portal: Interview with C++ Guru Herb Sutter, Part 1
    Intel Go Parallel Portal: Interview with C++ Guru Herb Sutter, Part 2--The Future of Concurrency
    Avaya Article: Setting Up a SIP A/S Development Environment
    IBM Article: How Cool Is Your Data Center?
    Microsoft Article: Managing Virtual Machines with Microsoft System Center
    HP eBook: Storage Networking , Part 1
    Microsoft Article: Solving Data Center Complexity with Microsoft System Center Configuration Manager 2007
    MORE WHITEPAPERS, EBOOKS, AND ARTICLES
    Webcasts
    Intel Video: Are Multi-core Processors Here to Stay?
    On-Demand Webcast: Five Virtualization Trends to Watch
    HP Video: Page Cost Calculator
    Intel Video: APIs for Parallel Programming
    HP Webcast: Storage Is Changing Fast - Be Ready or Be Left Behind
    Microsoft Silverlight Video: Creating Fading Controls with Expression Design and Expression Blend 2
    MORE WEBCASTS, PODCASTS, AND VIDEOS
    Downloads and eKits
    Sun Download: Solaris 8 Migration Assistant
    Sybase Download: SQL Anywhere Developer Edition
    Red Gate Download: SQL Backup Pro and free DBA Best Practices eBook
    Red Gate Download: SQL Compare Pro 6
    Iron Speed Designer Application Generator
    MORE DOWNLOADS, EKITS, AND FREE TRIALS
    Tutorials and Demos
    How-to-Article: Preparing for Hyper-Threading Technology and Dual Core Technology
    eTouch PDF: Conquering the Tyranny of E-Mail and Word Processors
    IBM Article: Collaborating in the High-Performance Workplace
    HP Demo: StorageWorks EVA4400
    Intel Featured Algorhythm: Intel Threading Building Blocks--The Pipeline Class
    Microsoft How-to Article: Get Going with Silverlight and Windows Live
    MORE TUTORIALS, DEMOS AND STEP-BY-STEP GUIDES