20 hours. That's the amount of work we estimated each one of us would have to do to write the code, setup the database and move the servers for what would become TransferBigFiles.com. With that estimate in hand, we decided we'd work two 10-hour days: Saturday and Sunday, 8am - 6pm. Here is how it went down...
If you want to know more about the beginnings of this project, take a look at these blog entries:
Day 1 - Saturday
By 9am, we had made good progress. The SQL database was created, the graphics were done (logo & the #s, 1, 2, 3, 4) and we were well under way with the implementation of the Sprocs for the database when we realized the first hole in our project plan. We had forgotten to estimate time for a Data Layer. In most of our application designs, we create a simple data layer on top of Microsoft's Data Application Block. Our data layer typically provides simple objects which make life a lot easier when dealing with the database. The omission of the data layer in our timeline was not a big deal, as it would be done in about 45 minutes and we easily had more than 45 minutes of buffer in our timeline.
The next 4 hours we spent on TransferBigFiles.com's home page. We already knew what the page should look like from our design the previous day. Just implementing the UI part of the page (just HTML & Javascript) took longer than I had expected. With Dan and I pair-programming the page, it took us about 2 hours for UI and another 2 hours to implement the code-behind for the page. Everything looked great and our tests were going smooth as silk, except for the fact that we were still unaware of a major problem we were about to face.
By 3pm, while playing around with the work-in-progress, Dan tried to upload a 200MB file through the service. Suddenly, nothing seemed to work. Memory usage shot through the roof, processor was going crazy, the app did a full crash - all this and we were just uploading 1 file! How was this thing going to handle thousands of users? With more testing we realized that files larger than about 25MB were going to be a problem. This was a huge set-back. 25MB was just too small of a file and renaming the service “TransferLittleFiles.com” was not an option.
After more digging we realized that the problem was with ASP.NET's built-in file upload mechanism. It turns out that ASP.NET allocates memory for the ENTIRE post (which is how files are uploaded by default). This was causing the system to allocate large amounts of memory. To make matters worse, ASP.NET has a feature that monitors the process' memory usage. If the memory usage goes above a certain percent of system memory, it automatically shuts down the process to ensure against memory leaks. This prevented us from being able to successfully upload large files.
Google to the rescue - Googling the problem we found that the most practical solution was to use a commercial ASP.NET upload component. The rest of the day, I spent researching components while Dan started implementation of the file retrieval page for TransferBigFiles.com.
Day 2 - Sunday
At this point, I was starting to have my doubts about whether or not we would make our self-imposed deadline of 6pm. We had run into some unanticipated problems which were not yet resolved and we were still working on the development machine - nothing had been tested on the production servers. By 11am, we had made a decision on an upload component. We went with Media Chase's FileUploader.NET, which is a $249, .NET component to do exactly what we needed. Their component allows for large file uploads without any noticeable affects on memory and processor usage. This was a huge relief and it bought us an unexpected feature. The Media Chase component offers a nice upload progress and status feature which is much nicer than IE's built-in file-upload indicator.
We were back in business! By lunch-time, we were confident that we were making good time. We even took a 1-hour lunch at a nice fancy restaurant. We came back, finished things up and started to load the application on to the production servers. The deployment to the production servers went relatively smooth. We were deceived, once again. Our 2nd biggest challenge still lied ahead.
Going back to our design for TransferBigFiles.com, we had planned on using UNC paths to access the 1TB+ NAS Server. We figured this would be the most flexible solution that would allow us to easily add more NAS units as we need more storage space. To make sure load-balancing would be done, we created a simple web-service which returns the appropriate UNC path to the web server. Ideally, the web service would first look to see which server is least busy and provide the path to that server. For now, the web-service would simply return a single UNC to the only NAS we have allocated for TransferBigFiles.com.
The problem we now faced is that by default, ASP.NET applications can't access resources on a secondary machine. This meant our entire design of using UNCs to access our NAS storage was flawed. We had to figure out a way to fix this, so again, Google came to the rescue. As it turns out, the ASPNET process and the user that IIS runs under don't have any network privileges. The solution was relatively simple (run the IIS process as a different user that has proper privileges on the NAS), but this little problem set us back by at least 2 hours! By 4pm, we had it all working, so we started testing.
4:40pm, with a little more than an hour to go on our self-imposed deadline, we loaded up Dan's SUV with the servers in the back and headed to the Limelight data center to install the servers at their final location. Our biggest problem at the data center was getting the firewall setup. I had forgotten how to setup NAT on our firewall and couldn't get any traffic to reach the main web server for TransferBigFiles. Finally, at 5:58pm, the firewall issue was resolved and it was all working. It was done! 2 minutes ahead of schedule ;-)
Here we are at the Limelight Datacenter setting up the servers:
We have video taped about 20 minutes worth of film for our weekend project, but unfortunately, I don't have the time right now to edit the video and make it available. Plus, Microsoft's Movie Maker has been crashing on me lately (just opening the MPG files!).
Summary
This was a great project! I'm actually quite surprised at how much fun we had working through the weekend. It was quite satisfying and I hope you've enjoyed reading along.
The fact that we were able to imlement TransferBigFiles.com in a weekend shows .NET's power and ease of development. I really don't think we could have done everything we've done without the use of Visual Studio.NET and SQL Server 2000. We also used SourceGear's Vault as our source-code control product. Although I wish Vault had better integration with VS.NET (it took us nearly 30 minutes to add our solution to Vault on Saturday morning - a task that should have taken less than 1 minute). But Vault is great and it still does the job. And of course, to track the project and perform time checks, we used our very own, OnTime 2005 product. OnTime provided a great project management and task tracking tool for our needs.