What modern web frameworks do for you

Motivation

If you are like I was, a former CS major whose programming experience consisted of writing command-line programs using C++, the world of web applications might seem mysterious and foreign.

You know how computers work i.e. you know processor architecture, you know the basics of operating systems, compilers, networking protocols… but this web stuff is something else. And it’s hard to get started – for one because there’s all this terminology, but even if you learn the terminology it feels like there’s all this magic going on.

This blog post is for people who were in the position I was in 6 months ago. It is not a tutorial – it is meant to be read before your tutorials so that things make sense.

Web Application – Basic Flow

First let’s break down the basic structure of a web application.

1. An application will expose multiple endpoints, which usually correspond to different parts of the application but don’t necessarily need to. An endpoint is represented by a URL, like www.myapp.com/register or www.myapp.com/logout.
2. Clients make requests to these endpoints. These use the HTTP protocol, which I won’t get into other than to say that a request is made up of:
1. Headers, some of which are used during transit and delivery, but you can add your own
2. The desired endpoint
3. An HTTP verb, like GET or POST
4. GET requests will often have a query string attached, like                                              /getInfo?name=Rohit&age=36
5. POST requests will usually have a body, which could be XML, JSON or something called “form-data”
3. When the application starts up, it registers its endpoints with the web server (a separate process running on that machine). Registration means providing callbacks for these endpoints.
4. When a request comes in, the web server matches the URL against the endpoints it has registered, to see if there is a listener available. If not, it returns an error message (following what the HTTP protocol defines, hopefully). Otherwise it passes it on to the registered handler.
5. The handler does what it needs to, and then sends an HTML response back to the web server
6. The web server sends this response back to the client

Ok, simple enough.

Web Frameworks

So what do web frameworks do for you? Keeping in mind the steps above, the framework should provide:

1. A simple endpoint & handler registration mechanism
2. Automatic parsing of the request, putting the information into some simple data structures
3. Simple ways for handlers to redirect a request (possible modified) to other endpoints

Another useful feature is an authentication framework, so you can mark certain endpoints as "protected" i.e. requiring a logged-in user. Of course to do this you need the concept of a user, so let's not get into that.

But an even more useful feature is layout templating. Think of your final page as being data placed in a layout container. The layout defines the colors, the text areas, navigation controls, where the scroll bars are, which buttons do what etc. Some of these don't change depending on the data being presented, and some do. The template will define the data-independent portions and indicate where the data is meant to go. It might be as simple as

<div id="userName" class="userNameDisp">

    {{userNameVar}}

</div>

The framework will read this HTML template, figure out that userNameVar is a variable whose value is required, and get that variable's value from the interpolation context. You can imagine this working like this:

getPage (templateName, dataContext):

    response = loadTemplateAsString (templateName)

    for every {{varName}} in the response string

        look up varName in the dataContext

        replace {{varName}} by this value in the response string

    return response

The two web frameworks which I know of which work this way are Django, written in Python, and Ruby on Rails, which is written in Ruby.

A different strategy

Nowadays everyone loves JavaScript, supposedly because browser support is so much better than it was before (I don't know how it used to be, this is just what I hear). In fact browsers are so good at running JavaScript now that applications are being written so that a large part just sits in the browser! 

One of these frameworks is called AngularJS, and I've actually written a small application using it. But I don't fully know how it works yet. Here's what I do know/ believe at this point:

1. Request to the web application loads a whole bunch of JavaScript code. This contains all the endpoint definitions and associated handlers.
2. As the user interacts with the application, requests will be triggered to different endpoints. These will, as far as possible, be trapped by the client-side application and NOT be translated into fresh requests to the server (since the application logic has already been loaded into the browser)
3. Of course certain requests do need to go to the server, like requests for data. The server will now just send back JSON or XML.
4. Layout templates will also be sent on request, though a framework might also provide a mechanism for pre-requesting and caching them locally. One would hope that a layout template is not loaded more than once though.

Now the server sends only three types of responses: the initial application code, layout templates, and raw data. Less network traffic, but more work for the browser.

I hope this helped. If you want to learn more about these JavaScript frameworks, you can Google 

1. AngularJS for the client side (and yes there are others)
2. Express for the server (again, there are others) 
3. and for solutions which combine both the client and server - Meteor and Sails! 

App Architecture and the Basics of Scale

Motivation
Recently a friend asked me about building applications. He is a graphics designer, and doesn't want to learn how to write code. He just wants to get a "big-picture" view of some of the common themes that arise when designing applications today. This blog post is aimed at people like him - interested but non-technical.

I'm going to start off by saying that what I'm about to describe does NOT describe all the different types of software being written today. In fact I know developers who never touch these issues. But my feeling is that this will be accurate for the vast majority of developers.

What Kind of Applications are We Talking About?
Here are some examples:

1. A blogging platform
2. An order and procurement system
3. A recruitment platform
4. A corporate HR system
5. A corporate finance system
6. A (software) bug tracking system
7. An email and chat platform

These are all very different, but let's see what's similar for most or all of them.

1. There are UI elements through which users can enter data. Very often these take the shape of forms.
2. There are UI elements which show data to users. This could be:
1. the data which they, or other users entered into the system,
2. data from external systems (machinery, sensors, the Internet),
3. the results of aggregation and computation, generally shown in a report. Think of "monthly sales totals", or your bank statement.
4. Often people want to see entire collections of similar data (your MakeMyTrip search results); these are displayed in a list.
3. There are processes which run in the background which actually do something with this data. These processes may only involve computers (generating your bank statement), or sometimes other people (processing your loan request). For something like email, one process would be the delivery of mail, which might involve looking up email addresses from nicknames; another process would be receiving mail, and running it through a spam filter.
4. All this data has to be stored somewhere, since it is generally not just sent from user to user. Even in the case of a chat system, users generally want a history of their chats stored for later. So there needs to be a database.

More and more applications today are being developed on the browser - whether Internet-facing or inside an organization, the browser is the portal of choice. We can book hotels, apply to university, check football scores... all through a single portal.

(I am completely ignoring stand-alone applications like Photoshop, Excel, Mathematica, StarCraft... I'm not understating their importance, but they're not relevant right now).

Another medium which is gaining in popularity is the mobile app. They don't run in browsers, but many of them provide functionality which would be natural on a browser. Many of them (LinkedIn, Facebook, Twitter) actually have browser counterparts.

How They Work

The following flow is widely applicable:

1. A request arrives at some endpoint. For web applications, an endpoint is just a URL, like http://www.my-awesome-app.com/login 
2. The web server decides whether or not the application can handle this request. For this to happen, the application needs to have told the web server what it can handle. Anything not on that list is a fail.
3. If the request is valid, and the endpoint exists, the web server forwards it to the appropriate handler in the application.
4. The handler takes the request and does what it needs to. This usually involves talking to the database. The list of tasks which needs to be performed fall into a workflow. Workflows will have portions which are straightfoward - "after checking that their ZIP code is valid, check their area code" - and portions which are conditional - "if their account balance is too low then deny their ATM request. Otherwise give them their money".
5. If the request has arrived "bearing data", it gets written to the database. Before that, there will usually be
1. validation e.g. the user entered their birth date. People from the future are not allowed.
2. clean-up or sanitization e.g. the user entered their name all in lowercase, so the application capitalizes the first letter
6. If the request is for data from the database, the application will ask the database for the data. When it comes back, maybe it gets re-formatted, or cleaned-up, before being sent back to the user
7. Many types of requests need to be authorized, so that not just anybody can get into the system and do just anything. Sometimes this is done using usernames and passwords. Sometimes it is outsourced to another service (sign in with Facebook!).

In summary (and since I'm too lazy to draw a picture):

    User --> Web Server --> Application --> Database --> Application --> Web Server --> User

Scaling Them Up

Now suppose you have more users - more customers, more employees, whatever. It might be obvious that the system will slow down, but how exactly does this happen?

Let's consider an analogy from the real world (which is actually more than an analogy, it really works just like this). Think of a fast-food restaurant, say one which makes sandwiches. They have 4 cashiers where customers place their orders (their requests) and the cashier forwards their order to the kitchen (the application, which starts the workflow). The workflow might look like this:

1. Get some bread
2. Put mustard on the bread
3. Put meat and cheese on the bread
4. Put the bread in the oven for 3 minutes
5. Put it on a plate with some chips and signal that its ready

Then the cashier would pick it up and give it to the user. We also assume that the restaurant hasn't figured out how to let a cashier serve more than one customer at a time - which means that until the sandwich has been made, that cashier's tied up.

Now imagine that the restaurant has a lot of customers one day. What happens?

Well, the cashiers work pretty fast, so they can handle the load. And the cooks are pretty fast too. But:

1. The oven can only handle one sandwich at a time, so there's more waiting there

Okay, so the customers wait. But if this happens repeatedly, the restaurant owners will want to do something about it. They have a few options, all of which will have an impact, but none of which will "solve" the problem permanently.

1. Buy a bigger oven
2. Buy more ovens
3. Hire more cooks

(Also, until the kitchen problem is solved, there's no point in hiring more cashiers).

Here's the analogy: The oven is the database server. A bigger oven is a more powerful server - more memory, faster CPU. More ovens means more database servers. The cooks are the application servers and the cashiers are the web servers.

The cashiers are all the same, and their job is easy. The cooks are all similar, but maybe some specialize in Reubens and some in pastrami on rye. 

The database is the real work-engine, that's where the time is spent. The performance of the application will eventually just be the performance of the database.

Terminology

Let me close with some terminology.

• Scaling up: This means making a component more powerful, generally by throwing CPU and RAM at it
• Scaling out: Add more components of the same type at that layer (like hiring more cashiers). Scaling out is cheaper (2 Marutis are cheaper than a Mercedes), but the application needs to have been designed to work that way.
• High Availability: A deployment that is fault-resistant due to intentional redundancy. Like having extra cooks waiting in case a working cook gets sick or injured.
• Disaster Recovery. How does the application react when something really bad happens e.g. the data center gets hit by a hurricane, or a virus attack. DR strategies generally involve setting up an entirely separate (and geographically separated) center, ready and waiting for such scenarios. Of course if they're ready and waiting, they cost money even while there's no disaster in-progress!
• Replication: Keep copies of the data, to protect against drive failure. Disk drives are mechanical objects, and they are the components which fail first and fail often. Data must always be stored in multiple locations

One part I left out is the entire portion of flow between the client and the web servers i.e. the network part. I left it out because one generally has no control over the network, especially if one is using the Internet. Your control starts at the web front-end, and that's where you start optimizing.