How To Think About Programming: A Data-Centric Point of View

A Data-Centric Point of View

I'm going to start by trying to convince you of the data-centric point of view, how it is useful and why it should be your primary point of view when building computer systems.

Focus on the data. This could be

1. The source code which a compiler looks at
2. A list of strings to be sorted by a library function
3. A collection of invoices in an accounting software
4. A video travelling over the internet
5. A configuration file for a web server sitting on a filesystem

You get the idea - I mean data in the most general sense, as they exist in IT systems - in software, in hardware, through the network cables, all of that. I could go even further and talk about the data outside IT of systems, but I won't do that today.

Now all operations on data can be put into two types:

1. Transportation
2. Transformation

Transportation refers to any change of location. This could be copying bytes from one buffer in memory to another, or from memory to a register in the processor. It could be the transfer of a video file across a network. It could be from a database, through an application to an output device.

Transformation is harder to define, but easy enough to explain with examples. Think of sorting an array of numbers, or of tokenising a string. Think about adding TCP headers to an HTTP packet, and IP headers on top of that. Think about adding up the elements of a list of numbers.

At a low level, transformation is always expressible in terms of arithmetic operations. How do you find the sum of an array of numbers? Well, you maintain a running sum in one memory location, and an array index in another. Each iteration increments the index and updates the running sum.

Aside: Levels of abstraction

An operation without side effects is essentially a no-op. In order to know whether or not there are side effects, one has to restrict to a certain "scope". Here's an illustrative example, consider the following Python code:


    if a < b:
        pass

If your scope is low enough, there is plenty of transformation as well as transportation happening:

1. The value of a is copied from memory into some register on the chip
2. As is the value of b
3. A subtraction is performed
4. The sign of the result is observed
5. An instruction pointer is moving throughout this process...

And so on...

But if you move to a higher scope, this operation does nothing, and it may as well not have happened!

Your questions have different answers depending on the "level" at which you look at them. To decide which level your should be considering, you have to decide what you want to be thinking about. The example above is valid if your focus is on what happens on the chip. If you don't care about that, the code is equivalent to a no-op.

Here's an example at a much higher level. Suppose you're trying to insert a row of data into a relational database, and that this particular row violates a primary-key condition. You send the data across the network, through a bunch of routers and firewalls, through network interfaces and finally into the DB engine, which rejects the transaction and returns the error code.

Lots of stuff happened! But from the point of view of the data in your RDB - nothing happened, because nothing changed. The initial state is the same as the terminal state. You may as well not have sent that data at all because it made no difference.

Ok, enough about levels, you get it! From whatever point of view you want to reason, you can identify how your data moves and how it changes.

Don't think about your programs

Let me being with a term: Architecture. As a beginning programmer, I was fascinated by this. Architecture. It sounded so mysterious, so exciting, and so out of reach. How did people reason at that level? I was smart too, what was I doing wrong?

Here's my attempt at explaining how to do this: think about your data. Think about, and visualise, what it looks like, where it comes from, where it goes, how it changes. How it splits up and recombines. If your program reads a set of numbers, puts them into a binary tree and serialises it for transport, visualise these operations. If you accept input from a web form and need to validate it before storing it in a database with high availability and replication, think about how your data needs to move in order to achieve this.

Don't think about the programs which make this happen. Think about what is happening. Think about the data.

Once you have a picture of the data flow (transportation & transformation!), then you can think about how the computation has to happen.

Ok, now think about your programs

All transportation and transformation is enabled by computation. Again, let me explain this using examples at different levels. In each of the following examples, first ask yourself where the data is, how it is changing, and where it is going. Then ask yourself where the computation is happening.



Example 1


printf takes a format string and a list of data. It combines these two to produce another string, which is then sent to standard output.

• Think about this as a "service" being offered. Much like a valet service at a restaurant or laundry service at a hotel; you submit a request and something gets done.

Example 2

From within my program I invoke a function called foo(). This function accesses a cacheing service to get the name of the latest version of a file on my HDFS. It then asks HDFS to read that file, after which it can extract the last two lines and parses out the URL www.service.com and the integer 691. foo() then calls another function bar(). bar() packages the integer 691 inside a JSON object, which it sends via HTTP to www.service.com using the networking stack on that machine.

• This about this as a service calling a bunch of other services.

• Function calls are service requests! A function call takes parameters, and may return a result or have a side-effect. Or both.

• API calls are service requests!

Putting it Together

At each level of abstraction, and don't mix levels unless you want to end up confused,

• Think about your data - their creation, modification, transportation and eventual destruction.

• Then think about the services (or "operations") acting on this data. These could be function calls within a program, or calls to an external library, or even across computers over a network.

• Find out which of these services are already available for reuse

• Build the remaining services i.e. maybe write some code. To do this, level-down and repeat this process.

• Then design the framework which contains all your logic i.e. all your service requests.

(This would be a good time to say that if you're designing a system, you should require that every operation performed on your data have an associated authorisation... this could even default to "allow everything", but in your mental model it should be there).