Do You Understand What Your Web Framework is Doing?

It's a new year, and I'm going to start off 2008 wrong with a code-free post. Sorry about that. This stems from realizing how little many developers (judging from postings to a variety of mailing lists) seem to understand about what their web frameworks do for them when it comes to generating code in other languages (particularly JavaScript and SQL). It's Rails-flavored, but not Rails-specific.

Here's a quick quiz. I'm assuming that you, the reader, are a web developer familiar with JavaScript, SQL, and some reasonably modern web app framework:

1. Does JavaScript validation on a web form guarantee that when the form is submitted the server will receive valid data?
2. Should foreign key columns each get their own index?
3. How is JSON parsed into data structures in memory in a browser?
4. Are multi-table joins inefficient?
5. Can a web page make requests to a host other than the host from which the page itself was requested?

We'll come back to that. I am going to start by talking about a web browser (client) interacting via HTTP(S) with a web server. There are three pieces here, not two. The HTTP protocol matters since it is easy to work with and understand and there are lots of tools for working with it. There are some important differences between the web client/server environment and a more traditional client/server system:

• Connections are not persistent, and consist of only a single request and response. (Note that HTTP keep-alive does not change this; what persists is the TCP connection, and does not affect the application layer.)
• Interaction can only be initiated by the client, not the server. This is a result of the previous difference.
• The server cannot assume anything about the data received from the client.

Most people developing web sites/applications think in terms of the server software they are developing. Much as first-time GUI developers often find it baffling, the inversion of control involved in modern web programming confuses many developers. The server has full control when responding to a request but, once it has generated that response, control reverts to the client. For one thing, that means that data on the client does not get to the server unless the client decides to send it. It also means that data from the server does not get to the client unless the client decides to request it. One needs to work from the point of view of the user in front of the browser.

A common question on the Rails mailing list is how to use RJS to retrieve some value from the client. While the desire isn't ridiculous, and it can be done in a roundabout way with a certain amount of jumping through hoops, phrasing the question that way shows a lack of understanding of where the RJS-generated code will be executing. (The hoop jumping involves having the RJS generate an AJAX request back to the server to submit the value back to some URL on the server.)

There was a recent thread on the Rails list complaining about the functionality in Rails (largely RJS and various helpers) that attempts to hide the complexity of interactions between client-side and server-side code and largely results in maintainability problems in the code and misunderstandings for the developer. I don't agree with everything in either the original post or the various responses, but it highlights a problem Assaf identified months ago.

Assaf is concerned with bad (inefficient and/or incorrect) SQL being generated because the developer doesn't understand what the framework is doing underneath. I'm concerned about bad (incorrect, unmaintainable, and/or hard to debug) JavaScript being generated. Rails makes it easy to get results without understanding what it is doing for you, which is great for prototyping and dangerous for production.

It is important to understand what code is being executed where, when, and how. When developing a rich user experience in a web browser, one must understand the DOM, the event model, the browser security model, the JavaScript language, the single-threaded nature of JavaScript execution in the browser, XMLHttpRequest, etc. just as one must understand database indexing, column types, SQL, table/row locking, etc. to develop a production-quality database-backed web application.

Let's go back to that quiz. You shouldn't have had to think too hard about any of these, and you should feel certain about your answers. And those answers should be:

1. Does JavaScript validation on a web form guarantee that when the form is submitted the server will receive valid data? Nope. The server receives data over HTTP, and that HTTP connection could come from any program, not just a web browser. Furthermore, JavaScript can be turned off in most browsers. On top of that, most browsers make it possible to mess with the web page live and/or the data being submitted. Client-side validation is a user interface nicety, but provides no guarantees about the data the server sees.
2. Should foreign key columns each get their own index? Sometimes. It depends very much on what queries will involve them. A join table (i.e. one with more than one foreign key that represents a many-to-many relationship between tables) usually benefits from an index on all foreign keys, sometimes even multiple indices of the same columns in different orders. Tables only queried by columns other than their foreign keys generally don't benefit from indexing those foreign keys, even if the table is usually joined against the tables to which those foreign keys refer.
3. How is JSON parsed into data structures in memory in a browser? Since JSON is JavaScript, it is executed with eval() to be parsed with JSON.parse() into memory in the browser.
4. Are multi-table joins inefficient? This depends on the number of tables, available indices, and the database engine. Joining 18 tables in MySQL can make the query optimizer hang for hours (that's the query optimizer, not executing the query), regardless of available indices on the tables involved. A query on tables lacking indices on appropriate columns will require full table scans in any database, which is always slow (unless the unindexed tables have very few rows). It is always worth asking your database engine to explain and profile the queries you'll be running. Incidentally, database logs from running your unit/functional/integration/whatever tests are a great place to start.
5. Can a web page make requests to a host other than the host from which the page itself was requested? Yes, but not with XMLHttpRequest. At the simplest level, an img tag makes a request from any arbitrary URL, though the response is not available to JavaScript. To interact with a different host with almost the same flexibility as an XMLHttpRequest, one uses a script tag. See this blog post for a discussion.

How did you do? If you didn't get them all, you need to keep learning. If you got them all right, don't get too cocky; you may still not know everything you need to know to avoid the pitfalls of a code-generating framework. I keep learning about things I thought I knew thoroughly, and I wouldn't have it any other way. Enjoy!

The Magic in the Box

Disclaimer: This is a rant. If you are not interested in reading the rant, please don't, but please also do not comment unless you have read it all the way through.

I'm a pretty easy-going guy, and I like to be helpful on mailing lists (and in person), but I expect a basic level of knowledge from people, particularly about the tools they are using. For anyone writing software, the computer is their tool. (So is a text editor or IDE and compiler or interpreter, but that's not where I'm headed with this.)

A strong formal education in computer science provides an understanding of data representation, logic, memory hierarchies, algorithmic analysis (and common algorithms), concurrency, common/basic data structures, stack and heap layout, compiler design, and probably a dozen other things I'm omitting. (Ideally, program design and development is a part of that education, but it should not be at the expense of anything I mentioned.) It's challenging, but self-taught programmers can learn the same things on their own. And should. As a professional of any sort, one has a responsibility to understand both the unavoidable tools (in particular, memory and CPU details) and the available tools (go look up the heap data structure right now if you don't know what it is).

One can argue that newbies should be able to jump right into programming, but at the same time there really is such a thing as a stupid question. Nothing says, "I don't understand how computers work and shouldn't be paid to write software until I do," like a question that shows a complete unawareness of the accuracy limitations of floating point representation. There are certainly other stupid questions, but that one came up recently.

I could have ended this rant on the previous paragraph, but I was afraid it would come across as unreasonable and mean-spirited, and my goal is to express the reasoning behind the distress I feel when I read such questions. First, though, I want to address the newbie issue in greater depth. If one is teaching oneself how to program, that is to be encouraged, but it involves some work beyond following along in a book or online tutorial. For the floating point example above, the autodidact (great word, look it up) should be expected to follow roughly these steps:

1. discover an anomaly in floating point arithmetic
2. in irb, run something like "0.1.class" to discover that 0.1 is a Float
3. ri Float to discover that "Float objects represent real numbers using the native architecture's double-prescision floating point representation"
4. google for "floating point representation"
5. Read the very first link in the results, which has a specific section on the issue
6. achieve enlightenment

The point is that teaching oneself involves more than just reading a book and playing around, sometimes in involves taking time to investigate the parts that don't immediately make sense. (Note that those steps are the shortest, simplest path from confusion to understanding; it might take someone more work than that, but the fact remains that there is a simple and reasonable path to understanding and that it isn't particularly hard to find.) If one does not know how to learn on one's own, self-teaching is the wrong choice and it is time to go get that formal education.

A bit more on why these questions I'm calling stupid distress me so. Imagine yourself as a master carpenter who is training an aspiring apprentice. The apprentice has assembled some Ikea furniture and decided he likes carpentry, so he's going to give it a whirl. One day, he brings in a stool with a leg that needs replacing. He says, "I've been trying to remove the leg, but the screw is just stuck or something, and I think something is wrong with it." You look, and ask him to show you how he's tried to go about it. He picks up a screwdriver and desperately tries to fit its flat head into the cross of the phillips screw. You tell him to stop, take out a phillips screwdriver, and proceed to unscrew things easily. He gawks and asks, "There are different kinds of screws that need to be treated differently?"

Imagine your horror. How could he not know something so basic? Aspiring to be a carpenter requires understanding the tools a carpenter uses. Likewise, aspiring to be a programmer requires understanding the tools a programmer uses. Enjoy!

Randomizing an Array and Other FAQs

This is a quickie, but I'm so incredibly tired of seeing it on the ruby-talk list (and occasionally Rails list) that I'm posting it anyway. It's also (kinda, sorta) a followup to the previous post.

If you need to randomize an array, use the following:

random_array = my_array.sort_by{rand}

Now quit asking about it. Here are some other little FAQs I'm tired of seeing on the lists:

• Symbols can be thought of as immutable, internalized strings if you like, but strings and symbols are, indeed, different. If you really need to know more you can search the web, since others have covered it in more depth.
• Hashes are unordered. Deal with it. There is no good reason for a hash to maintain the order of its keys; the purpose of a hash is amortized constant time lookup. If you need to iterate through the hash keys in some order, use #sort or #sort_by before iterating.
• Rails has a bunch of stuff added to Ruby. This includes things like HashWithIndifferentAccess (which allows the params hash to be accessed by string or symbol) and Symbol#to_proc (which lets you use &:foo in place of {|x|x.foo}). Ruby Facets makes some of that available to normal Ruby. Ruby 2.0 will also have some of it built in.
• Rails isn't the only thing Ruby is good for. For that matter, it is neither the only Ruby web framework, nor is ActiveRecord the only Ruby ORM.

Phew. Are we done? Anyone else want to post their pet peeve FAQ? Leave a comment. Oh, yeah, almost forgot.

Enjoy!

Updated! See Randomizing an Array Revisited.

Kernel#qualified_const_get

The question of how to get a class by name comes up with some regularity on the ruby-talk and rails mailing lists. The first response is usually to use Object::const_get. The response to that is that it doesn't handle classes within namespaces, e.g. Foo::Bar. I might argue that Object::const_get("Foo::Bar") should do the right thing and retrieve the value of Bar from the Foo class/module, but the fact of the matter is that it does not. Having an itch to scratch, I wrote Kernel#qualified_const_get:

module Kernel
  def qualified_const_get(str)
    path = str.to_s.split('::')
    from_root = path[0].empty?
    if from_root
      from_root = []
      path = path[1..-1]
    else
      start_ns = ((Class === self)||(Module === self)) ? self : self.class
      from_root = start_ns.to_s.split('::')
    end
    until from_root.empty?
      begin
        return (from_root+path).inject(Object) { |ns,name| ns.const_get(name) }
      rescue NameError
        from_root.delete_at(-1)
      end
    end
    path.inject(Object) { |ns,name| ns.const_get(name) }
  end
end

One of the advantages of this usage is that it handles partially qualified constant names. The following does the right thing in both cases:

require 'qualified_const_get'

module Foo
  module Bar
    class Baz
      def initialize
        puts 'You found me!'
      end
    end
  end
  def self.find_it
    klass = qualified_const_get("Bar::Baz")
    klass.new
  end
  module Quux
    def self.find_it
      klass = qualified_const_get("Bar::Baz")
      klass.new
    end
  end
end

Foo::find_it
Foo::Quux::find_it

Enjoy!