OS X Minutiae: DTerm, Environment Variables, and Launch Services

I've been using DTerm for years, but it recently stopped working for me. I guessed it had to do with switching my shell to zsh (from tcsh, which I'd been using since the early 1990s), and a little searching found a blog post from 2008 pointing to exactly the issues I was having and the solution: setting the SHELL environment variable to /bin/bash on launch. The post recommended launching from the commandline with env SHELL=/bin/bash TERM=dterm open /Applications/DTerm.app and, in fact, that works. My problems are solved, right?

Of course, I want DTerm to launch on login, so I either have to write a script (AppleScript or shell) to launch it and put that in my login items or dig a little further. That led me to an answer on StackExchange about setting environment variables for launched apps on OS X. I added the following block to the Info.plist:

 <key>LSEnvironment</key>
 <dict>
  <key>DISPLAY</key>
  <string>:0</string>
  <key>SHELL</key>
  <string>/bin/bash</string>
  <key>TERM</key>
  <string>dterm</string>
 </dict>

That had one effect I didn't expect, but not the one I did. I had to go into the Accessibility pane under the Privacy tab of the Security & Privacy System Preferences and explicitly authorize DTerm. DTerm offers to set it automatically at a button press and administrator authentication, but it doesn't succeed; doing it manually from System Preferences does the right thing. I was expecting adding this LSEnvironment block to make DTerm launched from a double-click work the way it does when launched with the commandline above, but it did not. More digging was required.

The "LS" part of LSEnvironment refers to Launch Services, which keeps the system aware of available apps and what sorts of documents they can handle. In looking for how to make sure Launch Services was picking up the environment variables I was setting, or at least why it wasn't, I found a (slightly outdated) hint on how to work with the Launch Services database from the commandline. The lsregister command (along with the entire LaunchServices.framework) has moved since then, and your best bet on finding it is to use locate lsregister from the commandline. I found it in /System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister but it may move in future versions of OS X. I used -dump with it to figure out what it thought was set, but what ultimately fixed things was the following:

/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister -u ~/Applications/DTerm.app -apps local,network,system,user

That unregistered the app from Launch Services, and when it reregistered the LSEnvironment block was picked up. DTerm now launches from my login items and happily bahaves the way it used to.

Enjoy!

P.S. Sorry for the long absence. I can't promise to get back to posting with any regularity, but at least you know I'm not dead.

Mobile OS Musings

I haven't had much time for hobby programming since becoming a father and I can't really talk about my day job, so all I really have to talk about are things I've read about. I fear that I'm indulging in punditry, but I'm aiming for something closer to thinking out loud.

What I'm thinking about, in this case, is mobile app platforms, and I'm starting from the three mobile app platforms I find interesting: Apple's iOS, Google's Android, and HP/Palm's WebOS. I should probably explain why I care about WebOS, and why I'm not covering either the existing Blackberry OS or the QNX-based Blackberry Playbook from RIM.

Justifying Myself

The Playbook looks interesting, but its UI is very similar to WebOS. Furthermore, its app platform is Adobe AIR. As a VM-based runtime, AIR is similar to Android's Dalvik VM. It uses ActionScript, a dialect of ECMAscript/JavaScript, as its development language, similar to WebOS's JavaScript-based development. In short, it isn't different enough from the other platforms to rate discussion (at least for my purposes) and, worse, it isn't shipping yet.

WebOS has shipped on a few devices, but hasn't gained much popularity. It's pretty much a rounding error when it comes to market share. Its fortunes may be changing, however, since HP seems to be betting the company on WebOS. Most importantly, it has some unique features, particularly Synergy and Touchstone (scroll down to "Better Together" on the Touchstone link).

The existing Blackberry OS is a dead end as an app platform. RIM's purchase of QNX makes it clear that they are looking to replace their aging OS, and rightly so. Every Blackberry device currently on the market is already obsolete, and RIM cannot attract a development community to it's excruciating ecosystem of dozens of OS versions on dozens of hardware configurations. There are a few people out there who love their Blackberries, but the vast majority of Blackberry users have them through their jobs. Even if their share of that market lasts for the next decade, it isn't the consumer market in which the other platforms are competing and in which I am interested.

With that out of the way, I want to cover the similarities between iOS, Android, and WebOS development. They seem more different than the same, but there is one part that is and will probably remain nearly identical across platforms: 3D (game) development. Android has the NDK and WebOS has the PDK so that C and C++ code targeting OpenGL ES can be included in an app. C and C++ (and OpenGL ES) are part of iOS development anyway, since all iOS apps are written as C/C++/Objective-C/Objective-C++ code. This means that the core of most 3D games, particularly those from big development houses, are pretty portable across all the platforms. Porting has more to do with managing bundled assets, interfacing with user input, and extensive testing than any rewriting of the game engine. Other similarities include hardware functionality (e.g. GPS, accelerometer/gyroscope, camera, touch input, etc.) and common APIs (e.g. TCP/IP, audio/video, data storage, etc.).

The differences come down to the fundamental approach to development. Developing an iOS app is more similar to than different from developing a desktop application with a traditional event loop and GUI widgets. Android apps run in a VM (it isn't technically a JVM, but the differences are negligible), and are based on a radically different paradigm of activities, views, intents, and services. WebOS apps are basically web apps with JavaScript bindings to native APIs — they use HTML/CSS for presentation and rely on the normal web browser event loop.

Could They?

The really interesting question concerning these platforms is how easy it would be, both technically and legally, for one OS to support a competing app platform. This assumes that the hosting platform's app store(s) would accept app submissions for the competing platform rather than attempting to run apps purchased on a competing platform. Since Android is open source and its apps run in a VM, it should be both technically and legally easy to support on either WebOS or iOS. Various APIs would have to be implemented to call the native platform's equivalent APIs, and some (e.g. background processing) would have to change behavior or simply be removed, but most Android apps could be hosted reasonably on either WebOS or iOS. There may be some legal challenges, however, based on Oracle's lawsuit against Google concerning copyright infringement in Android, or various patent claims from Microsoft and others.

Likewise, WebOS apps run in a web browser and could be supported on either Android or iOS with the same sort of API implementation. The biggest technical hurdle is probably Synergy, but WebOS apps can make use of the available information, ignorant of its origins, rather than using anything specific to Synergy. Again, most WebOS apps could be hosted reasonably on either iOS or Android. The bigger issue for hosting WebOS apps is that a good chunk of the JavaScript provided by the WebOS SDK is protected by copyright, and would have to be reimplemented to avoid infringement. That isn't insurmountable, but it means that it isn't nearly as easy to create a hosting environment for WebOS apps as for Android apps. Hosting AIR apps for the Blackberry Playbook have similar considerations, though Adobe will happily license the AIR environment, possibly at a negligible or even zero cost.

iOS apps are a whole different story. An iOS app is a single native executable (plus a bundle of assets). Most of the libraries on which it relies are neither provided as open source nor in any way similar to any publicly available source code. The GNUstep project has free implementations of some of those libraries as well as the Objective-C runtime, but it would require a huge effort to reimplement those libraries. It is, of course, possible with the application of sufficient time and money, but it is much more difficult and, therefore, more expensive than trying to host either of the other two platforms' apps.

Would They?

At least as interesting as how easy hosting a competing platform would be is whether there would be a competitive advantage to doing so. Apple has shown over and over again that it has no interest in apps that run easily on multiple platforms. The rationale is that supporting a platform they don't control means they can't improve the platform as rapidly as one they do control. More importantly, iOS has no shortage of apps thus supporting apps from a competing platform would do more to help their competitor than iOS.

There would certainly be value to the Android platform to support iOS app, not only because are there more iOS apps but because they have a reputation for greater polish and beauty than Android apps. Since Android is open source, it's possible that the community of Android developers would get together, port GNUstep, reimplement the libraries an iOS app would need, and make it freely available. The work involved in that effort is, as previously discussed, pretty significant, making such development somewhat less likely to succeed. As for Android hosting WebOS apps, it is similar to the relationship between iOS and Android in that it would be more likely to help WebOS than Android.

WebOS, meanwhile, has a distressing shortage of apps. With such a small market share, it's been hard to attract developers to the platform, and the lack of apps is at least partly to blame for continued poor market share. HP's play to put WebOS on PCs may help that, but for now WebOS could benefit significantly from any additional apps it can get. Reimplementing the iOS libraries would be prohibitively expensive, but it isn't too far-fetched to imagine Android apps running on WebOS. Much of the technology is similar (Linux underneath, similar notification systems) and the parts that are different are available as open source. WebOS could become a very pleasant way to run Android apps. Even better, WebOS's unique features, Synergy and Touchstone, could be made available via additional Android APIs and encourage Android apps that only run on WebOS. For those who are interested in history, this is nearly identical to Microsoft's Windows-specific Java APIs that was ultimately considered a monopolistic abuse (see Breaking Java's Portability). In this case, however, HP and WebOS hardly have monopoly power to abuse.

Final Thoughts

I don't really think that HP will support Android apps on WebOS. Despite the ease with which they could, the ongoing support costs might not make it worth it. There is also the opportunity cost of porting the Android runtime rather than improving WebOS itself. I think it has the potential to be a great way out of the chicken and egg problem of attracting app developers and market share. They could even work with the Amazon Android App Store to offer a more curated set of Android apps, possibly avoiding the problem Android app marketing has of the userbase being largely unwilling to pay for apps. If HP can frame it as a bootstrapping opportunity instead of a betrayal of "native" WebOS app development, it has real potential. I won't place bets on whether it will come to pass, but I'd like to see WebOS get a little push toward success. Enjoy!

Update 2011-02-11! I should have considered the Blackberry Playbook as a host platform. Looks like RIM may be planning to support Android apps on the Playbook.

Why Software RAID?

The first question, of course, is why RAID at all? (Okay, the first question might be "what is RAID?" but I'm not going to address that here. See the Wikipedia article on RAID if you aren't familiar with it. For the purpose of this post I'm only going to be talking about RAID1. You may also find this posting on why RAID5 is the wrong choice interesting.) Some data you can always get again. Installed software (including the OS) is the big one, but also purchased (or free) audio/video (e.g. stuff bought through iTunes, music/video legally ripped from your own CDs/DVDs, downloaded podcasts, etc.). There is also plenty of stuff you could recreate, possibly even better the second time, such as various configuration files and user customizations. RAID is for everything else: pictures and video of your baby, programs you've written, all the email you've saved over the years, etc.

While RAID doesn't replace regular backups (including offsite backups), it protects you from minor disasters. Anything in the chain of hardware making your data available, including the disk, disk enclosure, cable, disk adapter, motherboard, power supply, memory, network adapter, network cable, etc. can fail. These components are all relatively inexpensive (the I in RAID) these days. Replacing the failed component brings you back up and running in no time, unless it's the disk with your data on it. RAID makes recovering from a failed disk as much of a simple drop-in replacement as a network cable (well, mostly — more on that later).

RAID is becoming increasingly popular in consumer-level machines, but the way machines tend to be preconfigured is exactly wrong for consumers. If I buy a desktop machine with two internal hard drives connected to a hardware RAID card that is preconfigured in BIOS and automatically boots into Windows with everything just working. It's dead easy, until a drive actually dies and needs replacing. If the user is tech-savvy, it's a matter of shutting the machine down, opening the case, pulling out the dead drive, replacing it, and possibly poking at the BIOS on boot to make it aware of the new drive. There is no good reason for that downtime. Furthermore, moving to a new computer means copying all the data from the old computer rather than just transferring the physical drives. Worse, if the hardware RAID controller dies there is no guarantee that it will be possible to find another RAID controller that understands the old controller's disk format. There is no standard disk layout for RAID, so each manufacturer's controller's use their own proprietary format. The format may vary from model to model or even version to version from the same manufacturer.

To be fair, it can be much easier to replace drives and move to a new computer. The Drobo is a good example. Where it fails is on both price and proprietary format. If your Drobo fails, you need a new Drobo to get to your data. There is no other manufacturer you can turn to, and if Data Robotics goes out of business then you will be obligated to pay whatever the asking price is on eBay to replace it, or lose your data. A good backup plan will let you restore from a backup to some other manufacturer's system, but now you're talking about serious downtime.

My setup uses software RAID under Debian GNU/Linux. I have an old machine I bought for $200 (including shipping!) from some surplus storefront on the web. I have a no-name eSATA PCI card, a pair of no-name eSATA enclosures, and a pair of disks which are probably either Seagate or Western Digital. When I get the chance, I'm going to add a third drive and enclosure as a hot spare. I keep around spare cables, and I can buy a replacement component, even the computer, for little cost and receive it quickly. Replacing a disk doesn't require any downtime, just unplugging the dead one, plugging in a new one, and letting Linux know about the new one. I know that I will always be able to move my RAID to a new machine because I will always be able to get the same (or backward-compatible) RAID implementation under Linux.

There are tradeoffs, of course. Software RAID will never be as fast as hardware RAID (at least not for a reasonable price). There is some manual configuration of software RAID under Linux, and it will never be as simple as a Drobo or a preconfigured system. If you don't use the Linux RAID machine as your primary computer, there is configuration involved in setting up network shares (especially in making them secure). If you prioritize availability, maintainability, and price over simplicity, as I do, Linux software RAID is the right choice. If you have other priorities and are fully aware of the tradeoffs, however, it may not be the right choice for you. Enjoy!

What WebDAV Enables

I have lots of ideas for iPhone apps. Ideas, of course, are a dime a dozen unless you execute on them. When I choose an app idea to execute, I base that choice on what I can do with a reasonable level of effort since I have to fit it into my free time. That means that I've shelved all of the ideas that require a WebDAV client.

All sorts of things would benefit from writing to or reading from a network-accessible filesystem, from a simple TODO list (I'd considered implementing TaskPaper, for example) to a game with user-generated levels. I'm thinking in terms of WebDAV for a few reasons:

• MobileMe's iDisk is a WebDAV filesystem
• So is a subversion server
• WebDAV works equally well over a VPN, through a proxy, to a remote server, or over a LAN
• There are already a number of iPhone apps that provide a WebDAV share

The problem is that no suitable WebDAV library exists. There are C codebases that might do the job (e.g. neon and cadaver), but they are licensed under the GPL. That's an inconvenient license for software going into a commercial iPhone app. Here's a list of requirements to define what I consider suitable:

• Its license must be MIT, BSD, Apache, LGPL, or something similarly non-viral and free.
• It must be based on the NSURL* class family so that it picks up and uses the iPhone's current proxy settings. ASIHTTP* is not acceptable.
• It must provide a reasonably simple Objective-C interface to:
  • reading files
  • writing files (guaranteeing, by default, atomicity via both explicit WebDAV locking and ETags)
  • listing directories (including files' MIME types)
  • reading byte ranges, i.e. partial file reading
• It must be easy to build a static fat library for iPhone, iPad, and simulator, preferably because the source is delivered with an Xcode project file

If anyone is interested in implementing and licensing such a library according to the requirements above, I'll offer a bounty of US$50. If I get enough (non-anonymous) comments on this post with pledges to increase that bounty to a total of $200 or more, I'll set up a bounty fund and open it to contributions.

To claim the bounty, comment on this post with an URL pointing to your repository on Sourceforge, Google Code, Github, or something similar and I'll check it out. Payment will probably be via Paypal.

Yes, I feel a little silly offering a bounty when I could implement it myself, but since I haven't found the time in the course of nearly two years, I don't see that changing. This way, we can all benefit. Enjoy!

Update! Apache Jackrabbit includes an Apache-licensed (of course) WebDAV client library... in Java. A port might be easier than starting from scratch.

Hardware Services Startup Idea

I have an idea for a startup. I am not the right person to start it (and it might even be a conflict of interest to consult on it), but I believe that the technology is almost cheap enough and almost (or maybe already) mature enough that it should start work now so that it can launch right about when the price point will be right. More to the point, I want to make use of the services this business can provide. That's right, I want someone to start a company that I can pay to make me happier.

I've been doing work with embedded systems for the first time over the last several months, and I'm astonished by how inaccessible it is to individuals and small businesses. An embedded system, for those who don't know, is a computer designed for a more limited set of tasks than a general purpose desktop or laptop computer. That includes a lot of stuff most people use every day, like microwave ovens, TiVos, digital thermostats, wifi routers, etc.

A big company, with plenty of capital to invest, can choose suppliers for the various components they'll need in a particular product, design a circuit board and housing for the components, develop prototypes, develop the software for it, manufacture several thousand, and sell it through traditional sales channels (e.g. brick and mortar stores or direct business-to-business sales) and maybe online as well. That's how most off-the-shelf (i.e. not custom developed for a client) embedded systems are created, produced, and sold. That won't work for a startup or a hobbyist. Honestly, it won't work for most small companies. Even putting the capital costs aside, it takes a lot of time. Worse, if you don't have market clout many chip and board suppliers won't do business with you at all.

Designing, manufacturing, and selling art was once in the same situation. Note that I'm talking about multiple instance art (i.e. not including things like non-cast sculpture, physical paintings, etc.). There were services to print your photograph or digital art, or cast your figurine, but you had to go about shipping and even warehousing your artwork yourself. Worse, small runs of items are costly, and cut into artists' profits. A few companies came along and changed all that. DeviantArt provides print-on-demand art sales on artists' behalf, CafePress provides storefronts for art printed on a wide variety of clothing, mugs, etc., Zazzle provides a similar service, Topatoco specializes in merchandising for webcomics, and Threadless sells submitted art on T-shirts and shares profits with the artist, to name a few. I don't know who was first, and it doesn't really matter. There is clearly a thriving market not just for art, but for providing artists with the services they need to manufacture and distribute their art effectively.

I'm pretty sure there is a similar market of individuals and small businesses just waiting to be tapped who would develop embedded systems in a creative explosion if the costs were within reach. There are companies and products that go part of the way, like Gumstix, buglabs, and Marvell's Sheeva, but they don't do anything to support sales and distribution, and have very limited manufacturing options. Providing embedded product services similar to the art product services linked above is more than that. As I said, the technology may already be mature enough, and it is almost cheap enough. If you don't believe these premises, feel free to skip right to the comments section to tell me I'm an idiot but please include why you thing I'm wrong.

If you're still reading, here's where I'll try to sketch out what the services would look like. For the purposes of this discussion I'm going to talk about customers, who are the ones developing embedded products, and consumers, who are the ones buying those embedded products. I'm also going to call the company WidgetCo for the sake of convenience. If I'm a customer thinking about developing an embedded product, the first step is figuring out what I need. I certainly need a processor and some memory, but I could need any combination of the items on this clearly incomplete list:

• wired ethernet
• wireless ethernet
• bluetooth
• USB
• serial port (RS-232)
• removable storage (SD card, etc.)
• video out (VGA, DVI, HDMI, etc.)
• video in (IEEE1394, S-video, etc.)
• LCD screen
• LCD touchscreen
• 3D acceleration
• ASIC for some set of codecs (e.g. MPEG)
• knobs or dials
• digital audio out
• headphone and/or microphone jack
• mono or stereo speakers
• condenser mic

There are probably lots of other options I haven't thought of, but I think it will be a while before even that long a list of options will be feasible for an early startup. (One thing I deliberately left off that list was a bunch of pins with fine control over each one's voltage/current. That sounds like a support nightmare to me, but I may be wrong.) There will need to be a few development board configurations at different price points, the most expensive having everything on it. If I'm going to develop a product, I'll need to be able to develop on something. A good pricing strategy might be to mark up the development boards by some deposit amount that will cover the cost of a prototype unit (see below); it's an early revenue stream, and discourages time wasters.

WidgetCo will have to provide some sort of SDK/toolchain available for download as well. I recommend a VirtualBox VM with a Debian GNU/Linux system set up with a toolchain and libraries using Emdebian for the SDK. I'd recommend providing an Emdebian rootfs as an SD card image for the platform OS itself. It would be a good idea to include a standardized software mechanism for upgrading the device's firmware in the field to make it easier for the customer to release revisions. The development boards should have a physical switch (or jumper, if you must) to switch between using an SD card and an EEPROM (or similar) as the root filesystem; it's much faster and easier to develop with writable storage, but later stages of testing should be done in an environment closer to the shipping product.

Once I've developed my product and I'm ready to have it produced, I'll upload a firmware image (which has been tested by loading it into the EEPROM on the board). WidgetCo will send me a prototype unit, already paid for by my development board deposit (see above), with my firmware image installed. If multiple prototypes are required, they will have to be paid for separately. The prototype unit should be just like the final product that will be sold to consumers, but will have a mechanism to load a new firmware image. Think of the prototype as similar to galley proofs, where the customer is responsible for testing and approving it before any more units are produced.

Once the prototype is approved (which may require further revisions to the firmware image), the customer uploads the latest firmware and sets up their storefront to sell units. This will include pricing options, such as a minimum profit markup, requiring a minimum number of units per order, bulk discounts, etc. The profit to the customer will naturally be smaller for low volume and higher as the volume increases. Once the device is available for sale, all transactions, manufacturing, and fulfillment will be handled by WidgetCo on the customer's behalf. WidgetCo will send the customer a check (or, more likely, a direct bank transfer) on a regular (weekly? monthly? quarterly?) basis.

The customer should also be able to upload new firmware versions when necessary, and the storefront should include a place for firmware downloads. Assuming the customer used the standard firmware upgrade facilities, WidgetCo's standard firmware upgrade utility (which should be available for Mac, Windows, and Linux) will do the job. The utility will take some significant work, since it should be able to function over bluetooth, wifi, or USB depending on what capabilities the customer's device has. Even more importantly, it must be friendlier and easier to use than most of the firmware upgrade tools out there. A friendly firmware upgrade process is a competitive edge for both WidgetCo and their customers.

There are lots of details that will have to be worked out, especially involving the logistics of manufacturing, distributing, and storing units. What branding/logo options will be available? What housing variables need to be considered (battery-powered, digital (e.g. USB) line-powered, DC line-powered, fans, ruggedizing, colors)? What standard sets of components will be available as board configurations? If I'm a big enough business, can I skip the storefront and housing and just order configured and loaded boards in large lots to assemble and deal with myself? Even so, and even starting with very limited options, I see a lot of money to be made and a lot of customers (and consumers) to make happy.

As I said before, I'm the wrong person to make this happen. I just want to make use of WidgetCo services. If you take this idea and run with it, let me know. If you make a truckload of money, feel free to toss me some, but my intention is to make this idea available to anyone with no strings attached. Good luck! Enjoy!

In Defense of STI

I've been seeing a lot of hate for Single Table Inheritance. It's a part of ActiveRecord that a lot of people misuse, and in reaction many people have decided that it shouldn't be used at all. It's just a tool in the toolkit, though, and like most tools it can be used appropriately or inappropriately. I'm not going to claim that STI is so amazing that everyone should use it. I just want to present one use case for which it is a good solution, and try to generalize about what makes that use case lend itself to an STI solution. I'll also try to cover why polymorphic associations are not appropriate in this case, since I've seen people claim that anywhere you might use STI you should use PA instead.

My use case for STI is a storefront app template I worked on a while back. In modeling the products available in the store, I used a single Products table. Initially, there were two types of products: Intangible and Shippable. Shippable products have to have a weight, and intangible products must not. An intangible product is something like event registration or a donation, but it might be an electronic download. In fact, Downloadable is its own class, and subclass Intangible. In addition there are bundled products for placing a special price on some grouping of basic products, e.g. event registration plus a paper copy of the proceedings plus a PDF of the proceedings.

There are several fields (e.g. name, description, image) and relations (e.g. price, since it was actually tied to the customer type, which is often but not always "guest") that all of these products share, as well as some code, but that isn't where using STI shines. The differences between the products is where it's important. Inheritance (the I in STI) is a terrible way of sharing code, but it's a great way to model polymorphism. Yes, Ruby's duck typing makes it possible for any set of objects implementing the relevant methods to be used polymorphically, but inheritance has a semantic meaning. While various collections are enumerable, a shippable product is a product. A downloadable product is an intangible product. Modeling a meaningful hierarchy of product types is more maintainable than having a loosely coupled sack of product-like models.

Consider maintaining the administrative interface for bundles. Any time we need a new product model it is immediately available to be bundled if it's a descendant of Product. If we use polymorphic associations it is certainly possible to add some other model to a bundle, but the admin interface needs to be modified to make that new product type available to be bundled. Similarly, one store had a "featured product" and adding another product type would have required modifications to the administrative interface where it was exposed. The UI for administering products will need modification when a new product type is added, of course, but that's true regardless of whether we use STI or PA.

Both the shopping cart model and the submitted order model have to interact with product models, but that could be handled easily with polymorphic associations. The product catalog, however, gets much more complicated if products aren't all coming from the same table. Sorting all products by name is no longer an ORDER BY in SQL generated by ActiveRecord, but becomes an explicit sort_by on an array in Ruby. Pagination combined with a sort requires loading all the models from all the various product tables. It isn't just a matter of performance, it also requires reimplementing pagination instead of being able to use will_paginate.

Extending the model is also easy. Shippable products keep track of their inventory, but consider that event registration is a situation where an intangible product still has limited inventory. It makes sense to create a product subclass for electronic tickets of various sorts that deals with inventory. It could subclass Shippable to reuse the inventory code, but that's misusing inheritance; an electronic ticket is an intangible product. Instead, we would refactor some or all of the inventory code from the Shippable class and put it into a LimitedSupply module and include it in both the ETicket and Shippable classes.

What is it about this use case that lends itself to single table inheritance yet matches poorly with polymorphic associations? Part of it is that products are involved in so many parts of the system. It's browsed by the user, administered by the store owner, recorded by carts and orders, and bundled by other products. Another aspect is that the model is expected (and intended) to be extended. No one expects to be able to define a priori everything that could be considered a product (i.e. anything a customer will pay for). Finally, and perhaps most importantly, it really is an overarching concept that lends itself to being modeled as a hierarchy of subconcepts. Stores actually sell shippable, intangible, and bundled products.

So there you have it. I won't argue over whether STI can be misused, but here's a case and maybe some attributes of that case where STI is used appropriately. I'll try to keep this updated with insight from or in response to any comments. Enjoy!

Solidarity: Taking Action so Apple Will Care

We know that Apple's iPhone app approval process is too long, too opaque, too inconsistent, and too arbitrary. We know the parental controls are having frustrating repercussions. We know the ad hoc distribution process is frustrating and overly restrictive. We know that the rating system is biased toward the negative. We know that in-app purchases support a (mostly) broken business model. We know that iTunes Connect is buggy and user-hostile. And, worst of all, we know that Apple has no motivation to care or to fix it since their numbers look great regardless of how they treat developers. What can we do about it as mere developers playing in Apple's walled garden of a market?

Suppose Apple's vaunted numbers fell through the floor one day. 65,000 apps on the App Store? How about 200 apps on the App Store instead? It will require solidarity among developers, but I see no other way to be heard. I propose that on August 16 all developers set all of our apps' availability dates to August 23 (dates chosen arbitrarily). For one week, Apple will see App Store numbers in the toilet. Will it be enough? I don't know, but I haven't come up with a better idea and I don't know of anyone who has. Please comment if you are on board, or if you have a better idea.

Update! First, another good writeup on the problems with app submission. Second, I realized that sending a message to Apple means actually having a message to send beyond "we're not happy." Here are some ideas of what would improve things:

1. Lay out a sequence/checklist of stages for a review (e.g. awaiting assignment, preliminary review, UI review, API review, network review, etc.) and report which stages have been completed, and when, for each app within iTunes Connect.
2. Assign each reviewer a number which will be attached to each stage. This is similar to the "inspected by #7" slips found in (good) clothing.
3. Detailed remedy requests on rejection, e.g. "By changing the bookmarks button that does not access bookmarks functionality to a more appropriate icon or title will solve this rejection issue."
4. Enable parental controls on promo code redemption so that apps with a 17+ rating can give out promo codes safely.
5. Allow developers to remove up to 100 devices, individually, from the program portal over the course of a 1-year license term rather than a single opportunity on renewal to reset all.
6. Prompt users to rate apps they have had installed for a week instead of or in addition to prompting on deleting an app.
7. Support the try-before-you-buy (i.e. free apps with a paid upgrade path to a paid version) and subscription business models with in-app purchases.
8. Make iTunes Connect clearly and dependably tie App Store description/screenshot/etc. content to specific versions of the app, i.e. I should be able to make changes to the description for the version that is currently for sale and have it appear immediately, and I should be able to make changes to the description for the version that is currently in review that will be seen when the new version is approved, and not before.

Did I miss anything? Is any of that unreasonable? Please discuss.

Quick JSON to Plist Script

I need to do some gesture recognition in an iPhone app I'm working on and remembered hearing about GLGestureRecognizer. It looked promising, so I grabbed it from GitHub and took a look at the project. It comes with a JSON configuration file and requires (but does not include) TouchJSON to parse it. It seemed silly to include an entire JSON parsing library to read in a config file that could just as easily be an Apple-standard plist file (and thus trivial to load), so I decided to write a quickie script to convert it. This is the result:

#!/usr/bin/env ruby

require 'rubygems'
require 'json'
require 'plist'

outfile = case ARGV.size
          when 0
            text = ARGF.read
            self
          when 1
            text = File.read(ARGV.first)
            newname = ARGV.first.sub(/(\.json$|.js$|$)/, '.plist')
            File.open(newname, 'w')
          when 2
            text = File.read(ARGV.first)
            File.open(ARGV.last, 'w')
          else
            STDERR.puts "Usage: #{$PROGRAM_NAME} [infile [outfile]]"
            exit 1
          end

outfile.puts JSON.parse(text).to_plist

To use this you will need the json and plist gems. Also, the json gem won't tolerate the trailing commas present in the Gestures.json file in the GLGestureRecognizer project, so I had to tweak that by hand. It does the job, though. Enjoy!

Managing iPhone Development

I've spent a lot of time fighting with certificates, keys, and provisioning profiles in the time I've been working on iPhone apps for clients and myself. I finally figured out how to make it easy to manage multiple sets of certs/keys (i.e. one per client). Even if you are only working with a single set, though, it's still helpful to keep iPhone stuff separate from the rest of your keychain. I'm going to approach this as if you are starting from scratch, but it's easy enough to fix if you have already set up certs and keys and it should be pretty obvious how to go about it from these instructions.

First off, you need to know your tools. Keychain Access is where all of your certificates and keys (and passwords, and a variety of other things irrelevant to this discussion) live. Xcode, iPhone Configuration Utility, and iTunes all deal with the same store of provisioning profiles, but only the configuration utility is actually good at it. Download it from Apple right now and install it. When working within the iPhone Configuration Utility (hereafter referred to as iPCU) neither iTunes nor Xcode should be running since they will need to be restarted anyway to see any changes you make.

Before anything else, you need to download and install the WWDR intermediate certificate if you haven't already. Get it here and open it in Keychain Access (hereafter referred to as KA). You'll want to install it in either the login or System keychains (it doesn't matter much). Now that you're in KA, create a new keychain (File menu) and name it for the particular program portal you're working with at the moment. I recommend saving it in the default location, ~/Library/Keychains, but if you save it somewhere else just make sure you remember where. You'll need to set a password for it, and you can choose to be as secure or insecure as you like about it; the certs/keys would otherwise be in the login keychain, which is open by default as long as you are logged in, so anything is more secure than the alternative. Follow the program portal instructions for creating a Certificate Signing Request (CSR). Notice that creating the CSR created a private key in the login keychain in KA. Drag that private key to the new keychain you just created (the client keychain). Upload the CSR and go through the process of getting a developer certificate and a distribution certificate. (You'll need both eventually, and you can use the same CSR for both; if you don't, a new private key may be generated for the second CSR, and you'll need to drag that from the login keychain to the client keychain as well.) Install the certificates in the client keychain rather than System or login.

When you are done with this process, you should have a private key (maybe two — see above), a developer certificate, and a distribution certificate in the client keychain. I like to set the keychain to lock after a period of inactivity so Xcode asks me for a password when it codesigns and I know it's doing what I expect. Remember where you saved the keychain file? Make a backup copy of it now and put it somewhere safe (source control, offsite backup, optical media, whatever). If this is the only program portal you deal with, you're done. If not, right-click (or ctrl-click) on the client keychain and choose Delete Keychain "[whatever]".

IMPORTANT: Be sure to choose "Delete References" and not "Delete References & Files"!!! If you choose the wrong one, you will be glad you made that backup copy. KA will close the keychain, but you can open it again when you are working with that program portal again. You can then repeat the process for any other program portals that involve you.

Clearly, if someone else has created the distribution certificate you need to use you will need to get the cert and private key from that person instead. You can still put them in the client keychain once you have them, of course. If you have already been dealing with certs in your login keychain, you might have lots of private keys lying around and no good way to tell which key goes with which cert. I feel like there should be an easy way to tell, but I haven't found it. Instead, create a new keychain and put all but one of the private keys into it, leaving one in the login keychain, then lock that temporary keychain (i.e. click its lock icon in KA). Build something in Xcode that requires the codesigning cert you are testing and see whether it asks for a keychain password. If not, the key you left in the login keychain goes with that cert; otherwise, switch keys and try again. (You can be cleverer about it by locking away half the keys so it's a binary rather than linear search, plus you can test more than one cert at a time, but I leave that as an exercise for the reader.) Eventually you will be able to associate keys with certs and put them in their appropriately separate keychains.

Next up, we'll look at provisioning profiles. There are three kinds of profiles: development, ad hoc, and app store. Both ad hoc and app store are considered distribution profiles, but they behave differently. In fact, ad hoc profile behave more like development profiles than app store profiles. (Note: there may be still yet another profile type for enterprise distribution, but I have no experience with that.) A development or ad hoc profile permits an app with a particular app ID (or ID prefix) to be installed on any of a set of physical devices when signed by one of a set of certificates. For ad hoc, it's only one certificate: the distribution certificate. A device must have the provisioning profile installed on it to run the app, which Xcode does automatically for development profiles. I've had a lot of trouble with ad hoc profiles, and I'm still not confident I can get things working 100% of the time, but I have a better grasp on it than I used to. For the sake of my own sanity I am going to assume that you have figured out how to set up app IDs, devices, and provisioning profiles in the program portal.

There isn't a whole lot more to it, really, except keeping track of which profiles belong to which portals if you are dealing with more than one. I recommend naming the profiles carefully when you create them or, failing that, keeping a text file listing what each profile identifier is for. Whenever you have a new profile you'll want to use to build an app, I recommend installing it in iPCU rather than Xcode. It seems to work more dependably for me. Also, if it's an ad hoc profile, I recommend installing it on the device using iPCU rather than iTunes if at all possible. If you use the multiple certs/keys keychains trick, I'd like to tell you that there is a similarly good way to manage provisioning profiles; I don't know of one. The good thing, though, is that Xcode is smart enough to check the currently open keychains when presenting you with a list of provisioning profiles in a project's (or target's) build settings. If the cert for a particular profile is not available (even if the keychain is locked, its contents are available as long as KA has it listed), it will be grayed out with a message saying <matching certificate identity with private key not found in login keychain>. No matter how many provisioning profiles you have installed from other program portals, only the ones related to the certs and keys you have open will be available, which helps avoid silly mistakes.

I hope this is helpful to someone out there. I know I wish I'd known this stuff when I started developing for the iPhone. Enjoy!

Update! The 3.0 SDK deals with things a little differently. The separate keychains trick still works well, but you need to manually set the default keychain in Keychain Access to whichever one is appropriate for your current project.

Less Ruby, More Cocoa

I've been doing mostly iPhone development recently, so I haven't had a whole lot to say about Ruby. As such, I'm going to start talking about some Objective-C stuff here. I am still doing Ruby/Rails work, so there will be more of that, too. I'm also hoping to be chosen to speak at either or both of Scotland on Rails and RailsConf.

SSL Certificates and Net::HTTPS

I was getting tired of seeing "warning: peer certificate won't be verified in this SSL session" from Ruby's net/https library, so I started looking around for how to get it to actually verify the SSL certificate. I found lots of links on how to tell it not to bother verifying, but it wasn't until I found someone's Japanese blog that I found the clue I was looking for. Now, I don't know Japanese, but I can read Ruby. For the benefit of other English speakers/readers out there, I now present the solution.

First off, I'm giving a full example request using basic authentication (not that GMail uses basic authentication, but this is an example) because I was unable to find a good example elsewhere and made it this far by trial and error. The following will produce the warning I mentioned:

require 'net/http'
require 'net/https'
require 'uri'

url = URI.parse 'https://myname:mypass@mail.google.com/'
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = (url.scheme == 'https')
request = Net::HTTP::Get.new(url.path)
request.basic_auth url.user, url.password
response = http.request(request)

To avoid the warning, we can either tell it not to warn us and blithely accept whatever certificate we receive, or we can give it enough information to authenticate the certificate against the root CA certificates. In the following example, we'll do both. If we find the file /usr/share/curl/curl-ca-bundle.crt then we will verify, otherwise we will silently ignore the issue:

require 'net/http'
require 'net/https'
require 'uri'

RootCA = '/usr/share/curl/curl-ca-bundle.crt'

url = URI.parse 'https://myname:mypass@mail.google.com/'
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = (url.scheme == 'https')
if File.exist? RootCA
 http.ca_file = RootCA
 http.verify_mode = OpenSSL::SSL::VERIFY_PEER
 http.verify_depth = 5
else
 http.verify_mode = OpenSSL::SSL::VERIFY_NONE
end
request = Net::HTTP::Get.new(url.path)
request.basic_auth url.user, url.password
response = http.request(request)

And there you have it. Not all that complicated, but poorly documented. Now it's more findably (i.e. Googleably) documented. Enjoy!

Update 2009-05-21: Thanks to a comment from Chewi, here's an even better approach:

require 'net/http'
require 'net/https'
require 'uri'

RootCA = '/etc/ssl/certs'

url = URI.parse 'https://myname:mypass@mail.google.com/'
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = (url.scheme == 'https')
if File.directory? RootCA
  http.ca_path = RootCA
  http.verify_mode = OpenSSL::SSL::VERIFY_PEER
  http.verify_depth = 5
else
  http.verify_mode = OpenSSL::SSL::VERIFY_NONE
end
request = Net::HTTP::Get.new(url.path)
request.basic_auth url.user, url.password
response = http.request(request)

CalTerm: Ncurses Calendaring

It's no iCal-killer, but I needed something to let me at least view my ical calendars from the terminal. At this point, other than bugs that can be traced to vpim (having to do with recurring events that hit the wrong days, e.g. leap day), it works. It's far from my first Ruby code, but it's my first public Ruby project.

That's the good news. The bad news is that it's definitely still rough around the edges, it isn't packaged in any way, and you'll have to install the vpim and ncurses gems to use it. Grab it from subversion at http://calterm.rubyforge.org/svn/trunk/ or start at the CalTerm home page.

This is sort of a preliminary announcement before I've figured out how I want to package it, so there's no release version.

Enjoy!

The Quest for the UPS API

This is a bit of a departure for this blog, but it's been a while since I posted and I'm inspired by the hoop-jumping UPS (yes, that UPS) requires for developers to begin using their web service APIs. There are NDAs involved, so this is not about the APIs themselves; it is about getting to the point where one can start working with the APIs. It is presented in the format of a text adventure. Pedantic corrections to format are welcome, since it's been a long time since I've played one.

Welcome to Colossal API Quest! Find the documentation, get authorized to use the development integration environment, and meet the needs of your client to achieve fame and fortune!

• You are in a maze of twisty little passages, all alike. There is a door to the North.
• N
• Welcome to UPS. A GATEKEEPER blocks the door to the North. There are also doors to the East, West, and South.
• talk to the gatekeeper
• "Username and password or register?"
• say register
• "Tell me everything about yourself. Also, what is your UPS account number?"
• N
• "Username and password or register?"
• E
• Welcome to UPS. There are several signs here advertising the wonderful web services UPS provides and pointing to the door to the North. There is also a door to the West.
• N
• The door is locked.
• W
• Welcome to UPS. A GATEKEEPER blocks the door to the North. There are also doors to the East, West, and South.
• W
• Your CLIENT is here. There is a door to the East and to the West.
• talk to the client
• "Is my website ready yet?"
• need UPS help
• What?
• say "need UPS help"
• "You should talk to our UPS contact. She's in the next room."
• W
• There is a notepad in an otherwise empty room. There is a door to the East.
• get notepad
• You can't take that.
• look notepad
• It says, "Sorry, I'm on vacation for the next 6 turns. Please leave a message."
• wait
• wait
• wait
• wait
• wait
• wait
• look
• There is a notepad in an otherwise empty room. There is a door to the East.
• look notepad
• It says, "Sorry, I'm not here right now. Please leave a message."
• write "please tell me how to get the API documentation"
• Write with what?
• inventory
• You have:
  • frustration
  • deadlines
  • a pen
• write "please tell me how to get the API documentation" on notepad with pen
• OK
• wait
• look
• There is a notepad in an otherwise empty room. There is a door to the East.
• E
• Your CLIENT is here. There is a door to the East and to the West.
• talk to the client
• "Is my website ready yet?"
• say "can't find UPS contact"
• "You should talk to our UPS contact. She's in the next room."
• xyzzy
• You are in a familiar room, but there is no notepad. A CONTACT is here. There is a door to the East.
• talk contact
• "How can I help you?"
• say "UPS API documentation"
• "Here is the CLIENT's username and password"
• inventory
• You have:
  • frustration
  • deadlines
  • a pen
  • username
  • password
• say "UPS account number"
• "Here is the CLIENT's account number"
• E
• Your CLIENT is here. There is a door to the East and to the West.
• E
• Welcome to UPS. A GATEKEEPER blocks the door to the North. There are also doors to the East, West, and South.
• give username to gatekeeper
• "Password?"
• give password to gatekeeper
• The GATEKEEPER leaves the room. You have defeated the GATEKEEPER! You have earned 6/120 points.
• N
• Welcome to My UPS. There is a pile of documentation here. There is a WEBSITE here. There are two doors to the North. There are also doors to the East, West, and South.
• get documentation
• Which documentation?
• look documentation
• There are lots of... look, let's just skip this step. Get the bits you want, they're all here.
• get time in transit documentation
• Okay, I lied. Do you want HTML or XML documentation?
• both?
• OK. You have earned 10/120 points.
• inventory
• You have:
  • frustration
  • deadlines
  • a pen
  • username
  • password
  • account number
  • Time in Transit XML documentation
  • Time in Transit HTML documentation
• get tracking documentation
• Okay, I lied. Do you want HTML or XML documentation?
• both
• OK. You have earned 14/120 points.
• look time in transit xml documentation
• It says, "You need an XML access key to use the development integration environment."
• look tracking html documentation
• It says, "You need an HTML access key to use the development integration environment."
• look N
• The two doors to the North are marked "XML access key" and "HTML access key". You can refer to them as N1 and N2.
• N1
• The door marked "XML access key" is locked. You need the developer key.
• N2
• The door marked "HTML access key" is locked. You need the developer key.
• talk website
• How can I help?
• say "developer key?"
• "I can resend your developer key to the email address in your profile, or you can give me another email address to send it to and I will magically update your profile."
• ponder wisdom of changing email address on client's account
• What?
• say "send to profile email"
• OK. You have earned 15/120 points.
• S
• Welcome to UPS. There are doors to the East, West, and South.
• W
• Your CLIENT is here. There is a door to the East and to the West.
• talk to the client
• "Is my website ready yet?"
• say "need developer key"
• "What?"
• say "need UPS developer key"
• "What?"
• say "need UPS email"
• "You mean this thing?"
• inventory
• You have:
  • frustration
  • deadlines
  • a pen
  • username
  • password
  • account number
  • Time in Transit XML documentation
  • Time in Transit HTML documentation
  • Tracking XML documentation
  • Tracking HTML documentation
  • an email
• look email
• It contains a developer key. You have earned 87/120 points.
• inventory
• You have:
  • overwhelming frustration
  • looming deadlines
  • a pen
  • username
  • password
  • account number
  • Time in Transit XML documentation
  • Time in Transit HTML documentation
  • Tracking XML documentation
  • Tracking HTML documentation
  • a developer key
• E
• Welcome to UPS. There are doors to the East, West, and South.
• N
• Welcome to My UPS. There is a pile of documentation here. There is a WEBSITE here. There are two doors to the North. There are also doors to the East, West, and South.
• N1
• Welcome to My UPS. There is an XML access key here. There is a door to the South.
• get key
• There is no key here
• get xml access key
• OK. You have earned 95/120 points.
• S
• Welcome to My UPS. There is a pile of documentation here. There is a WEBSITE here. There are two doors to the North. There are also doors to the East, West, and South.
• N2
• Welcome to My UPS. There is an HTML access key here. There is a door to the South.
• get html access key
• OK. You have earned 103/120 points.
• S
• inventory
• You have:
  • overwhelming frustration
  • looming deadlines
  • a pen
  • username
  • password
  • account number
  • Time in Transit XML documentation
  • Time in Transit HTML documentation
  • Tracking XML documentation
  • Tracking HTML documentation
  • a developer key
  • XML access key
  • HTML access key
  • the ability to finally make some progress

To be continued, maybe. Enjoy!

Randomizing an Array Revisited

It was pointed out in a comment on my post about randomizing arrays in Ruby that the sort_by{rand} is O(n log n), and it can be done in linear time, i.e. O(n). This is, of course, correct. Efficiency wasn't my primary concern in the original post, so much as a quick and easy to remember solution. That said, it's worth presenting the more complicated but more efficient algorithm.

I could just give the link to the blog post linked in the comment, but for the convenience of the reader I'll repeat the solution here (with minor changes that make me happier without changing the algorithm):

class Array
  def shuffle
    array = dup
    size.downto 2 do |j|
      r = rand j
      array[j-1], array[r] = array[r], array[j-1]
    end
    array
  end
end

Enjoy!

Do You Understand What Your Web Framework is Doing?

It's a new year, and I'm going to start off 2008 wrong with a code-free post. Sorry about that. This stems from realizing how little many developers (judging from postings to a variety of mailing lists) seem to understand about what their web frameworks do for them when it comes to generating code in other languages (particularly JavaScript and SQL). It's Rails-flavored, but not Rails-specific.

Here's a quick quiz. I'm assuming that you, the reader, are a web developer familiar with JavaScript, SQL, and some reasonably modern web app framework:

1. Does JavaScript validation on a web form guarantee that when the form is submitted the server will receive valid data?
2. Should foreign key columns each get their own index?
3. How is JSON parsed into data structures in memory in a browser?
4. Are multi-table joins inefficient?
5. Can a web page make requests to a host other than the host from which the page itself was requested?

We'll come back to that. I am going to start by talking about a web browser (client) interacting via HTTP(S) with a web server. There are three pieces here, not two. The HTTP protocol matters since it is easy to work with and understand and there are lots of tools for working with it. There are some important differences between the web client/server environment and a more traditional client/server system:

• Connections are not persistent, and consist of only a single request and response. (Note that HTTP keep-alive does not change this; what persists is the TCP connection, and does not affect the application layer.)
• Interaction can only be initiated by the client, not the server. This is a result of the previous difference.
• The server cannot assume anything about the data received from the client.

Most people developing web sites/applications think in terms of the server software they are developing. Much as first-time GUI developers often find it baffling, the inversion of control involved in modern web programming confuses many developers. The server has full control when responding to a request but, once it has generated that response, control reverts to the client. For one thing, that means that data on the client does not get to the server unless the client decides to send it. It also means that data from the server does not get to the client unless the client decides to request it. One needs to work from the point of view of the user in front of the browser.

A common question on the Rails mailing list is how to use RJS to retrieve some value from the client. While the desire isn't ridiculous, and it can be done in a roundabout way with a certain amount of jumping through hoops, phrasing the question that way shows a lack of understanding of where the RJS-generated code will be executing. (The hoop jumping involves having the RJS generate an AJAX request back to the server to submit the value back to some URL on the server.)

There was a recent thread on the Rails list complaining about the functionality in Rails (largely RJS and various helpers) that attempts to hide the complexity of interactions between client-side and server-side code and largely results in maintainability problems in the code and misunderstandings for the developer. I don't agree with everything in either the original post or the various responses, but it highlights a problem Assaf identified months ago.

Assaf is concerned with bad (inefficient and/or incorrect) SQL being generated because the developer doesn't understand what the framework is doing underneath. I'm concerned about bad (incorrect, unmaintainable, and/or hard to debug) JavaScript being generated. Rails makes it easy to get results without understanding what it is doing for you, which is great for prototyping and dangerous for production.

It is important to understand what code is being executed where, when, and how. When developing a rich user experience in a web browser, one must understand the DOM, the event model, the browser security model, the JavaScript language, the single-threaded nature of JavaScript execution in the browser, XMLHttpRequest, etc. just as one must understand database indexing, column types, SQL, table/row locking, etc. to develop a production-quality database-backed web application.

Let's go back to that quiz. You shouldn't have had to think too hard about any of these, and you should feel certain about your answers. And those answers should be:

1. Does JavaScript validation on a web form guarantee that when the form is submitted the server will receive valid data? Nope. The server receives data over HTTP, and that HTTP connection could come from any program, not just a web browser. Furthermore, JavaScript can be turned off in most browsers. On top of that, most browsers make it possible to mess with the web page live and/or the data being submitted. Client-side validation is a user interface nicety, but provides no guarantees about the data the server sees.
2. Should foreign key columns each get their own index? Sometimes. It depends very much on what queries will involve them. A join table (i.e. one with more than one foreign key that represents a many-to-many relationship between tables) usually benefits from an index on all foreign keys, sometimes even multiple indices of the same columns in different orders. Tables only queried by columns other than their foreign keys generally don't benefit from indexing those foreign keys, even if the table is usually joined against the tables to which those foreign keys refer.
3. How is JSON parsed into data structures in memory in a browser? Since JSON is JavaScript, it is executed with eval() to be parsed with JSON.parse() into memory in the browser.
4. Are multi-table joins inefficient? This depends on the number of tables, available indices, and the database engine. Joining 18 tables in MySQL can make the query optimizer hang for hours (that's the query optimizer, not executing the query), regardless of available indices on the tables involved. A query on tables lacking indices on appropriate columns will require full table scans in any database, which is always slow (unless the unindexed tables have very few rows). It is always worth asking your database engine to explain and profile the queries you'll be running. Incidentally, database logs from running your unit/functional/integration/whatever tests are a great place to start.
5. Can a web page make requests to a host other than the host from which the page itself was requested? Yes, but not with XMLHttpRequest. At the simplest level, an img tag makes a request from any arbitrary URL, though the response is not available to JavaScript. To interact with a different host with almost the same flexibility as an XMLHttpRequest, one uses a script tag. See this blog post for a discussion.

How did you do? If you didn't get them all, you need to keep learning. If you got them all right, don't get too cocky; you may still not know everything you need to know to avoid the pitfalls of a code-generating framework. I keep learning about things I thought I knew thoroughly, and I wouldn't have it any other way. Enjoy!

I'm at RubyConf!

I probably should have mentioned this earlier, but I'm at RubyConf this weekend. Find me if you feel like it. Here's a (crappy) pic to help you out:

The Magic in the Box

Disclaimer: This is a rant. If you are not interested in reading the rant, please don't, but please also do not comment unless you have read it all the way through.

I'm a pretty easy-going guy, and I like to be helpful on mailing lists (and in person), but I expect a basic level of knowledge from people, particularly about the tools they are using. For anyone writing software, the computer is their tool. (So is a text editor or IDE and compiler or interpreter, but that's not where I'm headed with this.)

A strong formal education in computer science provides an understanding of data representation, logic, memory hierarchies, algorithmic analysis (and common algorithms), concurrency, common/basic data structures, stack and heap layout, compiler design, and probably a dozen other things I'm omitting. (Ideally, program design and development is a part of that education, but it should not be at the expense of anything I mentioned.) It's challenging, but self-taught programmers can learn the same things on their own. And should. As a professional of any sort, one has a responsibility to understand both the unavoidable tools (in particular, memory and CPU details) and the available tools (go look up the heap data structure right now if you don't know what it is).

One can argue that newbies should be able to jump right into programming, but at the same time there really is such a thing as a stupid question. Nothing says, "I don't understand how computers work and shouldn't be paid to write software until I do," like a question that shows a complete unawareness of the accuracy limitations of floating point representation. There are certainly other stupid questions, but that one came up recently.

I could have ended this rant on the previous paragraph, but I was afraid it would come across as unreasonable and mean-spirited, and my goal is to express the reasoning behind the distress I feel when I read such questions. First, though, I want to address the newbie issue in greater depth. If one is teaching oneself how to program, that is to be encouraged, but it involves some work beyond following along in a book or online tutorial. For the floating point example above, the autodidact (great word, look it up) should be expected to follow roughly these steps:

1. discover an anomaly in floating point arithmetic
2. in irb, run something like "0.1.class" to discover that 0.1 is a Float
3. ri Float to discover that "Float objects represent real numbers using the native architecture's double-prescision floating point representation"
4. google for "floating point representation"
5. Read the very first link in the results, which has a specific section on the issue
6. achieve enlightenment

The point is that teaching oneself involves more than just reading a book and playing around, sometimes in involves taking time to investigate the parts that don't immediately make sense. (Note that those steps are the shortest, simplest path from confusion to understanding; it might take someone more work than that, but the fact remains that there is a simple and reasonable path to understanding and that it isn't particularly hard to find.) If one does not know how to learn on one's own, self-teaching is the wrong choice and it is time to go get that formal education.

A bit more on why these questions I'm calling stupid distress me so. Imagine yourself as a master carpenter who is training an aspiring apprentice. The apprentice has assembled some Ikea furniture and decided he likes carpentry, so he's going to give it a whirl. One day, he brings in a stool with a leg that needs replacing. He says, "I've been trying to remove the leg, but the screw is just stuck or something, and I think something is wrong with it." You look, and ask him to show you how he's tried to go about it. He picks up a screwdriver and desperately tries to fit its flat head into the cross of the phillips screw. You tell him to stop, take out a phillips screwdriver, and proceed to unscrew things easily. He gawks and asks, "There are different kinds of screws that need to be treated differently?"

Imagine your horror. How could he not know something so basic? Aspiring to be a carpenter requires understanding the tools a carpenter uses. Likewise, aspiring to be a programmer requires understanding the tools a programmer uses. Enjoy!

Named Array Slots

Sometimes you have an array of data that isn't quite complicated enough for a full-fledged data model, but you want to access elements by name rather than positionally. Probably you even have a bunch of these arrays with positions corresponding to named fields. These arrays might have come from a DBI query, or CSV, or parsing some arbitrary data file, but ultimately you have a need to make your code more readable and avoid poking at these data structures with error-prone magic numbers. What you actually want is a Ruby module that defines methods to access the fields, with which you can then extend the Array objects.

Suppose that your arrays represent users and have four elements, in order: name, gender, email, zip. The naïve, ad hoc way of doing things, then, is:

module MyFields
  def name
    self[0]
  end
  def name=(val)
    self[0] = val
  end
  def age
    self[1]
  end
  def age=(val)
    self[1] = val
  end
  def email
    self[2]
  end
  def email=(val)
    self[2] = val
  end
  def zip
    self[3]
  end
  def zip=(val)
    self[3] = val
  end
end

What a mess, and that's for just four fields! Let's do a little dynamic programming. It's still simple and ad hoc, but it's better:

module MyFields
  %w(name age email zip).each_with_index { |field,i|
    define_method(field) { self[i] }
    define_method("#{field}=") { |val| self[i] = val }
  }
end

Much better, and we can change the list of fields pretty easily. Still, if we have several different sets of fields (e.g. rows from several different database tables) that's a lot of syntax for something pretty simple. Also, if both the field names and data are coming from an external data source, you may only care about some limited number of those fields but still need to get all of them properly named in the correct order. Ultimately, you'd like to be able to take an array of arbitrary objects, convert the objects to strings, and get a module with which you can extend your row arrays out of it. Something like this:

class Array
  ConvertElementsToFields = lambda { |f|
    f = "#{f}" # get as a new string, even if it's already a String
    f.downcase!
    f.gsub!(/[^\w]+/, '_')
    f
  }
  def field_names_module(&convert)
    convert ||= ConvertElementsToFields
    fields = self
    Module.new do |mod|
      const_set 'Fields',
        fields.map(&convert).each_with_index { |f,i|
          f.freeze
          define_method(f) { self[i] }
          define_method("#{f}=") { |val| self[i] = val }
        }.freeze
      unless instance_methods.include? "field_list"
        define_method("field_list") { mod::Fields }
      end
    end
  end
end

The simple case, where we know the list of fields ahead of time, looks like this:

MyFields = %w(name gender email zip).field_names_module

The more complicated case where we don't know the field names/positions ahead of time is almost as easy. Consider a result from a DBI query:

MyFields = result.fetch_fields.field_names_module { |field| field.name }

Still pretty easy, even for the complicated case. Enjoy!