Cold Hard Code

No One Trick Ponies - N(ot)O(nly)SQL

When any new fancy technology comes into existence, there is a desire to jump on it and use it to its fullest potential. Document databases are now reaching a level of maturity that any ol' developer can set them up, use them and have success. This is a good thing.

What isn't good is when that becomes the only tool you are using. All too often developers decide that any given new tool is exactly the only tool they need and proceed accordingly. NoSQL is another example of this.

When we built Jarvis, our new asset manager, we had already built other products to test some various ideas surrounding document-databases and implementation details. The real result is that we figured out that it's really quite trivial and still efficient to split between a traditional RDBMS and a Document database. In Jarvis, we do this by taking all the relational data out and storing it in the RDBMS (MySQL, in this case).

As we touched on briefly in our technical notes, we store the User and Organization relationship information in MySQL. The Organization itself is a document and is stored in Mongo.

The Organization table in MySQL is simply a record that tells us what to look at and has some other denormalized data that we use to keep things a bit faster. This data is highly relational, because there are full permission sets both on the role of a person in the organization and the person's individual permissions (so we have two levels).

This is good because we can traverse the relationships from any permission and see what people directly and roles, then people have that. Doing something like this in a document database is certainly possible but it just doesn't make sense. This is relational data at the core, so we're using a relational storage.

Through a more sensible model class design (instead of just assuming that whatever web framework we use is a good enough API) we have a standalone, useful API that can seamlessly cross between Mongo and MySQL without the application developer having to think much about it.

There are still some idioms based on what storage mechanism is in scope, but by and large it works very well.

This hybrid approach makes a lot of sense and the end result is very nice code, that's easy to follow and extend.  I think we have done a very good job of minimizing any technical debt on a project in its very first stages.

(hat tip to Bradford W. on the title for NoSQL)

Jarvis Development Process

We have a very simple method for quality control and deployment that is working well with us and scaling nicely. We're using this with Jarvis, our new asset management service. Here is a brief overview of how we do it, with a pretty picture to illustrate the general flow.

We work in topic branches, when ready for testing merge into a qa branch. After QA is complete, it gets merged into master.

We have git hooks in place that every commit to qa then relays and pushes to a series of configured hosts. Each host has a git repository configured to do what it should (update Solr configuration, restart the web application, etc.) This system has worked very well for us, and we just setup per-branch rules on what hosts to push to. This leverages the decentralization that git gives us and the ability to quickly rollback by just moving qa/master to a previous tag.

On that note, it's important to note that we use tags at all the appropriate points in this. We also are evaluating several continuous integration tools which will ultimately be the gatekeeper for what goes out the door.

This will just be an intermediary step where pushing into the master branch will fire off a CI process, and on successful build will release.

We will also have CI on the QA branch, which leads me into...

The Problem

One of our issues is we don't have a long-running development branch and waiting until we get changes into QA for CI to take hold defeats the entire purpose. We're trying to figure out how any topic branch can have CI automatically apply.

The easy and poor solution is having a post-update hook on the branches just set it up and auto-vivify the configuration. We'd like something just a little more formal and easier to track. Still evaluating options, but that's a sticking point we're trying to sort out now.

Jarvis Technical Notes

We've had some questions about how Jarvis is setup. We're happy to talk a bit about that, especially since we've leveraged so many open source projects.

First I'd like to point out that we proudly detail all the bits we use on Jarvis' About page. We'll do our best to keep that up to date as out technology and infrastructure grow.

Let's start off with a diagram:

nginx starts things off. It talks to Jarvis::Web, our Catalyst application. We use Log4perl to direct logging to a few different log files. Early on we decided to log lots of user activity so that we could analyze it later and learn how our customers are using the site.

We chose a hybrid approach for data. We store user and organization data in MySQL but asset information is in MongoDB. This lets us have a traditional schema for organizations having many users and users that have many organizations. We chose a document database for our asset information because the "schema" we had in mind would be a hot mess in a relational database. The freedom of a Mongo enables some really great features which we'll be unveiling over time.

One of those features is the ability to add sub-types to an asset. You can choose Leased or Purchased and add more data. We've used Moose and some of it's extensions — roles and serialization — to create a powerful system for customizing your assets.

Lastly we come to Solr. It gives us amazing search capabilities that power our datatables in addition to the conventional search. Let's not forget faceting, sorting and filtering!

Jarvis is built from components with a very strong pedigree. We've leveraged dozens of open source projects, multiple programming languages and a vast array of technologies. We look forward to adding more and making Jarvis even more awesome every week!

Simple and useful YUI3: Checkbox Toggling

I had to write a simple checkbox toggle mechanism, which is a very common pattern in a datatable. You have one checkbox, and all other checkboxes in the same pattern should be selected (or deselected) with it.  Nothing magic, there.

With YUI3's event delegation and Node API, I think this is the simplest mechanism I've built (and it only looks at elements in the containing form).

Thanks for an awesome JS lib, Yahoo!.

Jarvis: Stop Reacting and Start Planning

Jarvis is an asset management application. Asset management is a system for keeping track of the things you own, rent or lease. The current focus of Jarvis is management of servers, but we've got much bigger plans in mind. More on that later...

Today you can enter all your servers — along with a lot of data — into Jarvis. Got a lot of them? No problem! You can use our awesome search feature to find whatever you need. There is even a sidebar of search facets to further refine your search to get exactly what you want.

If you lease your servers we have some extra features. You can enter the dates and prices to stay on top of that as well!

I mentioned we had bigger plans. I can't reveal them all yet, but Jarvis will get smarter and smarter. Jarvis will ultimately be able to help you plan for the future rather than reacting. How about a heads up on an expiring lease, a notification of a system update or a hint about a new line of servers?

Sign up for Jarvis today and start entering your servers. Sign up for our newsletter so we can let you know about the new features we'll be adding every week. If you feel something is missing or have some ideas then give us some feedback.

We'll be writing more about Jarvis and pushing out new features as often as we can. Thanks!

My thoughts on the iPad

I've been a dedicated Mac user for about 2 years now. I purchased a MacBook Pro 3 years ago, and used it just for traveling. The turning point for me was struggling for about 20 minutes to connect to an Airport WiFi network, and while I was sitting there 3 different people with Macs sat down and immediately loaded webpages. That made me envious.

The other part is that I saw how quickly the Apple laptops wake up (and sleep). That also made me envious. I've happily trusted Apple to make a device that I want to use to accomplish the task they built the device for. This is an important distinguishing factor, because trying to use an Apple device for a use not intended is very frustrating. This is why I still use Linux servers, and can't imagine ever switching.

With this all in mind, when I looked at the iPad there was a moment of confusion. What is this device? What are the competitors? The UMPC segment isn't there, and the iPad is different than tablets. I honestly don't think there are competitors on the market now (JooJoo doesn't count), but there certainly will be (the HP tablet, etc).

The iPad is the natural evolution of the iPod. The very first generation iPod hit the market with an MSRP of $399. Not very far off of the $499 for the iPad, released 9 years later.

The iPad is pushing the limits of current technology, much like the iPod did back then (for those people questioning this statement, I dare you to look at iPod competitors back then.)

With technology at the point it is now, the current crop of personal media devices is going to be the future. Apple did not do this abruptly, they have been slowly removing and testing the waters. The AppleTV was the first device for media consumption, and it largely failed. The iMac and Mini were full computer that was largely non-servicable by the end-user. The MacBook Air was the closest shot, a closed device without a lot of ports. This was the perfect litmus test to see if people in general were willing to start giving up on things they didn't use.

This is where we are at. We have purpose-built devices that excel at what they're built at doing. In the case of the iPad, it's exactly that: a pad. Much like a pad of paper, this is the new generation. It isn't a computing device, nor should it ever attempt to be one. It will fail at that, just as much as the current "Tablets" fail at being competitors to the iPad.

You don't want background tasks on this. It's a single-task type of device. Push notifications and perhaps event loops are important, but on the iPad (or any other Pad device) you should be focusing on a single task. This is the expectation the device has of its user.

The technology is to be able to quickly flip between tasks. This is where Apple has a leg up on competitors and I am not sure if the competitors are going to truly understand that the quick-task switching is the killer feature. One which Apple does well with, but I still think they could do better.

So after sitting with my iPad for a couple days, both using it as a productivity enhancer and just a media consumption device, I still think they have a hit. it's exactly what it should be. It isn't perfect, but none of the first generation of devices are. I'll be buying the next generation when it comes out, and be very happy.

Pros:

1. Vivid screen. Very impressive, readable and perfect.
2. Orientation Lock is fantastic.
3. It works exactly as you would expect.

Cons

1. It's heavy. I'll build up forearm strength simply reading in bed (a pro?)
2. There is disparity between applications on which way is down. The case should resolve this. Certain applications have fixed orientation (usually games) and that "Down" direction may be counter to what is natural to you, and it doesn't rotate. As an example, Harbor Master's "Down" is contrary to the case.
3. Apps are expensive. More screen real estate does not automatically make something worth more. The app itself needs to be more. There isn't the "14 Day Free Trial" editions on the iPad, and because of that it precludes me from buying a lot of applications I'd like.
4. The App Store. It has too many limitations. No free trials, the iPad/iPhone split isn't clean enough.

Now, each and every one of those cons is temporary and I'm certain will be resolved within the next couple of years. I'll be on my 3rd iPad and probably my 5th iPhone by then, and enjoying every one.

An early project migration, from MogileFS to MongoDB

As vaguely announced, a few days ago we launched a new project called CodePeek.  While we have our own reasons for this particular project, it was really just a very quickly done project that, by default, was built using the tools that we had available to us and experience with.  One of those tools we picked simply because of familiarity was MogileFS.

MogileFS has always been my default choice whenever I needed to store a lot of data somewhere, without any relational context to that data.  It handles replication, and for the most part just works.  For things like pastes, with the support for images and other media types, it was a reasonable choice.

Except it is just a bit fragile and lacks certain features that we really wanted to have available to us.  I don't want this to be interpreted as being critical of MogileFS because I think it's good software.  However, I do think that it is software that requires more attention than what we wanted to give it and also is not an exact fit for our purposes.

This discussion was happening while we are also exploring various document-based storage engines (commonly referred to as "NoSQL").  This movement has produced CouchDB and, of course, MongoDB (as well as many others).  For us, MongoDB looked like the obvious solution because it supports replication, has an incredibly easy API and was very fast in our tests.

After about an hour of hacking and debugging last night, I had everything working.  I spent 20 minutes this morning writing a migration script and everything has just simply worked.

With a total investment of about 2 hours — including setup, "apt-get install mongodb" wasn't much trouble — we have replicated MongoDB storage (across 2 Linodes), and new storage code.  I'm pretty impressed with the agility in which we can move to replace a storage backend, and believe it is a testament to the quality inherently available when using superior tools, like Moose, Catalyst and DBIx::Class.

Designing Codepeek: The Logo

Much of my work to date with Cold Hard Code has been for relatively large applications.  These big, complex applications can quickly overwhelm so a lot of time is spent keeping things simple.  Our most recent project, Codepeek, gave me a chance to make something less intense.

The logo came together very fast.  Hoefler & Frere-Jone's Tungsten is CHC's corporate font so it's use was predetermined.  I had it in my head to create something playful, classy and more pop than my usual work.

From jump I knew that I wanted to set the text on an arc. After a few attempts I finally found what I wanted with a negative leading. This gave it a presence that reminded me of an announcer crowing to the crowd: Codepeek has arrived!

The two-tone effect is intentional, providing some clarification for the words.  The name is inspired by the concept of "peeking at code".  My original idea was "peekcode" but the k and c felt like a linguistic train wreck that refused to roll off the tongue.  Codepeek worked much better but was backward from my original concept.  Placing "peek" front and center in bright white balanced the message I was trying to send.  Careful not to forget the "code" I added a strong gradient and some subtle dots that are intended to look like bubbles rising in the code.  Those embellishments can be easily removed and the logo reduced to two colors very easily, fulfilling one of the rules of logo design.

The small leaves set behind the name are decorative fleurons that provide some distinction for an otherwise wordy logo.  It's a purely subjective decoration that I have an odd affinity for.

Codepeek was a great project that Jay and I took from concept to production in just over a week.  It gave me an opportunity to create a light, playful identity that I'm very proud of.