Saturday, December 31, 2011

First impressions of the Zappos API

I thought of an idea for a web application when I was browsing the Zappos store recently, and was pleased to find out that external developers have access to a tremendous amount of information via the Zappos API.  I decided to jump right in and explore what it has to offer.

Signup was easy and the license agreement didn't contain anything I found terribly upsetting.  After signing up I created an API key.  I had a small issue generating a key initially, but Will Young of the API team at Zappos solved my problem literally five minutes after I sent an email requesting help.

I followed some of the examples in the documentation and had no problems as I poked around the API.  Reading unformatted JSON string responses got old pretty quickly so I installed the JSON formatter extension for Chrome and that made running ad hoc requests in the browser more pleasant.  After perusing the documentation and executing several exploratory requests I was sure that I had access to enough information to build my application, so I got to work on it right away.

My application is only interested in Zappos' shoe inventory.  Since Zappos sells a wide variety of products in addition to shoes, most of my searches have to use filters to narrow the scope appropriately.  It took some trial and error with the Search API before I fully understood "facets" and how they are used to describe products, but with that understanding I am now making a lot of headway.  Of the available APIs that I've used in my application so far, the Search one is by far the most powerful (and complex).  I use it to retrieve the distinct shoe sizes, shoe sub categories, and colors.  For these kinds of requests I exclude product results.  Once the user selects among those facets I then retrieve matching product results.

At first I considered writing my entire application without any server-side scripting, using JSONP to overcome the Same Origin Policy, but I eventually decided to route all my AJAX requests through a server-side script in order to keep my API key private and also to have more control over the requests.  During development I am finding the 2500 daily request limit more than sufficient, but when I make the application public I will need to implement a caching strategy in the server-side layer in order to keep the number of requests as low as possible.

On the server-side I use PHP/cURL to perform HTTP requests to the Zappos API.  I wrote a couple wrapper classes around requests to their Product and Search APIs in order to make URL/parameter generation easier.  Most of my application code will sit on the front-end as JavaScript/HTML/CSS, but I decided to write separate actions inside my server-side controller to field each request type.  This way I can hide some of the complexities (particularly of the Search API) from my JavaScript.  Eventually I think I would like to build some of this abstraction into a separate JavaScript library, but for the time being I'm keeping it in PHP so that I can reuse it if I need to execute non-XHR requests.  This is starting to sound like another perfect use case for Node.js...

I am happily using Backbone.js to organize my JavaScript on the front-end.  With it I have been able to rapidly assemble a functioning interface and try out my initial application design using real Zappos data sooner than I expected.  That has been quite helpful, and pushed me to refine several features and design elements that didn't mesh with the data as well as I originally imagined.  For example, some features of my application work perfectly under the assumption that there are always several matching products for each request the user performs.  As I tested various filter combinations I realized that it's easy to limit yourself to only one or two products in the less populated categories.

As I delve further into the APIs I realize that Zappos stores a lot more information about each of their products than I anticipated.  At the outset I thought it would be trivial to retrieve all the different categories of shoes.  But if you take a close look at their store you'll notice that there are many other ways to group shoes -- by brand, occasion, theme, materials, etc.  While it can be a little overwhelming, the availability of all this normalized information about the products opens up a lot of possibilities.

Saturday, December 17, 2011

Backbone.js is making client-side development fun (and maintainable)

I have been using Backbone.js to develop client-side web applications for a while now. I decided to write about my experiences in the hope that it will help others gauge whether it is something they should consider adding to their toolbox. This isn't a tutorial -- it's just my story as a web developer, and it describes some frustrations I had that Backbone.js smoothed over.

I had been using jQuery happily for a couple years when I came to the realization that something just wasn't right. I noticed that with each new JavaScript-enabled feature I added to a page, the difficulty in maintaining that page would rise disproportionately.

It wasn't problematic to add client-side sorting for a table, or form validation, or popups and tooltips here and there. But when I implemented Ajax-enabled CRUD or complex interactions things would quickly spiral out of control. Eventually a single page could have many different kinds of elements with primary keys from the database in their HTML id attributes. The HTML would become bloated and the line between styles and identifiers for use by my DOM selectors would get very blurry.

Meanwhile, my JavaScript code would be growing larger and larger without any kind of organization. Finding out where I set a particular event handler required me to wade through hundreds of lines of code. Browser consoles helped, but it's frustrating to routinely resort to setting breakpoints and stepping through code in the debugger just to comprehend what was happening where.

After repeating this scenario one too many times I realized (or at least hoped very strongly) that there had to be a better way. I looked for ways to reorganize my jQuery scripts, in the hope that it would at least solve that aspect of my troubles. What I discovered in my search is that I was going to have to finally learn the fundamentals of the JavaScript language before I could assess my options. With frameworks like jQuery it's fairly easy to learn the API and follow by example without truly understanding the underlying language itself.

I began relearning much of what I thought I knew about JavaScript. I started with the basics of the language and corrected a lot of my misunderstandings. Things that previously seemed mysterious to me about certain aspects of jQuery were explained. To this day I am still learning more about JavaScript/ECMAScript, and I feel that gaining a deep understanding of it is a worthwhile endeavor for anybody building web applications. Learning how prototypal inheritance works is the big hurdle, but it is key to unlocking the language's full potential.

As I learned more about JavaScript I applied techniques that helped me organize my code in ways that made it easier to read and maintain. I started using the module pattern and namespacing to organize my applications. I found these practices to be useful, but still far from my hope of a silver bullet. I kept experimenting with new organization techniques until a colleague of mine mentioned Backbone.js.

The Backbone.js site describes the project as follows:
Backbone supplies structure to JavaScript-heavy applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing application over a RESTful JSON interface.
That's quite a mouthful. It sounded powerful, but also quite complex. I was intrigued, though unsure of what exactly I was getting myself into. I read through the rest of the site, which includes samples and descriptions of each of its features. They are concise and pretty easy to understand, but afterwards I felt that I still didn't understand the big picture. I moved on to the official example TODO application but found it to be somewhat difficult to digest.

I read over several introductory tutorials until I had a solid understanding of the core concepts of Backbone.js and how they relate to each other. My frustration at that point was that each example seemed very clear in isolation but many of them varied in the way they handled similar problems. This confusion seemed to be a common theme for people who were learning the framework at the time.

It took me a while to fully accept the extreme flexibility of the framework. Only after I accepted that did I begin to feel comfortable during the learning process. Another reason the initial digestion period can be frustrating is that using Backbone offers several features but forces you to use none of them. You could forgo routers, models and collections, and only use views if you wished. Most of these features are decoupled and can be added in one-by-one.

Backbone.js effectively solved all the frustrations that drove me to find a better way to develop client-side web applications. I no longer store primary keys in my HTML id attributes. Instead I use Backbone's views and models together to handle record-level events and render the record's current state. This helps me keep my styles and application code almost entirely separate.

Where I used to write procedural JavaScript in huge blocks, it's now object-oriented and modular. The code is easier to read and more organized. The other benefit is that my code is more reusable. As my applications grow larger this benefit is amplified.

In general, client-side development is a lot less frustrating using Backbone.js. I spend much less time tracking down the location of bugs. If something is not working properly I can almost always immediately isolate the model or view that is responsible for the feature that is not working as expected.

Something you may miss when considering Backbone is the value of its helper library Underscore.js. During the process of learning the core concepts of JavaScript I realized that it has some rough edges. Those rough edges can make seemingly simple tasks rather verbose (when done correctly) and prone to error. Underscore.js provides a series of functions that can be used to smooth out these rough edges and speed up general JavaScript development. I find myself using these functions more and more over time, and would absolutely use Underscore.js by itself in any sizable JavaScript application even if it wasn't using Backbone.

I used to struggle to decide if my server-side code should respond to Ajax requests with JSON or with HTML. JSON was a cleaner exchange, but the tradeoff is that I had to do ugly and messy string concatenation in my JavaScript callback function.  With Underscore's template function I am able to have the best of both worlds -- JSON data exchanges and HTML templates. While JavaScript templates are not a brand new concept, it's one that I adopted when I started using Backbone.js and now I can't imagine doing it any other way.

If you are interested in Backbone.js but are having trouble understanding how it is different than traditional jQuery-heavy web applications, here's the way I would describe the difference: While your application will still be event-driven, its state will reside in JavaScript objects rather than in the HTML/DOM. You won't have to push/pull data from/to the DOM in order to retrieve and store state between events. When your data changes, affected portions of your display can be notified and will update independently to reflect the current state.

Following are links to resources I found helpful:

Sunday, February 27, 2011

How to untar and ungzip .tar.gz files on Windows

If you need to extract the contents of a tarred, gzipped file (*.tar.gz) and you use Windows, do yourself a favor and download Cygwin. Once you've installed Cygwin, run it and you'll get something that resembles a Windows command prompt. Be careful and only run commands you are confident of, because Cygwin is quite powerful and you could unintentionally modify or damage files if you are not careful.

By default the Cygwin prompt's current working directory should be something like C:/cygwin/home/<your windows username>/

You can either copy the tarred, gzipped file here to unzip and untar it, or you can navigate to its current location on your hard drive.

If you wish to navigate to the file's existing location on your hard drive you need to access it through the /cygdrive/<drive letter>/ directory. e.g. if the file is located in C:/Users/johndoe/Downloads, you would execute the following:

cd /cygdrive/c/Users/johndoe/Downloads

Once you are in the same directory as the file, just replace "MyFile.tar.gz" with your file's name and "output-directory" with a directory name of your choosing below, and run the following commands. The first will create a directory for the files and the second will unzip and untar them:
mkdir output-directory
tar -zxvf MyFile.tar.gz -C output-directory

I use Cygwin for a variety of other utilities, such as verifying file checksums using md5sum. grep and sed also very nice to have on a Windows machine. These utilities just scratch the surface of Cygwin's capabilities.

Thursday, November 11, 2010

Yii CRUD: "Please fix the following input errors: {MODEL} cannot be blank"

Today I generated a CRUD scaffold for my 'User' model using Yii Framework's GUI code generation tool, Gii. When I attempted to insert some data to my User table with the Gii-generated CRUD scaffolding I got a rather unhelpful error message:

Please fix the following input errors:
* User cannot be blank

It didn't name a specific property/column so I didn't immediately know what the problem was. When I took a look at the 'User' model I found that the 'user_id' column's label alias was 'User'. That prompted me to look at my 'User' table in the database where I discovered that I forgot to set the 'user_id' primary key column to AUTOINCREMENT. I set that column to AUTOINCREMENT but there were still issues in my 'User' model.

The Yii code generator analyzes the database table metadata during model creation, and because my primary key column was not set to AUTOINCREMENT when I generated it the first time it figured this was a field that accepts user input and must therefore be validated. That's why my 'User' model had my primary key column set to 'required' in its 'rules' function. This validation rule is why I got an unhelpful error message; if it wasn't validating the presence of the auto-incrementing primary key prior to record insertion (which it should not be), it would have gone ahead and attempted to insert the record and I would have received a more useful error from MySQL like the following:

SQLSTATE[HY000]: General error: 1364 Field 'user_id' doesn't have a default value

After I changed the 'user_id' column to AUTOINCREMENT in the database I then used Gii again to regenerate the 'User' model and this time it did not create a 'required' validation rule for the 'user_id' column as expected.

Sunday, November 7, 2010

MySQL ERROR 1061: Duplicate key name 'my_tbl_fk'

I was creating a foreign key in MySQL Workbench 5.2 OSS when I got the following error:

ERROR 1061: Duplicate key name 'tbl1_tbl2_fk'

I checked the "Foreign Keys" tab for the table to which I was adding the FK and there was no key with that name. In fact, there were no foreign keys with that name anywhere.

However, when I checked the indexes on that table I found the problem: an index with the same name as the foreign key I was creating. I dropped that index and was able to create the foreign key without any issues.

What happened was I had previously created a foreign key with that name and then dropped it on the 'Foreign Keys' tab. The index backing this key was not dropped when I did that, so when I attempted to recreate the foreign key with that name it tried to duplicate the index.