Importing Delicious Bookmarks Into Drupal, or the Cloud is Disappearing

"Many":http://www.readwriteweb.com/archives/deliciouss_data_policy_is_like_setting_a_museum_on.php have "written about":http://www.salon.com/technology/silicon_valley/index.html?story=/tech/dan_gillmor/2010/12/17/yahoo_shuttering_bookmarks_service the impending demise of "delicious":http://delicious.com, one of the first so-called "Web 2.0" companies, and one of the few cloud-based services that I actually use. delicious was useful because of its simplicity, with a pared-down interface, clean bookmarklets or browser plugins, and the ability to easily tag new bookmarks. I used to use it extensively to find new sites, but stopped doing that a few years ago. These days it was basically a way to store bookmarks in a place that was accessible anywhere I had an internet connection. But as we have expected for a while, and are witnessing at the moment, the cloud is disappearing, its solidity nothing but a mirage. We are in for trouble in the future, methinks, because so many institutions, "including universities":http://www.hastac.org/blogs/nknouf/wikileaks-broadcast-internet-and-importance-new-media-assemblages, have moved their online activities from local hosting to this mythical "cloud". The point of this post is not to go into the deeper issues here, but to explain, in some detail, how you can setup a delicious-like interface on your own server. I'm going to be using "Drupal":http://drupal.org, one of the most well-known free software content management systems (CMS). Doing this requires a certain amount of technical ability, and I won't be able to go into that here. But if you're interested, read on. h2. Creating the content type While there are Drupal modules for creating "weblinks":http://drupal.org/project/weblinks, in my brief testing I found them to be both too cumbersome, and too limited for what I wanted to do. In any event I wanted to see how easy it might be to do this using the tools Drupal provides. Turns out that we can easily create a Bookmarks type using CCK. The two additional fields I created are for the "URL" and a "Post Date" (so that we can ensure we have the correct dates for the bookmarks when we import our data below). The first is of type "text" and the second is of type "date". The exported CCK code can be found below. *Update*: By using the "Unique Field":http://drupal.org/project/unique_field module, we can require that the URL is unique, redirecting you to the existing bookmark if desired. This helps prevent posting of identical bookmarks, and might remind you to look through your bookmarks if you keep saving things that you've already saved...not that that just happened to me. h2. Taxonomies I decided to create a separate vocabulary for my bookmarks tags, and I thus edited this vocabulary to allow editing by the Bookmarks content type. Make a note of the vocabulary ID, as we will need this later. h2. Services To easily import our bookmarks, it's best to use Drupal's "Services":http://drupal.org/project/services module. Using services you can add nodes, get taxonomy items, and so on using XML-RPC or JSON. Setting up and configuring Services is beyond the scope of this post, but the documentation is actually fairly good. h2. Delicious import Next, we need to import our bookmarks. To begin, we need to _export_ our bookmarks from Delicious. The code to this, on Mac OS X or Linux, is: where <username> is your Delicious username and <password> is your Delicious password. Your bookmarks will then be downloaded and saved in the @bookmarks.xml@ file. We're now ready to parse this XML file and post it to our new Bookmarks content type. I wrote a dirty python script called @parseDelicious.py@ to do this; you can find the link to it at the bottom of the page. It requires the @lxml@ library for @etree@. The script is probably highly inefficient, especially in the parts that check whether or not existing tags are in your Drupal taxonomy. The script requires a number of options that are explained by calling it with the @-h@ switch. To run: Here <server> is the hostname of the Drupal server (*without* @/services/xmlrpc@), <username> is the username of a user on the Drupal server, <password> is your password, <vid> is the numeric vocabulary ID of the desired vocabulary for your Delicious tags, and <bookmarks xml> is the path to the XML file just downloaded. Depending on how many bookmarks you have, this might take a _long_ time. h2. Creating a View of your Bookmarks Next, let's create a simple view that will give us our bookmarks in a format that is close to that of Delicious. Here we use the Views module, as well as Semantic Views to allow us to style the resulting data. I've designed the view to make it easy to style in CSS. Basically the logic is that we pull the node title, the CCK post date, the CCK URL, the node body, and the taxonomy terms. We then link the node title to the CCK URL, create useful classes on each element, setup a pager, and set our CSS classes. The is at the bottom of the page. It is also possible to create a tag cloud of all our bookmark tags using the Tagadelic module. h2. Creating a bookmarklet for posting Finally, we can create a simple bookmarklet for posting to your site. To do this we need the "Prepopulate":http://drupal.org/project/prepopulate module. The bookmarklet code below is modified slightly from the example given in the documentation for the module. notextile. Simple change "example.com" to the name of your host. The names of the values are based off of the content type we created above. You can see the bookmarks that I've imported from Delicious, with minimal CSS formatting, "here":http://zeitkunst.org/bookmarks. h2. Where's the social? Of course this method means that we only host the bookmarks on our own site, thus losing all of the social capabilities of delicious. The challenge for the future, given the disappearance of the cloud, is how to store data locally, but be able to access it distributively. There is no reason that something like Delicious could not be implemented using a "distributed hash table":https://secure.wikimedia.org/wikipedia/en/wiki/Distributed_hash_table, thus preventing any one company from being a source of failure. It seems like it would be possible, then, to write a program that would store bookmarks as hashes in this table, and then build on top of that the necessary metadata such as tags, users, groups, etc. This would parallel the development of "thimbl":http://www.thimbl.net/, using as well public-key encryption to provide security. But this is a project for more than a single day; if you are interested, please let me know. h2. Code h3. Bookmarks Content Type notextile. h3. Bookmarks View notextile. h3. parseDelicious.py notextile.