To help me remember what I’ve been doing all these years, a few stories from my almost two decades of DAM and Web development. (Work in progress.) See also: My (honest) résumé
The work on “my” DC-X installation at the WESER-KURIER that had begun the previous year took most of 2010. One colleague did most of the data migration, while the other developers helped me when I needed a DC-X feature that wasn’t finished yet. The product and the project evolved together in a pretty agile way. I did most of the project management and of the customization: Configuring search and edit forms, developing importers for newspaper pages and ads coming from ppi PlanPag (very complex), for articles and images coming from the Atex Hermes editorial system, and for news agency texts coming from com.box DAKO – and exporters from DC-X to Hermes, the MSP ePaper and SAP. For the first time, I took a front seat at a big go-live, which included training users, spending nights and weekends hotfixing bugs and adding features. Exciting and rewarding, and very exhausting.
I’m not in sales so I don’t usually dine out with our customers. But this year, two of my favorite customers invited us out to celebrate successfully-implemented projects. Work is so much more fun when customers turn into partners, or even friends!
In my spare time, I prototyped an alternate, iPhone-friendly user interface for DC-X. (Its first incarnation was simply a rip-off of Google’s search page.) It would take a few more years for it to turn into an official product.
In 2009, the first pilot customer went live with DC-X. Because some features were still missing, we decided to let each developer take care of one DC-X project so we could learn what was needed the most, and implement it quickly. That way I became project manager for the introduction of the DC-X DAM system at a large German local newspaper, the WESER-KURIER in Bremen. They were migrating from our old DC4 product, so we spent a lot of time moving legacy data over and cleaning it up. Together with the hardware vendor we had recommended using IBM GPFS for sharing storage across the server cluster. GPFS was great when it worked, but it blew up so many times that the project and our reputation took a serious hit. (IBM fixed some bugs, and after maybe a year the GPFS was finally stable.)
I designed and implemented the DC-X rights management functionality, and metadata structures for tracking asset usage in online and print publications (borrowing from the PRISM standard). DC-X needed a Web service API, so I took what I had learned about RESTful APIs and implemented the Atom Publishing Protocol (AtomPub). And I taught myself Adobe’s Flex and Action Script because we considered creating a DAM UI with Flex. To gain some experience, I built the DC-X administration UI in Flex, and we made a small file upload tool for the Adobe AIR runtime. (We learnt from these projects that we’d rather stick to HTML.)
We switched our version control from CVS over to Subversion. I set up a public blog for the company, and a Confluence Wiki instance for documentation. And I joined Twitter as @tistre!
I spent much of the year working on a highly-customized DC5 project I had taken over from former colleagues. They had added metadata structures for food photography (recipes and ingredients), and complex rights management rules to prevent competing magazines from reusing an image within a certain timeframe. Something seemed wrong with the implementation of the rules, but neither the customer nor the developer could pin this down since the rules were so complex. That was the first time I wrote unit tests – they helped me spot and fix several errors in our implementation.
It would take a while for DC-X to become a real product. In 2008, it was still in the prototyping phase. Together with two Web designers, we built two different UIs. But with the IFRA Expo deadline fast approaching (the boss wanted to demo DC-X there), we completely dismissed the frontend prototypes. In just three weeks before the Expo, the designer and I built a new DAM UI using the YUI framework. This wasn’t fun; I almost burnt out from that pace and pressure. As a “thank you”, the company got me my first iPhone and flew me to the IFRA Expo in Amsterdam.
Ironically, just that summer our boss had organized a Scrum training with an excellent it-agile coach to help us getting started with agile development (which has a “sustainable pace” principle). I loved the “agile” approach, even though it remained difficult to find the best way to implement it. I became the moderator of our “retrospective” meetings. And I was invited to join the company’s management team “Jour Fixe” (this lasted for about two years, even though I wasn’t a manager) and went on a Denmark retreat with them.
Only four years after we had started to build DC5, we were already making plans to replace it with a new DAM system. Not only did performance problems with Oracle’s fulltext search force us to look for alternatives, but we also struggled with a design decision we had made for DC5: Our core development department had lacked the capacity to fulfill all customers’ feature requests. Most of our project managers were able to write PHP code. So we had split the application into a standardized backend and a sample frontend that was copied and customized by the project manager for each customer. Everyone was happy, until we figured out that bug fixes or new features in our sample frontend would have to be manually patched into every customer’s system, every time. This certainly didn’t scale.
So our new product DC-X (the obvious name “DC6” was rejected for some reason) would have a standard UI that we’d be able to update no matter how the project manager configured it. It would be a single page Web application with “Ajax” and drag and drop. Other good stuff that would appear in DC-X: Solr fulltext search, MySQL and Oracle support, and a sophisticated workflow engine for scalable ingestion and export processes. Brainstorming and bootstrapping this new product, together with (among others) my colleague André Widhani and our boss Thorsten Mann, was exciting.
In early 2007, I switched from Windows to the Mac and never looked back. I started an internal blog on the company intranet (and had to learn that very few people like to write). The books I read that year, RESTful Web Services and The Pragmatic Programmer, influenced the design of DC-X.
2006 was a chaotic year: Several people left the company while we had to handle three large DC5 projects in parallel. We missed most deadlines that year. But we still built some nice things, like support for multilingual content for a Swiss publisher and the integration of a geographical thesaurus into search and edit forms.
We started to use the MantisBT bug tracking software. And because of performance problems with Oracle Text, we evaluated the FAST search engine (turned out it wasn’t the right tool for us).
Then my boss left the company. And I may have made a huge mistake: I declined the offer to co-lead the development team because I wanted to code, not manage.
A large German magazine publisher became a DC5 customer, but needed a native application for viewing huge amounts of photos. Having failed trying to build one ourselves in 2002, we found a partner company to develop a Java application talking to our DC5 backend. I had read a lot about REST Web services and created a more or less “RESTful” API for the communication with the Java client. For the same customer, I developed a two-way interface between our DAM and the Picturemaxx opengate (APIS) network for federated image search.
I started providing VMware machines with our DAM system pre-installed for colleagues, saving them time setting up demo servers. And we moved our development from PHP 4 to PHP 5.
We spent most of 2004 building the first version of our new DC5 DAM product, learning a lot about internationalization (with the help of CLDR) and PostgreSQL along the way. We started using APD for profiling PHP performance. I discovered Nagios and introduced it to the company (but it took us a while to adopt). The pilot customer Fairfax Media in Australia went live with a pre-release version of DC5.
In 2003 I worked on one of my favorite projects ever: Australian publisher Fairfax Media (Sydney Morning Herald, THE AGE) ordered a Web application to sell prints of their amazing photos to readers, and to license photos out to business customers. With the help of my boss and a designer, I built a completely new UI on top of our DC4 DAM backend – including e-commerce functionality (shopping cart, GST and shipping cost calculation, interfaces to fulfilment, credit card company and billing) and a custom administration UI for their staff. We flew over to Sydney for two weeks, and the Fairfax Photos site went live later that year. (The current site is not powered by our software anymore.) This was my first encounter with SOAP, e-commerce, and a large public Web site.
But our DC4 product was growing old, and there were so many areas we wanted to improve on that we started to develop its successor, the DC5 DAM system, as a full rewrite. Sadly, the geniuses had left the company. But it was great fun to, for the first time in my career, start a new product from scratch, together with my passionate and inspiring boss André Basse. My wonderful colleague Thomas Wülfken and I did large parts of the software architecture, and much of the implementation. We moved from a single-page Web app to a simpler page-based UI to allow for more branding and customization. The PHP code featured much nicer OOP, a clean backend/frontend separation, a pluggable importer toolchain, and a large collection of command line PHP scripts. I learnt a lot of new stuff: XSLT (I loved Doug Tidwell’s book), LDAP, Unicode / UTF-8, REST Web services, and Topic Maps.
My private work notes turned into a public blog: Tim’s Weblog, running on custom blogging software based on my XDD framework. More than a decade later, it’s still very much alive and I don’t want to miss it!
That same year, I published a tiny but relatively successful open source project: OracleEditor.php, “a standalone PHP script which allows you to browse your Oracle database tables and insert, update and delete rows in any table.” It received a lot of positive feedback on it, a listing at the top of oracle.com’s open source page, and even a chapter in a book. My boss and published a Gallery extension for reading embedded IPTC metadata.
Though we were sure that Web based DAM software was the way to go, photo editors frequently complained about a lack of speed and features compared to competitors’ native applications when handling lots of images. So we spent roughly two man-years in 2002 trying to develop a native Windows client for DC4 in Borland Delphi. It was a disaster (probably because all of us were new to Delphi); we moved way too slowly and finally cancelled the effort.
I became a bit bored with the old DC4 PHP code and started a few personal hobby projects to try out new PHP features, Smarty templates, and PostgreSQL. One project was a flexible layer for structured data on top of PostgreSQL which I called “XDD”, the “XML driven database”. Using XDD, I built a public database of German hymns which is still up and running (70,000 monthly visits the last time I looked). And I published WMLStart.php as a first attempt at open source, “a compact, understandable, yet fully-working sample WAP/WML application contained in a single PHP script”.
DC4 was rolled out to many more customers in 2001 – most of them overseas. Large newspaper publishers in Thailand and Malaysia started using it, and USA Today’s DC4 was put into production by GMTI whom I had visited the year before. I wrote custom code for these systems. For a Norwegian newspaper, I built an importer for archiving published pages, articles and images from their CCI editorial system. And we optimized the product, learning a lot about caching and Web application security.
I started a mailing list to keep our international distributors (in Scandinavia, Asia, Spain, Turkey) updated about new features. And I was blown away by VMware, which let me run a virtual Linux server on my Windows PC.
In the year 2000, the new DC4 UI was becoming a relatively stable product, so work shifted towards helping customers getting it installed and adapted to their needs. I travelled to Cincinnatti, USA to train our partner company GMTI. We flew to Barcelona for a requirements gathering workshop at a large Spanish newspaper. And I took over the customizations for a large German publisher of TV programming magazines. My code helped them search the DAM and download images from within Quark XPress, and I built a bridge between the pictures within their DAM and their TV programme database. ICC profiles and the CMYK color space were new to me.
In product development, we removed a few features to create a “DC4 Lite” product that almost no-one bought. (It was cancelled soon after.) To help design custom forms and views, I created DC4 PHP components that could be placed on HTML pages and configured from within Dreamweaver. Very cool, but few people had Dreamweaver installed… We started to use PHP 4, and struggled with Internet Explorer 5 on Mac, the worst Web browser ever. A highlight was the development team’s visit to the German PHP conference “PHP Kongress”.
There was another genius at Digital Collections: Co-founder Dennis Zierahn taught me proper Unix-style programming, using recursion, stacks, small command line tools, inter-process communication. (DC4 ingestion and other backend processes were handled by command line PHP scripts running as daemons.) And he handed over to me the alternative DC4 Web UI he had built in HTDL, which I ported to PHP. I wrote the documentation for this new product and helped colleagues and partners to get started with it.
Ghostscript and Image Magick, IPTC IIM metadata and IPTC 7901 text were some of the things I had to learn about. But I totally fell in love with XML, using it in more places than I probably should have…
Aside from core product development, I helped out with a few customer installations, like a newspaper archive in the Netherlands and a system the German news agency dpa set up just for the 1999 G8 summit in Cologne. Editing PHP files on the live production server (remotely over an ISDN line) was a very special thrill.
In November 1998, I became an employee of Digital Collections, who were happy to hire someone who already knew some of their technology. But they had moved from C and HTDL programming to PHP, which I had to learn from scratch. Thies C. Arntzen, one of the company founders, was an amazing teacher and mentor – a PHP core developer, genius, and very fun to work with. He taught me not just PHP, but good programming in general: Coding style, DRY, clean object-oriented programming (yes, PHP 3 did OOP), separation of concerns, reproducible test cases.
We were adding features to our DC4 DAM system, a single-page Web application (lots of frames and document.write). DC4 was used by several large publishing houses for their newspaper/magazine archives, news wire feeds and image databases. It ran on the Oracle database, the Apache Web server and Unix (Solaris, HPUX, AIX, Irix, Linux), so I learnt something about all those as well. The “vi” editor was pretty scary at first, I must admit.
Still a student (busy writing my diploma thesis [PDF]), I set up my first public Web site (in handwritten HTML). Even though it consisted of nothing but a few links and some travel photos, it helped me land my first job the next year.
As LIS students, we attended dBase and Turbo Pascal courses. I really liked to write code. But what I loved the most was the Microsoft Access 2.0 database programming we did for an architectural bureau – learning Access from scratch, over the course of several months, two fellow students and I wrote a real-world application for a real client and even got paid for it. During the following years, I kept doing MS Access work for that client.
As part of my Library and Information Sciences (LIS) education, I did a six-month internship at the text archives department of large German magazine publishing house Gruner+Jahr in Hamburg. It was amazing to experience the librarian side of sophisticated, large-scale Digital Asset Management – see my blog post “Where have all the librarians gone?”.