Lesson learned from a performance scare

Last Wednesday I posted this on Twitter and App.net:

It’s taking almost two seconds to completely regenerate all content on donmelton.com using Magneto. I need to speed that up.

As I’ve mentioned before, Magneto is a static site generator. It’s a tool I wrote myself, and what I use to create all the HTML, CSS, JavaScript, etc. that you’re reading now. The entire website is generated from this online source.

Some might think I was humble-bragging about Magneto’s speed, but I only had 42 posts here at the time and just under two seconds to generate that seemed a bit slothful to me.

Of course, I didn’t do anything about it right away — there are days when I make Jeff Lebowski look vigorous — but I did think on the problem for awhile. And then I was distracted by pictures of cats and bacon on Reddit. You know how it is.

But two days later, John Gruber made these two posts on Twitter:

Stunning — not the typo, everyone makes those — but that he’s made over 18,989 posts to his website. That is truly impressive. Of course, he’s been doing it for over 10 years so you would expect him to have written maybe one or two.

Rather than make the “Finally!” joke like everyone else about him fixing the typo, I replied to John asking, “How long does it take to regenerate the site with Movable Type?” And he responded, “Good question. It’s been years since I’ve done a site-wide rebuild.”

And that… Got. Me. Thinking.

A site-wide rebuild? I do that here every time I make a single change. It’s just easier that way — no dependencies to calculate. And my website is still small so it makes sense, even if it’s taking almost two seconds right now. If Gruber doesn’t want to do a rebuild with Movable Type, maybe this is a bad approach. How long would it take Magneto to generate a website with 18,989 posts? Hmmm, how long?

So, I put down my White Russian, hung up my bathrobe, and got to work figuring that out.

First thing to do was fake up a shitload of posts. My usual approach to problems like this is a single and convoluted line of shell and sed script, typed and executed directly on the command line in Terminal.app. It’s gruesome, but I’m a hack and that gets the job done.

My not-a-real-script started copying my existing posts and then replacing their published timestamps. A minute into the process and I realized my outside loop counter was wrong and I would wind up with more that 20,000 posts. Oh well, it would just be a better test. And Gruber’s total number of posts was probably closer to that anyway.

Then life interrupted. Not just more pictures of cats and bacon this time, but actual family interaction. For several hours, too.

When I managed to get back to my precious iMac, it was late thirty and I was tired. I kicked off Magneto and started it chugging on 20,394 posts, along with all of the other content already included in the website.

And I waited. Five minutes into staring at Terminal.app I remembered there’s no way to tell which post Magneto was processing because everything happens in memory before it writes anything out to disk. This isn’t as stupid, wasteful, and uncommunicative as it sounds. That behavior was, in fact, by design.

I yawned and thought, screw it, this is taking too long. Something’s wrong, but I’ll just let it keep running and check the results in the morning. Off to bed.

Saturday morning.

Made my coffee and started reading the news. Halfway into my usual RSS feeds when the caffeine finally loosened the knot in my frontal lobes — I remembered. Say, what about that test I was running last night?

I switched over to Terminal.app. Magneto was done. Oh, how nice. But how long did that take? I looked at the console spew. Hmmm, 24,415.52 seconds — a lot. Lesse, that’s… trying to do the math in my head… switching to Calculator.app because I can’t count that high… that’s … MOTHER OF GOD?!? That’s 6 hours and 47 minutes! What the hell was Magneto doing?

I thought, there goes my performance bona fides. My reputation on Safari is shot. They’ll ask for my union card back.

So, I set to work figuring out what went wrong.

The first task was to measure. Folks online pointed me at ruby-prof, a great little code profiler for Ruby. I reduced the number of posts in the test to a thousand or so since I couldn’t stand to iterate all weekend on the problem.

And then it was lather, rinse and repeat with ruby-prof, while also adjusting the number of posts in the test.

It turns out, the problem was not with Magneto. I made an even stupider mistake — I trusted the tool to protect me and didn’t pay attention to how I was misusing it.

Now, Magneto is very simple. It’s devoid of features, really. It’s not even “blog aware” like some other systems. It requires you to write a site controller script, plugins and templates to tell it exactly how to generate blog posts, index pages, RSS feeds, etc. And you get to make all the policy — that’s its power.

When I wrote Magneto, I generated a thousand pages with Magneto before I released it just to make sure it didn’t have any glaring performance issues. It scaled well. Of course, I tested other things, too. And it was plenty speedy with a simple site controller script. A. Simple. Script.

However, I was much more cavalier with my own site controller script, plugins and templates that I wrote to use here. And I made the cardinal sin of never measuring the performance implications of that sloppy-ass code.

I was right when I thought almost two seconds was too long to generate 42 posts. That’s about .03 seconds per post. I thought I could do better on my hefty iMac.

But taking 6 hours and 47 minutes to generate 20,394 means it was taking 1.2 seconds per post. That’s not linear — not even close.

Thank god for ruby-prof. It took awhile, but I was able to zero in on the problems. Yes, it was more than one. But they were all the same issue — not caching the unchanging result of a slow subroutine that was repeated in every post.

All these rookie mistakes were in one of my plugins and one of my templates. None of it in Magneto itself. And after some judicious use of static variables and other grotesque hacks, I was able to improve things a tiny, tiny bit.

Now it takes only 5 minutes and 19 seconds to generate 20,394 posts. That’s a much more linear .016 seconds per post. And a 76.576x improvement overall — that’s not a percent improvement, that’s 76.576 times faster.

Yes, I was that stupid.

I’m sure I can find more improvements, but it’s unlikely that I’ll be able to generate over twenty thousand posts in less than 10 seconds without a massive upgrade in hardware. And 10 seconds is about the most I ever want to wait for something like that. Right now, 10 seconds would be about one thousand posts.

No, if I ever approach the proficiency and productivity of John Gruber, then for daily use I’ll need to extend Magneto or my site controller script to only rebuild what’s changed and cache what hasn’t.

In the meantime, I’ll be measuring more up front. Maybe I’ll even write about how I made a similar mistake with Safari and the other lessons we all learned from that.

Obama’s fighting words

Conservative writer David Frum on President Barack Obama’s second inaugural address:

In the past, President Obama has spoken of the divisions between Republicans and Democrats as fundamentally unreal, subject to compromise and reconciliation by leaders of goodwill. Not this time. This time, he called his opponents out. Yes, many of them deserve it. Yet that does not make his words any less aggressive. Quite the contrary, actually:

We cannot mistake absolutism for principle, or substitute spectacle for politics, or treat name-calling as reasoned debate.

Those are not words intended to invite Republican cooperation, but to slam Republican non-cooperation; not to conciliate, but to confront. They were fighting words, and they portend a second term in which the president is fully as willing to take the fight to his opponents as they have been to take the fight to him.

Frum is at least more reasoned in his reaction compared to the faux indignation of other Republicans. Like schoolyard bullies, they just can’t believe the skinny kid wants his lunch money back now.

I don’t think President Obama is picking a fight. But it’s clear he’s not going to back away from one anymore.

15 things Kurt Vonnegut said better than anyone else ever has or will

During a conversation on App.net today, Dalton Caldwell linked to this article to source a snippet from Kurt Vonnegut he had just posted.

Today isn’t the anniversary of anything significant in Vonnegut’s life. At least, not that I know of. And this article dates back to 2007, the year we were all robbed of his acerbic wit. But I never saw it before.

And I’m fascinated by the selection and exegesis of each of these quotations. My favorite of the fifteen is this koan from “Cat’s Cradle” regarding the curse of sentience:

Tiger got to hunt, bird got to fly; Man got to sit and wonder, “Why, why, why?” Tiger got to sleep, bird got to land; Man got to tell himself he understand.

Dalton Caldwell liked that one too. So much so, it was the very first post he made — and only the third anyone ever made — to App.net. Good start, sir.

So it goes.

Space station commander gives us all a tour

In her final days as Commander of the International Space Station, Sunita Williams of NASA recorded an extensive tour of the orbital laboratory and downlinked the video on Nov. 18, just hours before she, cosmonaut Yuri Malenchenko and Flight Engineer Aki Hoshide of the Japan Aerospace Exploration Agency departed in their Soyuz TMA-05M spacecraft for a landing on the steppe of Kazakhstan. The tour includes scenes of each of the station’s modules and research facilities with a running narrative by Williams of the work that has taken place and which is ongoing aboard the orbital outpost.

After viewing this, I feel compelled to paraphrase the tagline from “Superman,” “You’ll believe a woman can fly.” And I do. She has the best job ever — superhero.

I can’t recall a longer look at non-fictional humans moving through a microgravity environment. Can you? Even with my Internet-depleted attention span, I couldn’t look away from this 25-minute video — watched the whole thing in one sitting. I bet you will too.

Via Andrew Sullivan.

Woody Allen on hypochondria

Differentiating himself from a common hypochondriac, Woody Allen writes:

What I am is an alarmist, which is in the same ballpark as the hypochondriac or, should I say, the same emergency room. Still there is a fundamental difference. I don’t experience imaginary maladies — my maladies are real.

You can’t help but hear his voice in your head as your read the essay. It’s old schtick but it’s classic schtick. Good to see him writing something besides screenplays these days. More, please.

Via Mark Evanier.

Update on Penny

Based on questions I keep getting on Twitter, App.net and elsewhere, let me update you on Penny, our Labrador Retriever, who stopped eating recently. Thank you all for your concern and my apologies for not posting this sooner. Penny has fully recovered. She’s no longer on an appetite stimulant and is eating normally — which means everything she can find. And we hope she keeps doing that for a long time.

A record hang gliding ride on the Texas wind

A neat mixed media presentation by The New York Times to clearly explain what Jonny Durand and Dustin Martin did and how they did it. 1 Of course, that was 475 miles and they still didn’t make it out of Texas.

Made me wonder how long it’s been since I took my first flight — a powered one — in the aborted pursuit of a private pilot license. Had to consult my log book. Turns out it was six years ago — yesterday. D’oh! Didn’t even celebrate.

What I should really do is haul my ass back to the airport and into that little Citabria again to finish getting my ticket.

Notes:

  1. Another reason print is dead. Or dying. You know what I mean.

Suicide reporting on the Internet

Charlie Lloyd on the response to the news of Aaron Swartz’s suicide:

You can tell people who’ve been near suicide before from those who haven’t. The ones for whom this is new are fitting it to a narrative. It’s the compassionate genius who was a little too good, or the activist hounded down by the government, or why would such a promising and beloved young person do something like this, or gosh there seems to be a link between creativity and mental illness, or some other well-meaning script.

Those of us for whom this brings back memories are, I think, a little less eager to see it as something that can be usefully explained, at least not by us.

There’s a collective sadness and anger on the Internet today. I have no idea if Aaron’s treatment by our government here in the United States provoked his action. Others can sort that out.

I did not know the man, but I did know his work — not just the breadth and depth of technologies he created and influenced, but his writing and activism.

Until I read Charlie’s article, I didn’t understand my own melancholy today. Years ago, one of my co-workers and friends took his own life. At that time, I wanted to punish myself for not seeing that act coming, and then for not understanding it after the fact. The thing is, neither reaction helps. Our responsibility is to help make sure it never happens to anyone else again.

Via too many folks on Twitter and App.net for me to remember where I saw this first.

Safari is released to the world

During the early development of Safari, I didn’t just worry about leaking our secret project through Apple’s IP address or our browser’s user agent string. It also concerned me that curious gawkers on the outside would notice who I was hiring at Apple.

Other than a bit part in a documentary about Netscape that aired on PBS, I wasn’t known to anyone but a few dozen other geeks in The Valley. Of course, several of those folks were aware I was now at Apple and working on some project I wouldn’t say anything about. And it doesn’t take many people in this town to snowball a bit of idle speculation.

I found out later that Andy Hertzfeld, an Apple veteran who I worked with at Eazel, had figured it all out by the time I showed up for my first day to work on the browser on June 25, 2001. Andy was very insightful that way. But thankfully he was also quiet about it at the time.

Hiring Darin Adler, also ex-Apple and ex-Eazel, in the Spring of 2002 was likely visible to others in the industry since he was much more well known than me. But because Darin had never worked on a dedicated Web browser like I had, no one made the connection.

However, when I hired Dave Hyatt in July 2002, then guesses started flying fast.

While at Netscape, Dave built the Chimera (now known as Camino) browser for Mac OS X and co-created the project that would later become Firefox. Both of these applications were based on the Mozilla Gecko layout engine on which Dave also worked. He was a true celebrity in the Web browser world, having his hands in just about every Mozilla project.

So, during the Summer of 2002, several bloggers and tech websites speculated that Dave must be bringing Chimera to the Mac. Except that Chimera was already a Mac application and didn’t need to be ported. So what the hell was Dave doing at Apple? Building another Gecko-based Mac browser? No one knew. And none of this made much sense. Which is probably why the rumors subsided so quickly.

But people would remember all of this when Safari debuted at Macworld in San Francisco on January 7, 2003. And at least one of them would remember it at full volume while Steve Jobs was on stage making that announcement.


Until I watched that video I found and posted of the Macworld keynote, I had completely forgotten what else was announced that day. Which is pretty sad considering I saw Steve rehearse the whole thing at least four times.

But you have to realize I was totally focused on Safari. And Scott Forstall, my boss, wanted me at those rehearsals in case something went wrong with it.

There’s nothing that can fill your underwear faster than seeing your product fail during a Steve Jobs demo.

One of my concerns at the time was network reliability. So, I brought Ken Kocienda, the first Safari engineer, with me to troubleshoot since he wrote so much of our networking code. If necessary, Ken could also diagnose and duct tape any other part of Safari too. He coined one of our team aphorisms, “If it doesn’t fit, you’re not shoving hard enough.”

Ken and I started at Apple on the same day so, technically, he’s the only original Safari team member I didn’t hire. But because we both worked at Eazel together, I knew that Ken was a world-class propellor-head and insisted Forstall assign him to my team — essentially a requirement for me taking the job.

Most of the time during those rehearsals, Ken and I had nothing to do except sit in the then empty audience and watch The Master Presenter at work — crafting his keynote. What a privilege to be a spectator during that process. At Apple, we were actually all students, not just spectators. When I see other companies clumsily announce products these days, I realize again how much the rest of the world lost now that Steve is gone.

At one rehearsal, Safari hung during Steve’s demo — unable to load any content. Before my pants could load any of its own, Ken discovered the entire network connection had failed. Nothing we could do. The IT folks fixed the problem quickly and set up a redundant system. But I still worried that it might happen again when it really mattered.


On the day of actual keynote, only a few of us from the Safari team were in the audience. Employee passes are always limited at these events for obvious reasons. But we did have great seats, just a few rows from the front — you didn’t want to be too close in case something really went wrong.

Steve started the Safari presentation with, “So, buckle up.” And that’s what I wished I could do then — seatbelt myself down. Then he defined one of our product goals as, “Speed. Speed.” So, I tensed up. Not that I didn’t agree, of course. I just knew what was coming soon:

Demo time.

And for the entire six minutes and 32 seconds that Steve used Safari on stage, I don’t remember taking a single breath. I was thinking about that network failure during rehearsal and screaming inside my head, “Stay online, stay online!” We only had one chance to make a first impression.

Of course, Steve, Safari and the network performed flawlessly. I shouldn’t have worried.

Then it was back to slides and Steve talking about how we built it. “We based Safari on an HTML rendering engine that is open source.” And right then is when everybody else remembered all those rumors from the Summer about Dave Hyatt bringing Chimera to Apple.

But I chose the engine we used — with my team’s and my management chain’s support, of course — a year before Dave joined the project. Dave thought it was a great decision too, once he arrived. But that engine wasn’t Gecko, the code inside Chimera.

It was KHTML. Specifically KHTML and KJS — the code inside KDE’s Konqueror Web browser on Linux. After the keynote was over, I sent this email to the KDE team to thank them and introduce ourselves. I did it right from where I was sitting too, once they turned the WiFi back on.

You can argue whether KHTML was the right decision — go ahead, after 10 years it doesn’t faze me anymore. I’ll detail my reasons in a later post. Spoiler alert: I don’t hate Gecko.

But back to Steve’s presentation.

Everyone was clapping that Apple embraced open source. Happy, happy, happy. And they were just certain what was coming next. Then Steve moved a new slide onto the screen. With only one word, “KHTML” — six-foot-high white letters on a blue background.

If you listen to that video I posted, notice that no one applauds here. Why? I’m guessing confusion and complete lack of recognition.

What you also can’t hear on the video is someone about 15 to 20 rows behind where we were sitting — obviously expecting the word “Gecko” up there — shout at what seemed like the top of his lungs:

“WHAT THE FUCK!?”

KHTML may have been a bigger surprise than Apple doing a browser at all. And that moment was glorious. We had punk’d the entire crowd.

Older dogs and newer tricks

Gary Marcus on the widespread view called the “critical-period effect”:

The critical-period effect is the idea that you can’t do certain things — like learn a language, or learn an instrument — unless you start early in life. It’s a discouraging thought for anyone past adolescence. But, recently, the evidence for this idea had started to unwind.

He goes to detail some of the new evidence against that idea and his own adventures in learning guitar at the age of forty.

I’ve always believed — not just hoped — that you can continue learning new things when you’re an adult. Not just information and facts, but real skills — behaviors that are normally envied as talents.

I know this is true because I’ve been doing it my whole adult life. And I plan on learning new things until I run out of air to breathe. I’m not saying it’s easy when you’re my age. But don’t tell me it’s impossible.

Via Andrew Sullivan.