The official web-log of Matt Meshulam is what you are currently viewing.

24Aug/13Off

Archiving a website with Amazon S3

About five years ago, my friend Zach and I made the EnerJar, an open-source hardware design for a power meter. We created a website, enerjar.net, to document the design. We used the Drupal CMS framework so that we could easily update the content and allow users to comment on it.

After a few months, the site sat pretty much unchanged, but it was still backed by a database, PHP, and a bit of configuration on a shared hosting environment. After having to fix the site a few times after surprise upgrades and configuration changes, I decided to transition it to a static website where it could live reliably for the rest of time.

Static websites have many benefits over ones which rely on server-side scripting: they're cheaper and simpler to host, have virtually no risk of security breaches, and are much more scalable. Of course the downside is that users can't really interact with static websites, so they only work if you're okay with a simple read-only product.

Here's how I transitioned enerjar.net into a static site hosted on Amazon S3:

Step 0: prepare the site

You're effectively taking a picture of each page on your website, so any dynamic features will no longer work. This includes comments, searching (sometimes), and submitting new content. If you're transitioning a site that uses a framework like WordPress or Drupal, you'll want to disable any of these features in advance so that visitors don't even see them. Otherwise they get an error message when trying to actually submit a comment or something.

Step 1: download the site

From any Unix/Linux/OSX command line, you can use wget to recursively download all the resources on your site. I used the following command:

wget --recursive --page-requisites --html-extension --convert-links --domains enerjar.net http://enerjar.net/

"recursive" says to follow links on the page and continue to follow any links on the sub-pages. "page-requisites" tells wget to download not just the page itself, but resources like CSS and Javascript files. "domains" limits the recursive link-following to the enerjar.net domain, so it doesn't try to download external links.

I begrudgingly used "html-extension" and "convert-links" to rename every downloaded page file so that it ends with the .html extension. This could break outside links to your site, but it means you don't need to manually configure content types for these files on S3. If a page URL doesn't end with a file extension, for example, something.com/blog/entry1, wget will download it without an extension, to a file called entry1. The problem is that S3 doesn't know that entry1 is an html file, so when visiting the re-hosted page, your browser will download the file instead of displaying it as an actual page. By adding the .html extension on download we avoid this problem. Maybe someone knows a better way to do this, but I was lazy and this was good enough for me.

Step 2: Test the downloaded site

Open the HTML files that you downloaded with wget and make sure that everything shows up correctly and that links still work. For some reason wget didn't download some of the images on the site, so I had to manually get them and put them in the proper directories. Once everything looks good, you are ready for the fun parts.

Step 3: Create a new S3 bucket

If you don't already have one, open an Amazon Web Services account and create a new S3 bucket. Put it in whatever region you want, but the name of the bucket must be exactly the same as your root domain name. So I named my bucket "enerjar.net".

Step 3: Upload the site contents

After fiddling with various command-line tools for AWS, I gave up and used Amazon's Java uploader in the browser. You need to use the Java one to upload directories, otherwise you'll need to manually recreate the directory tree and upload files into each folder separately. Be sure to upload the site into the root directory of the bucket.

Step 4: configure static website hosting

In "properties" for the bucket, select Static website hosting->Enable website hosting. Set your root and error documents as necessary.

Step 5: Custom domain

Follow Amazon's tutorial for setting up DNS for hosting a static site with your own domain. The only snag I hit was that it didn't autocomplete the domain name for the alias. I thought it needed to be the domain of the actual S3 bucket (e.g. enerjar.net.s3-website-blahblah), but it actually just expects the root domain for the region where your bucket lives. In my case, s3-website-us-west-2.amazonaws.com.

Step 6: DNS and finishing up

As covered in the linked tutorial in step 5, you'll need to set up your domain to use Amazon's name servers. Make sure you also transfer any other DNS records (mailservers, subdomains, etc) to your Route 53 configuration.

That's it! You should now have a rock-solid, super-cheap hosting setup for your now-static website.

Filed under: Tech Comments Off
31Jul/13Off

A way-too-deep look at Ubuntu Edge’s Indiegogo numbers

TL;DR: Prospective Ubuntu Edge buyers are ridiculously sensitive to changes in pricing.

If you follow tech news you've probably heard quite a bit about Ubuntu's crowdfunding campaign, which is trying to raise $32 million to build a cutting-edge smartphone called the Ubuntu Edge. In the 9 days since the campaign began, there's been a lot of speculation on whether it will fall short of its all-or-nothing funding target. One site, ubuntu-edge.info, shows the amount raised over time:

ubuntu1

Judging from the red on-target line, they aren't looking too good. But that's boring. Let's take a closer look at the rate of the funding so far:

ubuntu2 The weird shape is at least partially due to Ubuntu's pricing scheme. They began the campaign with two price levels: a limited number of units for $600, and the rest being sold for $830. Then, after a few days, they introduced limited quantities available at lower price points: $625, $675, $725, $775, $780, etc. See here for the full distribution.

What struck me from looking at this graph is how clearly you can see the slope changing when a price level sold out. I speculated the story looked like this:

ubuntu3

Fortunately, we don't need to rely on our eyeballs when there's data involved; the dataset is available from the site above if you know where to look. I manually picked a single point around each of the kinks I observed and put them into Excel, since I'm not cool enough to use R for this kind of thing.

First I wanted to confirm that the funds raised in each time interval roughly equaled the corresponding price level*available quantity. It's not going to be exact since there are some other contribution levels (e.g. $20 just to support them, $10,000 if you are rich and like having Limited Edition things), but most of the money is coming from people just wanting to preorder a phone so this will get us in the ballpark.

ubuntu4

For each interval I drew on the graph above, I took the ending timestamp and cumulative amount raised (columns A and C) from ubuntu-edge.info. By subtracting each dollar amount from the previous row, I got the amount raised per interval (column D). Then I entered the cheapest price level at each time and the quantity available, assuming my hypothesis was correct. Note that we don't know how many suckers bought the $830 device before Ubuntu added lower price tiers, but they did state that the "hundreds" of them would get a refund for the difference.

For the rest of the time intervals, we can easily calculate the amount we expected to raise assuming all revenue came from the cheapest presale price level (column I). Then we can see the amount (column J) and percentage (column K) that this estimate differs from the actual amount. Our initial model is actually a really good fit; the first three intervals we have error %'s for are hovering at 5%. Without looking at the numbers, it feels about right that 5% of the revenue is coming from altruistic contributions.

Before going any further I want to point out that the math is actually a little fuzzier than it looks. There's nothing to stop someone from buying a more expensive phone when a cheaper level is still available, thus throwing off our Quantity column. Indeed we can see this is happening on the campaign site, since 34 phones have been bought for $830 even with the lower prices available. But we're just going for ballpark numbers here, and the ballpark looks about right.

Since intervals 1, 3, and 4 all have a deviation remarkably close to 5%, let's assume the same holds true for the time period when just the $830 phone was available. To get column K to result in a value of ~5%, there must have been about 740 suckers:

ubuntu5

 

But what's going on with our error during the $725 and $775 periods? We shoot up to 14% and then 25%. Did people suddenly start buying $10k phones? Did we miss the real points when the price levels sold out?

Actually, there's an explanation. When introducing the new price levels during the campaign, Ubuntu also added a "Double Edge" deal, where you can preorder two phones for $1400. Which works out to $700 apiece. Which just happens to be a worse deal than the $675 level but a better deal than the $725 level. So naturally, when the $675 phones sold out, the Double Edge started to get picked up. Let's keep calibrating our model to a 5% altruism rate and solve for the # of units sold as a double deal:

ubuntu6

 

Note that the equation for column I has changed to account for the new component.

Our model estimates that 414 phones (207 units) have been sold as doubles. When I grabbed this data, the actual total was 264 double packs. Not terribly far off.

Finally, let's look at the rate of unit sales during each interval to get a feel for price elasticity and the effect of time. It's pretty obvious that sales slowed down after the initial cheap phones ran out, and we can see a distinct slope change when each subsequent level was depleted, so we know there's some correlation with price. We also know that sales are probably going to be faster at the beginning when there's more hype and will slow down after that, at least until it gets very close to the end of the campaign.

ubuntu7

Column B is just each timestamp subtracted from the previous. Column H is what we're interested in: total unit sales (cheapest single price level+doubles) normalized over time.

I'm staggered by how much price affects the rate of sales. After the initial $600 level sold out, sales trickled in at just 6% of their previous velocity. When the price went from $625 to $675, only an 8% increase, the rate slowed by almost 60%. This corresponds to a crazy-high price elasticity of demand of 7.3. Any way you cut it, prospective Ubuntu Edge buyers are ridiculously price-sensitive.

What about the effect of time? Early on in the campaign a $830 price point yielded a sales rate of 23.6 units/hour. Later on, this rate was crossed again when the price went up from $725 to $775. Let's split the difference and speculate that if the price only increased to $750, the sales rate would have once again hit exactly 23.6.

This would mean that the passage of 3.5 days (from the start of the $830 interval to the start of the $775 interval) has about the same effect on sales as increasing the price by $80 (9.6%).

Of course it's hard to conclude anything from this limited set of data. But perhaps these observations can inspire future crowdfunding projects to experiment with different pricing schemes. Personally, I like the idea of surprise price decreases. You can recapture some of the natural time decay as the campaign goes on, and whoever does this first could get some extra media coverage as well. But it may not be worth it if your early adopters feel like they got screwed.

Filed under: Data, Tech Comments Off
5Mar/13Off

Let them read text

In a country known for 100-megabit home internet and one of the highest smartphone usage rates in the world, there lurks an archaic and surprisingly popular mass-media technology. Text-TV, the Swedish national TV network's teletext system, boasts 3.7 million weekly viewers. In other words, more than one third of the population of Sweden regularly sits down at their TVs to read news headlines and sports scores in dazzling 6-color monospaced type.

To my dear American readers, teletext technology may be an alien concept (it certainly was to me before being introduced to Text-TV). It is essentially an analog broadcasting hack, where text data is encoded between video frames of a TV signal. It can be used for subtitles and program information, or the TV can be switched to a pure-text mode for viewing different numbered "pages" of text. A viewer navigates to a page by using the remote to enter the page's corresponding number; a primitive hyperlink.

texttv1

The image above is the current front page of TV4 Text. Note the numbered links for different news articles and sections. Quick Swedish lesson tangent: Avförings means "fecal."

The BBC pioneered teletext technology in the early 70's, and Sweden was the next country to follow a few years later. Many networks around the world adopted teletext systems since then, but in the past decade some have shut down because of the transition to digital broadcasting and the ubiquity of the internet.

But not in Sweden. SVT, the Swedish national TV network, has admirably kept Text-TV relevant in today's ever-changing media landscape. They adapted the text encoding for digital broadcasting, so it is available as a part of current cable and over-the-air service. But more impressively, they offer a Text-TV website which faithfully recreates the familiar page layout and navigation, including the option of a white-on-black "TV" color scheme.

Discovering this new old technology was a revelation for me, but I was skeptical as to how much it is really used among my own "Twitter this, Facebook that" generation. So I asked a Swedish friend whether young people actually watch the Text-TV. His response was, "It's mostly older people who grew up with it their whole lives, but the app is pretty good for sports."

The app?

Yes, that's right. A 40-year-old system for displaying text on a television has been ported to iPhone and Android, along with the same vintage fonts and colors.

TextTV Android app screenshot

Text-TV Android app screenshot

Apparently the 3rd-party iPhone version was one of the most popular apps in Sweden when it was released. According to an editorial director at SVT, "Text-TV has a minimalist style, straightforward language, and an interface with no pictures or anything to overload a mobile. I also think that Text-TV has become a little like a cult, there is some kind of nostalgia for many."

May Text-TV live to see another 40 years.

Filed under: Personal, Tech No Comments
26Jan/13Off

Amusing myself with basic Javascript

Just look at that URL. Madness!

Just look at that URL. Madness!

Call me crazy, but ever since I noticed the random 32-letter ID's for apps in Google Chrome Web Store URLs (see above), I can't help but imagine pronouncing them in my head. The thought of someone actually trying to read off the nonsensical URL is almost unbearably hilarious to me for some reason. It's just one of those weird things I do to keep myself entertained.

So today I decided to take it one step further. I made a bookmarklet that uses Google Translate's text-to-speech engine to get a real robotic pronunciation of the letters in a URL. Naturally it uses the Italian robot voice because it's the funniest one I found. Want to hear the results?

  1. Drag the following link to your bookmarks bar. URL Pronouncer
  2. Go to a page on the Chrome store, like this one.
  3. Click the bookmark you just added.

I've tested this on Chrome and Safari and it might not work on Firefox.

It sometimes works on other websites but not all the time, probably because of some restrictions on Google's side. If you know what the problem might be, let me know.

This useless little project is also the first thing I've uploaded to GitHub.

Filed under: Tech No Comments
22Jan/13Off

The most American cookie of them all

Discussion topic: What kind of cookie best represents the United States?

At work, the product managers have started taking turns bringing in baked goods every Wednesday. The lazier ones among us just buy something at the store, but for my first Wednesday fika I naturally wanted to bring a home-baked treat. One of my coworkers suggested I bring "something American", which got me pondering the above question.

So back to the question at hand. If you said chocolate chip cookies, come on. Be a little creative. My answer?

P1000766

 

Oatmeal-peanut butter Breakfast Cookies based off this recipe. Unfortunately I didn't have my mom's recipe for these cookies; that one is much better. But the idea is the same. So what makes breakfast cookies the most American cookies? Several reasons:

  • Peanut butter. In the U.S. we don't really consider PB to be typically American, but that's how the rest of the world sees it. In fact, more than one Swede has asked me why Americans like peanut butter so much. They just don't get it. Furthermore, it's in the international section of the grocery stores here. George Washington Carver would be proud.
  • Mix-ins. While the linked recipe just calls for raisins, I also included M&Ms, and the real recipe throws in chocolate chips too for an all-out attack of texture and flavor. I don't have any strong argument for why mix-ins are American, but it just feels right, doesn't it?
  • And of course the obvious one: Who else but Americans would have the audacity to suggest that cookies are a suitable breakfast food? And then this cookie goes and puts it right in its name!

To drive home the whole American thing, I made mine extra large. I will see how they go over with the Swedes tomorrow.

Filed under: Food 1 Comment
7Jan/13Off

If at first you don’t succeed…

Image credit: someone from the internet

Image credit: someone from the internet

Following up on my previous bread post, I finally got around to making my first sourdough bread this weekend. Or should I say, trying to make my first sourdough. I was not expecting perfection; baking good bread takes a lifetime to master, and the sourdough recipe I was using is many times trickier than anything I've tried before. And since this was hands-down the worst loaf I've ever made, chances are pretty good that the next one will turn out better.

So what actually went wrong? I could explain it myself, but instead I will quote a former coworker with whom I've been corresponding about baking. He managed to predict the exact problems I would have, unfortunately just a little too late for me to do anything about it:

The basic Tartine bread looks easy, but is actually quite difficult to master (many, including me, really struggle with it). It has a very high hydration, and working with the dough is a challenge. Shaping it is almost impossible. American flour tends to absorb more water than European flour. If you used American quantities with European flour, you probably had a super-hydrated dough, which is even more difficult to work with.

In other words, my dough was about halfway to being batter. Trying to form a loaf with this goop was like trying to make a pile of water.

The "dough" formed into a "loaf"

The "dough" formed into a "loaf." Note how it's bonded with the towel.

After a good deal of cursing I decided to just go with it and see what happens. But my problems were compounded: because I had put so much effort into forming this blob into a workable shape, I had driven out much of the carbon dioxide that had formed during the fermentation and initial rising. This would be one flat loaf. Perhaps as a consolation, I thought, I could get a rabbi to certify it kosher for Passover. (Sorry gentiles, I'll explain it to you later).

After one final rise overnight, I put the sad-looking mess into the oven, bracing myself for disappointment. And what came out was... about what I expected.

Brick o' bread

Brick o' bread

A good crumb would have more holes than bread

A good crumb would have more holes than bread

Yeah, there are some tiny holes, but this is one dense loaf. The combination of the overly-wet dough and the lack of dissolved gas meant this guy didn't really rise at all in the oven. I gnawed on a couple slices but the rest of it is just sitting on my counter, growing staler by the minute; an indestructible monument to crushed dreams.

But the next one will be better. That's how this sort of thing works: iteratively improve your technique, hone the recipe, and keep raising your standards. Until you get bored or develop a gluten sensitivity. This may have been a small setback in my quest for bread supremacy, but don't you worry, I will rise again.

Filed under: Food No Comments
2Jan/13Off

Generalizing entire nations of people for fun and profit

A coworker recently gave me a book that she hoped would help me assimilate into Swedish culture.

P1000753

Xenophobe's® guide to the Swedes is a handy guidebook of important Swedish stereotypes and cultural peculiarities. Naturally, it's written in a tongue-in-cheek style, and it covers every angle, from food and dinner parties to  family values and popular culture. Here are some of the profound insights that Xenophobe's provides:

On the environment:

The Swedes have a dream: to save Nature from Man. It's as close to a passion as the Swedes ever get.

My verdict: pretty good; see my previous post about recycling. Passion is indeed in short supply here, but I wouldn't even use the term to describe their environmentalism. It's more of an ideology ingrained in the culture, like that education and healthcare should be free.

On fame:

The only greater sin than being rich is being famous, though it is acceptable to acquire fame that rubs off on Sweden as a whole, allowing everyone to bask in the limelight. [...] But Ingrid Bergman, glamorous star of such Hollywood classics as Casablanca and Indiscreet, fell foul of her Swedish audiences because she earned her fame as an expatriate and failed to flaunt her Swedish origins at every opportunity.

I can't really speak to this one, but it seems like something that would be harder to spot as an outsider. I'll investigate further.

On coffee:

Anyone wondering what really makes the Swedes tick need look no further: it is coffee. A coffee embargo would bring the country to a halt within days.

My verdict: Spot on. My coffee consumption has at least doubled since moving here, and I'm still a lightweight by Swedish standards.

The book was written by an authentic Swede, and overall it captures a lot of the stuff I've noticed about this country. But I think it still has some gaps. For example, the food section doesn't once mention the Swedes' fixation with emulsified sauces or fermented dairy products. So I guess I'll need to fill in those gaps. Stay tuned.

Apparently Xenophobe's is a whole series, published out of Britain, with guides for most European countries as well as English-speaking countries around the world. You're probably wondering what the American one is like, so here's the excerpt they included:

A wise traveller realises that a few happy moments with an American do not translate into a permanent commitment of any kind. Indeed, permanent commitments are what Americans fear the most. This is a nation whose most fundamental social relationship is the casual acquaintance.

Discuss amongst yourselves...

Filed under: Personal No Comments
31Dec/12Off

Getting started with sourdough

Now that I've kind of settled in in Sweden, I decided to start a project that I'd been meaning to do for a while: make sourdough bread from scratch. In case you aren't aware of the process, sourdough bread does not use packaged yeast like most bread. Rather, you cultivate a sourdough starter: a culture of natural yeasts and bacteria which feeds on a flour/water mixture. Since the starter is a living thing, you need to feed it daily to keep it happy and in balance. So basically maintaining a sourdough starter is like having a pet that you can eat.

You can get started making sourdough by taking some of a pre-existing starter and continuing to feed it, but it turns out it's really easy to make starter spontaneously by letting some flour and water sit out for a couple days. The cooties naturally occurring in the flour and the air will begin to grow in the mixture, and it just takes a few days for the little ecosystem to get into balance. I used the basic guide in the Tartine Bread book: 50/50 mix of whole wheat and white flour, mixed with water to form a batter, put in a cool dark place for 2 or 3 days until it smells funny and bubbles. Then feed it daily by discarding 80% of it and adding fresh flour and water.

I just began this project 5 days ago, but my starter is already acting and smelling like a top-notch sourdough breed. I'm so proud of it. In case anyone is interested in jumping on the sourdough bandwagon, here's a rough timeline of how my starter developed so you can see if you're on the right track. The best way to detect changes in a fermentation process is by smell, so if you don't want to hear me describe what bacteria smells like, just skip over this part.

  • Day 0: Create initial flour/water mixture.
  • Day 1: No noticeable bubbles, not much smell
  • Day 2: Some bubbles and a strong vinegar/jet fuel fermentation smell. Not pleasant but a sign of life. I did the first feeding then.
  • Day 3: Less jet fuel, more vinegary and a little fuller bodied aroma. Before the feeding it looked kind of dead but bubbles started forming within 20 minutes or so of feeding.
  • Day 4: Almost pleasant-smelling. Yeasty with some blue cheese notes. Even more bubbling/growing in the first couple hours after feeding than in previous days.
  • Day 5: Much more mellow aromatically, for the first time it actually smells a bit like sourdough.

At this point it already seems trained to the healthy 24-hour feeding cycle, but I'll probably give it a few more days before trying to make bread with it.

Day 4

Day 4

What if I'm not around for a few days, you ask? Will my starter starve to death, tragically unaware that it lives mere inches from a container with a lifetime supply of its food? Probably, but if I grow too attached to let that happen, Sweden's got my back. A bakery here in Stockholm has opened a "sourdough hotel" where they will care for your beloved microorganisms while you are out of town. But that's just a little too insane.

Filed under: Food No Comments
25Dec/12Off

Photos: vacation edition

I just got back from a week-long trip to Spain. I traveled around to Granada, Sevilla, and Barcelona. The highlight of the trip was seeing the Sagrada Familia, a church designed by Antoni Gaudi in the 1880s-1920s. The building's design is unlike anything I've ever seen; it's described as "neo-gothic" but that doesn't really express the uniqueness of it. The most interesting thing, though, is that it's still being built, and isn't expected to be completed until the late 2020s. I guess I just admire the outrageous ambitiousness of the project. Anyways, here's some photos from the trip.

  • Granada at dawn
  • Beyond the fortress walls at the Alhambra
  • Street art in Granada
  • Strike a pose. Granada
  • Granada
  • Christmas fair in Granada
  • Granada
  • Horse and carriage at the Cathedral - Sevilla
  • Alcazar garden in Sevilla
  • Cathedral of Sevilla
  • Sevilla
  • Barcelona
  • Boqueria market, Barcelona
  • Church of Santa Maria del Mar, Barcelona
  • Dog sitting on a bench, Barcelona
  • Sagrada Familia, Barcelona. Expected to be completed by 2030.
  • Sagrada Familia, Barcelona
  • Sagrada Familia
  • Tortoise column at Sagrada Familia having a snack
  • Barcelona bullring. Now it's a shopping mall.
  • Bridge at Park Güell
  • Park Güell
  • Rock and Roll at the top of Park Güell
  • Casa Batlló

Gallery permalink

Filed under: Personal 1 Comment
16Dec/12Off

Wintertime photos

I realized I've had a lot of texty posts lately. Here's some photos.

Filed under: Personal No Comments