on{X}

A cou­ple of weeks ago I came back from a great trip to visit my very tal­ented and hip team at the Microsoft Israel Devel­op­ment Cen­ter in Tel Aviv.  Excite­ment was run­ning high as we geared up for the launch of on{X}, which I can now talk about, as it went live today.  Here’s the “trailer”:

 

Eran, the team lead, already wrote a nice and detailed blog post for the web­site talk­ing about on{X}’s capa­bil­i­ties, so prob­a­bly I should just stick with the link.  I can’t resist reit­er­at­ing some of his points, though, and giv­ing a bit of my own color.

Some­time in the mid­dle of the last decade, it became clear to every­one that mobile phones aren’t appli­ances or pieces of single-function con­sumer elec­tron­ics, but rather are lit­tle per­sonal com­put­ers— small enough to keep in your pocket, and with just enough bat­tery to stay on the whole day.  I think that because of a kind of metaphor hys­tere­sis, the impli­ca­tions of this still haven’t fully sunk in; we still call this device a “phone”, which is the same name we give to the fixed-function radio hand­sets some of us still have docked in a cra­dle on the kitchen counter… which are in turn noth­ing but “cord­less” ver­sions of their mid­cen­tury fore­bears, which in turn con­sisted of noth­ing but a speaker, a micro­phone, some wire, and some kind of dial­ing appa­ra­tus encased in a Bake­lite shell.  Even that silly word we use in Amer­ica, “wire­less”, sounds like more or less a syn­onym for “cordless”.

So we now agree that it’s not really a phone, but rather a telephony-capable pocket com­puter.  To call it a “phone” is an unnat­ural act of metonymy akin to call­ing a toolch­est a screw­driver.  Think of how funny it sounds to say “my phone comes with a phone app preinstalled”.

One odd­ity about this pocket com­puter idea is that unlike the big­ger com­put­ers those of us who like to hack have grown up enjoy­ing, smart­phones still inherit much of the fixed-function think­ing that per­vades con­sumer elec­tron­ics.  Deep in the phone’s OS is a kind of dis­patch­ing or command-and-control sys­tem, a switch­board, that takes all of those won­der­ful sen­sors and capa­bil­i­ties and orga­nizes them into rigidly pre­de­fined appliance-like behav­iors: when the always-on radio detects an incom­ing phone call, launch the phone app.  When there’s an incom­ing SMS, launch the SMS app.  You don’t need to know that there’s an app for that, because the phone itself does.

What if you could open up that switch­board on your own device and rewire it to do what­ever you want?  With all of those sen­sors and capa­bil­i­ties, and all the power of the Inter­net, one should be able to do a lot more than per­form fixed phone-like func­tions and run apps.

This is espe­cially inter­est­ing when it involves the automa­tion of actions based on events on the phone or in the world, which one could call “push” or “reac­tive” behav­iors.  (This is to dis­tin­guish them from “pull” behav­iors, which are char­ac­ter­ized by begin­ning with an explicit user action with intent— for these cases, the app model works well.)  We think there’s great untapped poten­tial in push and reac­tive pro­gram­ming.  The sam­ple sce­nar­ios and scripts we’ve put on the site begin to explore the pos­si­bil­i­ties, but we imag­ine that the devel­oper com­mu­nity will come up with a much, much larger set.  That’s why we didn’t restrict on{X} to pre­scripted rules, but rather made it pos­si­ble for any­one with JavaScript skills to hack a new behav­ior.  This think­ing (and of course our choice of script­ing lan­guage) is very much inspired by node.js.

The most excit­ing thing about this project, for me, will be see­ing what peo­ple do with this wide-open field.  Mak­ers, have at it!

In the mean­time, we’ll be busily adding capa­bil­i­ties and shar­ing ideas.

3am update: appar­ently we’ve just made App­Brain’s top 10 hottest Android apps :)


Posted in mobile | Tagged , , , , | 4 Comments

translator

I was pleased to see David Pogue’s pos­i­tive review of the new Win­dows Phone, Nokia’s Lumia 900, a cou­ple of weeks ago in the New York Times.  Win­dows Phone has made great progress these past cou­ple of years, and has advanced a beau­ti­ful and fresh design lan­guage, Metro, which we see being adopted all around Microsoft.  (I’ve been a big advo­cate for Metro lan­guage and prin­ci­ples in my own part of the com­pany, Online Ser­vices.)  Pogue’s only real com­plaint is that apps for Win­dows Phone  are still thin­ner on the ground than on iPhone and Android— though as he points out, what really mat­ters is whether the impor­tant, great and use­ful apps are there, not whether the total num­ber is 50,000 or 500,000.  Many apps doesn’t nec­es­sar­ily imply many qual­ity apps, and most of us have got­ten over last decade’s “app mania” that inspired one to fill screen after screen with run-once wonders.

What really made me smile was Pogue’s char­ac­ter­i­za­tion of what those impor­tant apps are, in his view.  After reel­ing off  a few of the usual sus­pects— Yelp, Twit­ter, Pan­dora, Face­book, etc.— he added:

Plenty of my less famous favorites are also unavail­able: Line2, Hip­munk, Nest, Word Lens, iStop­Mo­tion, Glee, Oca­rina, Songify This.

Even Microsoft’s own amaz­ing iPhone app, Pho­to­synth, isn’t avail­able for the Lumia 900.

I’ve also been asked (a num­ber of times) about Pho­to­synth for Win­dows Phone... hang in there.  A nice piece of news we’ve just announced, how­ever, is a new app for Win­dows Phone that I hope will join Pogue’s pan­theon, and that is con­sid­er­ably more advanced than its coun­ter­parts on other devices: Trans­la­tor.  Tech­ni­cally this isn’t a new app, but an update, though the update is far more func­tional than its predecessor.

Trans­la­tor has offline lan­guage sup­port, mean­ing that if you install the right lan­guage pack you can use it abroad with­out a data con­nec­tion (essen­tial for now, I wish inter­na­tional data were a prob­lem of the past).  It also has a nice speech trans­la­tion mode, but what’s per­haps most inter­est­ing is the visual mode.  Visual trans­la­tion is really help­ful when you’re encoun­ter­ing menus, signs, forms, etc., and is espe­cially impor­tant when you need to deal with char­ac­ter sets that you not only can’t pro­nounce, but can’t even write or type (that would be Chinese).

Word Lens, men­tioned by Pogue, was one of our inspi­ra­tions in devel­op­ing the new Trans­la­tor.  What’s impres­sive about Word Lens is its abil­ity to process frames from the cam­era at near-video speed, read­ing text, gen­er­at­ing word-by-word trans­la­tions, and over­lay­ing those onto the video feed in place of the orig­i­nal text.  This is quite a feat, near the edge of achiev­abil­ity on cur­rent mobile phone hard­ware.  In my view it’s also one of the first con­vinc­ing appli­ca­tions of aug­mented real­ity on a phone.  How­ever, the approach suf­fers from some inher­ent draw­backs.  First, the trans­la­tion is word-by-word, which often results in non­sen­si­cal trans­lated texts.  Sec­ond, there isn’t quite enough com­pute time to do the job prop­erly in just one frame, yield­ing a some­what slug­gish feel; on the other hand the inde­pen­dent pro­cess­ing of each frame is waste­ful and often makes words flicker in and out of their cor­rect trans­la­tions, just a bit too fast to fol­low.  For me, these things make Word Lens a good idea, and bet­ter than noth­ing in a pinch, but imperfect.

The visual trans­la­tion in Trans­la­tor takes a dif­fer­ent approach.  It exploits the fact that the text one is aim­ing at is printed on a sur­face and is gen­er­ally con­stant.  What needs to be done frame-by-frame, then, is to lock onto that sur­face and track it.  This is done using Photosynth-like com­puter vision tech­niques, but in real­time, a bit like the video track­ing in our TED 2010 demo.  Selected, sta­bi­lized frames from that video can then be rec­ti­fied and the opti­cal char­ac­ter recog­ni­tion (OCR) can be done on them asyn­chro­nously— that is, on a timescale not cou­pled to the video fram­er­ate.  We can do a bet­ter job of OCR and trans­la­tion, using a lan­guage model that under­stands gram­mar and multi-word phrases.  Then, the trans­lated text can be ren­dered onto the video feed in a way that still tracks the orig­i­nal in 3D.  This solves a num­ber of prob­lems at once: improv­ing the trans­la­tion qual­ity, avoid­ing flicker, improv­ing the frame rate, and avoid­ing super­flu­ous repeated OCR.  It’s a small step toward build­ing a per­sis­tent and mean­ing­ful model of the world seen in the video feed and track­ing against it, instead of doing a weaker form of frame-by-frame aug­mented real­ity.  The team has done a really beau­ti­ful job of imple­ment­ing this approach, and the ben­e­fits are pal­pa­ble in the experience.

Use this app on your next visit to China!  I’d love to read com­ments and sug­ges­tions from any­one try­ing Trans­la­tor out in the field.


Posted in mobile | Tagged , , , , | Leave a comment

bear pond espresso

Much has been writ­ten about the food scene in Tokyo.  In the end, maybe it’s just a very, very big city.  The “largest met­ro­pol­i­tan area in the world”, says Wikipedia, and among the dens­est (at more than 37,000 per square mile).  I’m sure that if we plot­ted the num­ber of Miche­lin stars per city against either of these vari­ables we’d find a more-than-linear rela­tion­ship, for the same rea­sons one finds such rela­tion­ships with the num­ber of patents or any other mea­sur­able cor­re­late of creativity.

Well, on a recent trip to Tokyo I expe­ri­enced this cor­re­la­tion first­hand.. or maybe I just ran into a very spe­cial tal­ent.  Not that either of these pos­si­bil­i­ties excludes the other.

Morgan-san intro­duced me to this very cute neigh­bor­hood, Shimo-Kitazawa, just a cou­ple of tube stops from our hotel in Shin­juku, and home to the extra­or­di­nary Bear Pond Espresso, with owner and appar­ently sole barista Katsu Tanaka pre­sid­ing over his tricked out La Marzocco.

Some pic­tures first of the neighborhood:

  

The scale and pro­por­tions of neigh­bor­hood archi­tec­ture in Tokyo are so dis­tinct.  I feel as if, were a streetscape reduced to noth­ing but fea­ture­less white boxes, I could still pick out a street in Tokyo from one in any other city in the world.  When it works, there’s some­thing pleas­ing and inti­mate in this con­fig­u­ra­tion of space, a sense of implicit trust, of social con­tract and effi­ciency, of highly inten­tional design.  In Shimo-Kitazawa the Japan­ese love of tex­ture and mate­r­ial is also evi­dent, and thank­fully there isn’t so much of that ubiq­ui­tous rec­tan­gu­lar ceramic tile on the build­ing façades which I find so oppres­sive.  Instead, lov­ingly rusted iron plates, sal­vaged twists of wood and art­fully rough cal­lig­ra­phy dec­o­rate the bou­tiques, which rub shoul­ders with what look to me like tra­di­tional Edo houses among the back streets.

And now, Bear Pond.  There was a “no cam­era” sign inside, so I was only able to snap a pic­ture of the white slid­ing doors from the out­side.  The space is small and cement-floored, com­bin­ing the shabby genius sen­si­bil­ity of a Bay Area garage with the curated aes­thetic Miyazaki and Kondō cap­ture so won­der­fully in Whis­per of the Heart.  There’s enough care­fully deployed clut­ter in the minia­ture space to make it clear that this is a neigh­bor­hood cof­feeshop and not a bou­tique, yet even the furred, ingrown cor­ners of the “order here” sign seem to sig­nify, like the ful­some red drips of the graf­fiti on the cof­fee machine (cour­tesy Cur­tis Kulig).

     

(Note that sev­eral of these images are from the lit­tle Bear Pond book, which I bought a copy of although it’s unde­cod­able by my non-Japanese-reading self.)

OK, let’s talk about the cof­fee.  It was amaz­ing.  Mor­gan and I had an espresso and a mac­chi­ato each.  The espresso rede­fines ristretto.  It was a messy dark syrup, just a few cc’s, and dense, dense, dense.  There were plenty of trapped aro­mat­ics, but the thick extrac­tion didn’t cre­ate any­thing like the usual tiger-stripe crema— at the bot­tom of the cup, noth­ing but an inky, intense depth of fla­vor and color, sweet, funky and com­plex, cocoa-ish, with no hint of the “south­ern” burnt fla­vor and even less of the acid lemoni­ness of an improper extrac­tion.  This really was some­thing new, and I’m afraid I don’t really have the words for it.  It’s just a dif­fer­ent beast.

The mac­chi­ato was good— I would have been happy with it under any other cir­cum­stance— but I think it’s a waste to blend such exquis­ite nec­tar with milk, even small amounts.  The caramel-like effect Vivace achieves in the mac­chi­ato with their Vita blend was not in evi­dence here; this espresso felt both too dense and too ephemeral some­how to fold with the milk fla­vors, they seemed not imped­ance matched.  Per­haps an issue of pH?  The Japan­ese milk is dif­fer­ent, also, with less body it seemed to me, and this may have con­tributed to my impression.

The work­man­ship of the lit­tle rosetta was of course per­fect, but alas, “no cam­era please”.

In any case, if you find your­self nearby, treat your­self well— wan­der the neigh­bor­hood, drop in, taste what I sus­pect might be the most beau­ti­fully crafted espresso anywhere.


Posted in food, travel | Tagged , , , , , , | Leave a comment

fish stew

As I’ve been com­pelled to email this very good bouillabaisse-like recipe from Ondi to myself a num­ber of times over the past cou­ple of years, it seems more con­ve­nient and socia­ble to put to put it on the blog.  Here goes.

  • An onion.

Fry in olive oil.  Add:

  • Cumin seed
  • Corian­der seed
  • Sev­eral sprigs of thyme
  • 2 bay leaves
  • Fen­nel seed
  • Sev­eral cloves gar­lic (pressed or finely chopped)
  • Saf­fron
  • Sea salt to taste
  • Pernod.

When onions are golden add:

  • 1 can of San Marzano DOP toma­toes (can be chopped before­hand, and can sub­sti­tute fresh Roma toma­toes if suf­fi­ciently good ones are available)

Cook down for a good 20 min­utes.  Then add:

  • Chicken broth or fish stock till the base is as thick or thin as you like.
  • When that’s cooked about 20 min­utes, add assorted seafood.  Typically:
  • Monk­fish
  • Salmon, in chunks cut from a steak is easiest
  • Spot prawns
  • Clams
  • Mus­sels

I use the prawn shells and heads to make the stock.

For extra joy, I also add sea scal­lops at the end, which I’ve seared in but­ter and flamed with Pernod.

This should be served with a crusty bread.


Posted in food | Tagged , | 2 Comments

escape from camp 14

Our good friend Blaine is gear­ing up for the pub­li­ca­tion of his new book, Escape From Camp 14.

It’s about a young man who has man­aged to escape from the most bru­tal of North Korea’s giant con­cen­tra­tion camps, even­tu­ally mak­ing it to the US.  The story is absolutely har­row­ing.  North Korea’s total insu­lar­ity seems to have kept most of the world from really under­stand­ing what’s going on inside— sense­less hor­rors on a truly grand scale.  It’s a bit like imag­in­ing a Nazi Ger­many turned entirely inward rather than bent on dom­i­nat­ing other coun­tries.  In this sce­nario, would the rest of the world have stood by while the atroc­i­ties were com­mit­ted?  The answer appears to be ‘yes’.

Over the past year, through lots of long fam­ily din­ners and hur­ried hel­los at the neigh­bor­hood cof­feeshop, I’ve had the good for­tune and honor to see Blaine develop Escape From Camp 14 from an idea, through sev­eral drafts, and of course through the ups and downs that go with this kind of project.  Blaine is a pro at this, and in addi­tion to his very pro­lific career as a reporter, he has writ­ten some great books— A River Lost: The Life and Death of the Colum­bia (hurry!  Ama­zon says only 11 left in stock!), and Africa: Dis­patches from a Frag­ile Con­ti­nent, which Adri­enne and I each read in about a day on an Africa jag that I remem­ber also included Beryl Markham’s auto­bi­og­ra­phy and What Is The What.  I have a feel­ing, though, that the time­li­ness of Escape From Camp 14, as well as the extra­or­di­nary source mate­r­ial, might make it his big best­seller.  I cer­tainly hope so— this is a story that needs to be read by many.  It comes out next month, and you can pre­order it now.

Blaine enlisted me to do the voiceover on his promo video.  He assures me it turned out OK, though I can’t actu­ally lis­ten to it.  There’s some­thing hor­ri­fy­ing about hear­ing your own voice recorded.


Posted in books | Tagged , | 1 Comment

invention

Before the five Equal­iz­ing pieces, I made an inven­tion in d minor to begin exper­i­ment­ing with counterpoint:

Inven­tion in d minor

This was the first thing I recorded on the lit­tle per­sonal record­ing device.  It’s a bit of a mess, but post­ing it any­way, as it has some ideas that might be worth return­ing to when I’ve improved.  It’s recorded in two passes, since I can’t yet artic­u­late all four voices in real­time (and this leads to some tim­ing prob­lems).  I used the harp­si­chord set­ting, both because it seemed appro­pri­ate and because in the begin­ning I felt unable to con­trol the piano dynam­ics well enough.


Posted in music | Tagged , , , | Leave a comment

equalizing

So I’ll now take a deep breath and post the first edi­ble fruit of my labor.  The per­for­mance is of course imper­fect, since I’m not an actual pianist, and the piece should also be con­sid­ered a draft, as there are places where it still needs edit­ing.  But I do hope it man­ages to cross the thresh­old into music.

  1. Equal­iz­ing I (Tone pattern)
  2. Sec­ond phases
  3. Equal­iz­ing II (Clockwork)
  4. Night train
  5. Equal­iz­ing III (Indonesia)

OK, some liner notes.

Equal­iz­ing is in five move­ments, where the first, third and fifth are vari­a­tions I, II and III on a cen­tral idea, while the sec­ond and fourth are interludes.

The ascend­ing octave tone pat­tern in the vari­a­tions is some­thing from child­hood.  It was an ana­log equal­izer, one of our pieces of stereo equip­ment, which when put in tone gen­er­a­tor mode would light up the slid­ers in sequence while play­ing the pro­gres­sion of octaves, from a bass you felt in your stom­ach to an inaudi­ble dog-whistle.  Some­times when we’re young we become obsessed with appar­ently triv­ial sen­sa­tions— I remem­ber watch­ing our kids at the beach very slowly drib­bling sand through their fin­gers for hours.  I’m not sure exactly what’s going on dur­ing these long unprompted med­i­ta­tions which would try the patience of a Zen monk, maybe it’s sig­nif­i­cant, maybe not.  But I cer­tainly remem­ber many of those times.  I remem­ber the sense of flow, the engage­ment of all of the senses at once even when the input was purely visual, or purely audi­tory.  And I remem­ber the lim­bic power of those inten­si­ties, the dis­em­bod­ied yet primary-colored emo­tion, hyp­notic and unreg­u­lated.  Even the echo of that is a treasure.

Yes, I know I made over­much use of the sus­tain pedal.

Con­nois­seurs of the WPRB school will notice the influ­ences of John Fahey and Guy Klucevsek, as well as Steve Reich and Arvo Pärt.  Like the last two, I’ve tried to build around a stripped-down har­monic arma­ture (though I haven’t been as rig­or­ous about stick­ing to it as those sages).  This feels con­sis­tent with the octave tone pat­tern at the core of Equal­iz­ing, which on the one hand is triv­ial in that it lacks any har­monic struc­ture at all.  On the other hand, con­tain­ing noth­ing but pow­ers of two, it’s nei­ther major nor minor, but hov­ers behind and above this pass­ing weather in a kind of mathy space beyond sad or happy.  Is it just me, or is this naked pre­cur­sor to tonal­ity already thrum­ming with feeling?


Posted in music | Tagged , , | Leave a comment

piano

A cou­ple of months ago I began play­ing with our new dig­i­tal* piano.  What a lovely thing a key­board instru­ment is!  You just push a but­ton, and out comes a per­fect note— hard if you hit it hard, soft if you hit it softly, and never out of tune.  The but­tons feel great and they’re laid out in a neat row.  The tonal range yawns across many octaves.  You’re lim­ited only by the physics of your hands.  It feels like cheat­ing com­pared to every other instru­ment I’ve ever played.  No won­der music began really tak­ing off when this tech­nol­ogy was introduced.

(*Why dig­i­tal?  Because it has headphones.)

Not that I can be said to actu­ally play the piano.  Over the past cou­ple of months I’ve “learned” it in much the same hap­haz­ard way I learned to type thirty years ago— with­out any dis­ci­pline or tech­nique what­so­ever, dri­ven purely by the urgent desire to trans­duce.  (Now that I think of it, most of what I know is char­ac­ter­ized by this rather sloppy approach.)  I’m sure I could achieve bet­ter results by study­ing it prop­erly with a teacher, and I’m still painfully slow at read­ing from the staff.  But: I began mak­ing music any­way, with the piano, a hand­held recorder, and some nota­tion soft­ware.  There have been some late nights.


Posted in music | Tagged , | Leave a comment

treasure hunt

Over the past year I’ve set the kids a num­ber of trea­sure hunts, which are both fun to make and always in demand.  Like any­thing one tries a few times, it’s got­ten quicker.  This morning’s was assem­bled in record time and under duress.  It turned out decently given con­straints and incor­po­rated a shop­ping trip, though I’m regret­ting not hav­ing made Eliot’s half of it harder.  Next time..

The struc­ture usu­ally works like this.  Clues are writ­ten on slips of paper, which are hid­den in var­i­ous places, each point­ing to the next.  There’s some sort of prize at the end.  These were the clues:

My notes:

The library is use­ful as a place to hide clues— it’s a source of fun things to read, and shelf col­umn, row and book num­ber form a coor­di­nate sys­tem, which is good for math problems.

The Dylan Thomas poem was Where once the waters of your face, with the fol­low­ing clues in a jar of salt and in sage:

Thanks to Ilse for tak­ing over from A5/E6.  She MMS’d me their favorite Schiele paint­ings from 1909-10,

.. after which I texted back the final clue lead­ing to their prizes:


Posted in children | Tagged , , , | Leave a comment

kindergarten: saudade do futuro

I hope not to get in trou­ble for post­ing, ver­ba­tim and with min­i­mal com­men­tary, an update we’ve just received from Eliot’s kinder­garten.  I find this fas­ci­nat­ing and, in its way, rather utopian:

As you know, we have been exper­i­ment­ing in our room with dif­fer­ent kinds of learn­ing sup­ports for our class­room, such as “wig­gle spots” to sit on, head­phones for qui­eter con­cen­tra­tion, and “pri­vate offices.” Many of the kids seem to have high oral needs right now, so we intro­duced two “chew tools” this week — chewy straws and gum. The gum is all nat­ural, no chem­i­cal sweet­en­ers, and sugar free (sweet­ened with xyl­i­tol). We will con­tinue to sup­ply straws as needed as well as chewy snacks at choice time. The gum seems to work bet­ter than the straws and be more hygenic as it does not touch hands or other sur­faces once it is in the mouth.

A need to chew on pen­cils, cloth­ing or hair is a com­mon kinder­garten behav­ior, and pro­vid­ing alter­nate chew­ing expe­ri­ences can help focus and con­cen­tra­tion as well as be relax­ing for kids. How­ever, gum is not a choice for all fam­i­lies, so if you would pre­fer that your child not use this chew tool, please let me know.

This is dif­fer­ent from the kinder­garten of memory.


Posted in thoughts | Tagged , | Leave a comment