photosynth app!

As every­one who’s been glanc­ing now and then at my face­book page knows, I’ve been post­ing a lot of panora­mas late­ly from the mys­te­ri­ous “Pho­to­synth pub­lish pho­tos” app.  Well, the app is now final­ly avail­able on the iPhone— just look in the app store for “pho­to­synth” or “bing” (Apple’s index looks like it’s rebuild­ing now, hope­ful­ly by the time I put this post up it’ll be ful­ly live).

I’m very excit­ed to final­ly have this app out there.  The team’s done a won­der­ful job on it.  Here’s the offi­cial blog post.  They also shot a very nice release video, which to my cha­grin ends with y.t. pon­tif­i­cat­ing on a posthu­man future.

A few notes on this evo­lu­tion in the Pho­to­synth and Bing Mobile sto­ry, in no par­tic­u­lar order.

experience

This app is a lot of fun to use, and the out­put is— I think— com­pelling.  It address­es a fun­da­men­tal lim­i­ta­tion of cam­eras as we now know them: field of view.  A phone’s cam­era is the right size and design for tak­ing a snap­shot of, say, someone’s face at a par­ty.  But as any­one who has tried to put up real estate pho­tos knows well, try­ing to cap­ture a view out the win­dow with an ordi­nary cam­era is an awful expe­ri­ence.  With your eyes, you see a whole for­est; in the cam­era frame, you see a cou­ple of trees.  With Pho­to­synth, you rotate the cam­era to take in the whole view, and the app fus­es that view togeth­er:

Of course there’s an inher­ent dif­fi­cul­ty in tak­ing these wide fields of view and pro­ject­ing them down into pla­nar images.  It’s the­o­ret­i­cal­ly pos­si­ble to do this with a per­spec­tive pro­jec­tion when the field of view is less than half a sphere, though in prac­tice the image starts to dis­tort unpleas­ant­ly when the larg­er axis exceeds 60 degrees or so.  Remem­ber that in our own eyes, the reti­na is hemi­spher­i­cal, not pla­nar.  Our very wide nat­ur­al field of view doesn’t rely on a rec­tan­gu­lar pro­jec­tion the way film or dig­i­tal cam­eras do.

This is why, when Pho­to­synth pub­lish­es pho­tos to a flat medi­um, like the inline news feed in face­book or the flat image above, it uses a spher­i­cal pro­jec­tion, which results in a dis­tort­ed image.  Straight lines turn into arcs.  This kind of imagery can be quite beau­ti­ful in its own right, though it can get a bit coun­ter­in­tu­itive, espe­cial­ly as the field of view grows all the way to the full sphere.

Pro­ject­ing the sphere down to a rec­tan­gu­lar image is of course just what one does when one makes a flat map of the Earth.  We’ve all seen pla­nar world maps so often that we think of them as “nor­mal”, while the above image looks “dis­tort­ed”, although real­ly they exhib­it the same dis­tor­tions.  Yes, Green­land is big— about three times the size of Texas.  Not twice the size of the whole USA.

The best way to expe­ri­ence a panora­ma is in an immer­sive view­er, which repro­jects the imagery inter­ac­tive­ly into a small­er win­dow, allow­ing you to rotate.  We’re work­ing on view­ers that let these things hap­pen in native Web-ese, though it requires advanced browsers (HTML5, Can­vas or CSS3).  In the mean­time, it can be done with Sil­verlight.  Here’s the panora­ma of the Neue Nation­al­ga­lerie above, ren­dered this way:

technology

In the past few years there has been a grow­ing trick­le of “smart” cam­eras and phone apps for stitch­ing togeth­er pho­tos or video into panora­mas.  Many of them aren’t par­tic­u­lar­ly good, but I do need to give a shout out to our friends at Occip­i­tal, whose 360 panora­ma app was an inspi­ra­tion to the team.

As far as I know, ours is the first app that goes beyond “strip” panora­ma mak­ing to allow cov­er­age of the entire visu­al sphere with real­time track­ing.  This relies on some pret­ty cut­ting-edge com­put­er vision hack­ing (thank you Georg).  Extract­ing fea­tures from the video stream on the fly, fol­low­ing them from frame to frame, and mod­el­ing the envi­ron­ment in real­time is hard­core stuff, and requires not only high-per­for­mance algo­rithms, but also very aggres­sive low-lev­el opti­miza­tion and trick­ery with the cam­era and graph­ics pipelines.  Fusion on the sphere isn’t all the way to full 3D mod­el­ing, but it already involves its share of topo­log­i­cal com­pli­ca­tions and trade­offs between local and glob­al recon­struc­tion.  By all means, dear CS grads, try this at home (and send us your screen­shots)— but be fore­warned, there’s a rea­son why we haven’t seen a mobile app like this before in the mar­ket!  This is a bit like Doom first appear­ing on the PC in 1993.  Now that it’s been shown to be pos­si­ble, we should expect quite a few fol­low­ers, and a rapid evo­lu­tion in real­time com­put­er vision on mobile devices.

design

There’s anoth­er rea­son I’m very pleased by Pho­to­synth for iPhone: it’s the first rea­son­ably com­plete appli­ca­tion of our design sys­tem to a non-Win­dows phone.  We’ve been work­ing for more than a year with the very beau­ti­ful design lan­guage cre­at­ed by the Enter­tain­ment and Devices peo­ple at Microsoft (yes, Microsoft can do beau­ti­ful design!  It’s true!) for Xbox, Zune, and Win­dows Phone 7, code­named Metro.  The “laven­der” map style we released last year is a car­to­graph­ic embod­i­ment of this lan­guage.  Trans­lat­ing Metro for an envi­ron­ment like iPhone, in which there’s a strong native look, feel and inter­ac­tion mod­el, is risky busi­ness.  Done poor­ly, the result is con­fus­ing and incon­gru­ous.  Some would argue that an app should always adopt native con­trols, look and feel, tai­lor­ing itself entire­ly to the host plat­form to min­i­mize cog­ni­tive dis­so­nance.  This is an old argu­ment.  I remem­ber it from the X win­dows days, and ear­li­er.  (More recent­ly, Apple’s first release of Safari on the PC gar­nered much crit­i­cism for its dis­so­nant non-PC look and feel.  Today they’ve moved clos­er to native.)

I think from this per­spec­tive the Pho­to­synth app is a great suc­cess.  Its inter­ac­tion mod­el, look and feel are very Metro, dis­tinc­tive­ly ours, yet it works in the iPhone con­text.  It has a dis­tinct voice, yet remains trans­par­ent and usable.  It isn’t anti­so­cial in its approach to the plat­form.  I was delight­ed to read this post Adri­enne found on Giz­mo­do (unchar­ac­ter­is­tic read­ing mate­r­i­al for her)—

Also, Microsoft, who as you may know makes their own mobile phone OS these days, has puck­ish­ly brought the Win­dows Phone 7 aes­thet­ic to the iPhone app, which, man, is just real­ly real­ly nice. You don’t realise how, I don’t know, corny all these bev­elled but­tons and 3D ani­ma­tions are until you see Microsoft’s flat, geo­met­ric UI on your iPhone’s dis­play... More apps that look like this, please.

One of the things that I think makes this eas­i­er to do on mobile devices than on PCs with win­dow­ing sys­tems is the fact that mobile apps are always full-screen.  They can cre­ate self-con­tained worlds by book­end­ing an expe­ri­ence in time.  This works for Web pages too, because the Web has evolved from the world of “con­tent” instead of “appli­ca­tion”— one wouldn’t com­plain about dis­tinc­tive design lan­guage on a web­site, any more than one would com­plain about incon­sis­tent fonts on the cov­ers of dif­fer­ent mag­a­zines lined up on a rack.  For­tu­nate­ly, between web­sites and mobile apps, we have the next decade pret­ty much cov­ered.

Here’s one of my favorite screens from the Pho­to­synth app:

The typog­ra­phy is beau­ti­ful and clean, the visu­al bal­ance rem­i­nis­cent of mid-20th cen­tu­ry Swiss design.  Sur­faces are flat, cor­ners square, and any con­tent shown is authen­tic, not just rep­re­sen­ta­tive.  That unstitched pano is a love­ly arti­fact in its own right, a bit of a nod to David Hockney’s join­ers.  I hope that with this app we prove that we can have our cake and eat it too— usabil­i­ty, beau­ty, a dis­tinc­tive voice.

bloopers

There are a cou­ple of sim­ple rules to fol­low in order to make a great panora­ma.  The first is to rotate the phone in place instead of hold­ing it out at arm’s length and sweep­ing it.  This is espe­cial­ly impor­tant in indoor envi­ron­ments, where many sur­faces are near­by and any move­ment of the focal point will result in images with dif­fer­ing per­spec­tives— which are much hard­er to stitch.  Admit­ted­ly it’s a bit awk­ward to do this.  In prac­tice it means doing a lit­tle dance around the phone— you orbit around it while it stays in place.  It helps to iden­ti­fy a land­mark on the ground and make sure the phone stays right over it.  If you want to do a full sphere, you’ll also have to point the phone down at the ground at some point, and if you don’t want your dis­em­bod­ied feet in there, you’ll have to back away from the phone and point it down care­ful­ly to avoid them.  You’ll look sil­ly, but the beau­ti­ful immer­sive pano will be worth it, right?

The oth­er rule is that when there are peo­ple in your pano, you want to try to get them to stay very still while you’re shoot­ing near them, or man­age your cap­ture in such a way that they only appear in a sin­gle pho­to.  Oth­er­wise, between the graph cut algo­rithm and the Helmholtz blend­ing, you’ll splinch your friends:

(The pano on the right is espe­cial­ly inter­est­ing.  Mike is wear­ing Heather’s legs.)

One final tip.  If you can, turn on “expo­sure lock” in the set­tings screen.  This will help the blend­ing.  With expo­sure lock off, the algo­rithms must do their best to blend shots tak­en with very dif­fer­ent expo­sure set­tings and col­or bal­ances, which will some­times leave arti­facts in spite of our best efforts.  It’s not always pos­si­ble to lock expo­sure, because in some panora­mas you’ll be shoot­ing both straight at the sun and into deep shad­ow.

what about windows phone 7?

I’m sure over the com­ing days and weeks we’ll be answer­ing, over and over, the “why didn’t this ship first on Microsoft’s own phone” ques­tion.  Our approach to the design of the Pho­to­synth app hope­ful­ly pro­vides some evi­dence that we very much think of Win­dows Phone 7 as brethren and inspi­ra­tion, not to men­tion proof that Microsoft can make beau­ti­ful things.  (Such a joy and a relief, after the pre­vi­ous gen­er­a­tion of Win­dows phones!)  If we could have shipped first on these devices, we would have.  But the lev­el of cam­era and low-lev­el algo­rith­mic hack­ing need­ed to make Pho­to­synth work meant that, if we want­ed to get this out as quick­ly as pos­si­ble— and we sure­ly did— we need­ed to do so on a plat­form that pro­vid­ed the nec­es­sary low-lev­el device access.  Win­dows Phone 7 doesn’t yet allow this for apps.  It will soon.  It’s worth keep­ing in mind that the first sev­er­al gen­er­a­tions of iPhone device and OS wouldn’t have allowed us to build this app either.  For now, iPhone’s plat­form matu­ri­ty— and of course the large num­ber of peo­ple with iPhones out there— meant that it made sense for us to go for it.

At Bing we’re always inter­est­ed in reach­ing as many peo­ple as pos­si­ble, which means we’ll always devel­op for mul­ti­ple plat­forms.  But over time, we’ll be doing more and more of our ear­ly inno­va­tion on the Win­dows Phone.

future

We hope that in addi­tion to being a good par­ty trick, the Pho­to­synth app will have lots of peo­ple record­ing places and events they want to remem­ber and share with friends.  And share with the world.  A major ele­ment in the larg­er vision of Pho­to­synth is to let many dif­fer­ent types of media con­nect togeth­er into a kind of shared “world tapes­try”.  I’ll be talk­ing about this, shar­ing both our think­ing and some of our lat­est work, at Where 2.0 on Wednes­day.  That might require anoth­er post.  It’s excit­ing, in any event, to be fur­ther­ing this sto­ry again, after a year spent most­ly on incu­bat­ing oth­er aspects of Bing Mobile.

This entry was posted in maps, mobile and tagged , , , , , , . Bookmark the permalink.

27 Responses to photosynth app!

  1. Pingback: Photosynth for WP7 « Go Microsoft !!!!!!!

  2. fred says:

    For the next ver­sion you should check out the work that Microsoft Research has done. They’ve also worked in this area.

  3. Pingback: My Microsoft Life » Photosynth: coming next to Windows Phone [Update]

  4. Great to see this app released — I’m look­ing for­ward to play­ing with it (as soon as I get some free time!)

    And con­grat­u­la­tions for mak­ing the deci­sion to release the app now on iPhone rather than wait for WP7. Microsoft makes great soft­ware. In this case, the right plat­form for that soft­ware hap­pens to be made by a dif­fer­ent com­pa­ny. Some would say a brave deci­sion, but it’s one that is based on sound tech­ni­cal rea­son­ing, rather than blind cor­po­rate loy­al­ty or mar­ket­ing hype. That demon­strates integri­ty, and can only increase respect for Microsoft.

  5. Pingback: Photosynth team explain their iPhone first decision: “ we needed … low-level device access”

  6. Pingback: Microsoft explains lack of Windows Phone 7 Photosynth app, promises one in future | WinRumors

  7. Pingback: Microsoft Photosynth promesso per il WP7 in un prossimo futuro | Windows Phone Magazine Italia il blog Italiano sul Microsoft Windows Phone 7

  8. Pingback: Don’t worry, Windows Phone will get the Photosynth app too, eventually

  9. jonny says:

    Do you real­ly think we (WP7 sup­port­ers) will see this kind of app in the NEAR future? As far as I know, there will be no Native Code avail­able in Man­go. If this is sole­ly because of the direct video stream... I would be dis­ap­point­ed. Why not take pic­tures and then stich them together...ofcourse, “live” is bet­ter but atleast give us some­thing?

    Btw. Isn’t Bing a part of Microsoft? How comes that not even you have prop­er access to the tools to make this hap­pen? It’s real­ly sad :(

  10. @Blaise, thanks for tak­ing the time to post pub­licly on a bit of the future.

    I’ve want­ed to ask: Have you con­sid­ered mak­ing Pho­to­synth apps for mobile gam­ing sys­tems like Nintendo’s 3DS or Sony’s NGP? I would also love to see Pho­to­synth view­ers (that load both panora­mas and synths) for the home con­soles: XBOX 360, Playsta­tion 3, and Wii. To be hon­est, I’d love Pho­to­synth view­er apps to be as per­va­sive as Net­flix.

    I’d also smile at see­ing stereo­scop­ic ads for you guys before 3D movies at the­atres.

    Last­ly, if you could give us an update on cam­era at Where 2.0 about where you guys are on synth link­ing and pano link­ing or when dense recon­struc­tion might ship, I will be very thank­ful. Tan­gen­tial­ly, I’d love to see the syn­ther updat­ed so that it reads full 360° panos stitched by ICE and export­ed as JPGs, so that I could make synths where each pho­to is actu­al­ly a 360° or even spher­i­cal pano, dis­played cor­rect­ly with­in the synth view­er.

    @Jonny, this will be a Man­go app.

    If you watch the MIX11 Day 2 Keynote, you can hear Joe Belfiore make the announce­ment about the sen­sor API that will be ship­ping in the Man­go devel­op­ment tools in May 2011.

    Here’s the keynote: http://channel9.msdn.com/Events/MIX/MIX11/KEY02
    (Pay atten­tion around 00:34:58)

    He also did a short recap with Lau­ra Foy after he got off stage, which you can watch here: http://t.co/Fqlf7VP (Sen­sor stuff from 00:07:50 — 00:10:50)

    You can watch for news of the avail­abil­i­ty of the Man­go devel­op­er tools at http://developer.windowsphone.com

    What this means, as I said at the begin­ning, is that we won’t see the Pho­to­synth app for Win­dows Phone before the Man­go update ships this fall.

    If you want an extra lit­tle inside con­fir­ma­tion, check this out:
    http://bit.ly/wp7synthmango

    • blaise says:

      Nate, great ques­tions, great sug­ges­tions (and use­ful info!). At Where 2.0 I’ll be talk­ing about some upcom­ing things very rel­e­vant to your points. In par­tic­u­lar with respect to view­ers, the emer­gence of next-gen­er­a­tion brows­er tech­nolo­gies will make panos and oth­er arti­facts acces­si­ble on many devices (they’ll all adopt HTML5 soon­er or lat­er). On the syn­thing side, we do need to make some of these things work as pure ser­vices, which opens up many devices/applications, though at the cost of real­time feed­back and dis­tri­b­u­tion of com­pute. (Real­time feed­back is pret­ty much essen­tial for mak­ing panos, unless one is stick­ing to “strip” or “loop” only, or using a fish­eye lens or motor­ized mount.) You make a num­ber of oth­er sug­ges­tions here too, which I’d be very hap­py to talk with you about on or off cam­era at Where 2.0.

    • Just a heads-up for any of my peers who read my last com­ment: The beta devel­op­er tools for cre­at­ing Man­go apps are up for any­one to down­load now, so I’m sure that the inter­nal teams have had them for a while now.

      Here are a few per­ti­nent links:
      Chan­nel 9 Video: http://t.co/w9dKizF
      App Hub: http://create.msdn.com/
      Blog: http://bit.ly/betamangotoolsblog
      Announce­ment: http://create.msdn.com/en-US/news/WPDT_7.1_Beta

  11. Sam Choi says:

    Hi Blaise,

    I still remem­ber you from the TED pre­sen­ta­tion, and I came across this post of yours from anoth­er Blog. Great job with the app! I tried it, and it is real­ly amaz­ing.

    To be bru­tal­ly hon­est, I think the most log­i­cal next-step for your team is to bring this app to the Android, not Win­dows Phone. The Android crowd is in the order of mag­ni­tudes larg­er than the Win­dows Phone crowd, and will con­tin­ue to remain so for a long, long time. It’s time to reach out to the mil­lions of Android users and show them what Bing is, and what it is capa­ble of.

    Bring­ing this app to Win­dows Phone is only preach­ing to the con­vert­ed.

    • blaise says:

      Thanks Sam! Re. Android, if our deci­sion were pure­ly based on mar­ket reach today, I’d agree. (Though if things go as we hope and expect, the cal­cu­lus will look dif­fer­ent!) That can’t be the only fac­tor, though. In addi­tion to lots of praise for the app, we’ve got­ten a cer­tain amount of trolling about not keep­ing our pow­der total­ly dry to release first on Win­Phone, and act­ing like “One Microsoft”. I’ll reit­er­ate that mar­ket reach or no mar­ket reach, we would have devel­oped first for Win­Phone if that had been tech­ni­cal­ly fea­si­ble with­out hold­ing the ship for months. It’s a del­i­cate trade­off, and one that caus­es angst no mat­ter which way we choose. It’ll be great not to have to keep mak­ing this par­tic­u­lar trade­off. We’re work­ing close­ly with Win­Phone to get the enabling capa­bil­i­ties into the plat­form.

  12. Pingback: Sites Thursday, 21 April 2011-Bing Read / Write World-Photosynth app-iPhone tracker map-Switch to Hotmail-Shift 2 Unleased tips and DLC « webDotWiz talks Windows Live

  13. Pingback: Windows Phone Dev Podcast » Episode 013 – “Ahead of Schedule”

  14. ricardo says:

    Hi Blaise,

    Con­grat­u­la­tions on this release. I’m a fan of Pho­to­synth. The appli­ca­tion for iOS looks great. One ques­tion, do you have any plans to release a REST­ful API to inte­grate with plat­forms such as HTML5 MCV3?

  15. mark k. says:

    I real­ly enjoy the app, now I’m wait­ing for the html5 view­er to actu­al­ly use it in order to embed the results on my blog ;)

    If the app could inte­grate some xml-rpc mag­ic to pub­lish direct­ly to blogs it will be even cool­er (just take some code from the win­dows live writer ....)

  16. Pingback: Photosynth goes mobile » learn | you | good - we're chained to the world and we've all gotta pull…

  17. Seth says:

    Blaise,

    Bra­vo! This is an amaz­ing app that puts the pow­er to the peo­ple. I man­age a grass­roots com­mu­ni­ty orga­ni­za­tion that pro­motes one of America’s largest urban his­toric dis­tricts which is under­go­ing some major rein­vest­ment (hun­dreds of mil­lions of dol­lars, thou­sands of new res­i­dents, dozens of busi­ness­es, etc.) and have been wait­ing for Google Streetview to update its images of the neigh­bor­hood because it’s old imagery is out of date. With Pho­to­synth and its con­nec­tions to Bing, I no longer have to wait. And, tourists to the neighborhood’s vast under­ground net­work of brew­ery tun­nels and lager­ing cel­lars can upload their panora­mas to the map too. Very excit­ing stuff!

    One ques­tion though, on the app could you add a way to con­nect to Twit­ter?

    Thanks again and bra­vo!

    • blaise says:

      Thank you Seth, very kind words! We’re for sure adding sup­port for more forms of shar­ing.. Twit­ter is on our list, but not sure if it’ll make v1.1. Which urban envi­ron­ment?

  18. Seth says:

    It’s Over-the-Rhine in Cincin­nati. Fea­tured in the NYT today on TR 11.

    Is there a way to part­ner with Bing/Photosynth like the Google Streetview Part­ner Project?

  19. Pingback: Quora

  20. Daniel says:

    Great app, been using it for a while and I love it.

    My expe­ri­ence is that in order to get excel­lent results in low light indoor con­di­tions a stan­dard tri­pod is a must and makes the process faster cause there’s almost no need for man­u­al shots.

    Of course I had to fig­ure out how to attach my iPhone firm­ly to the tri­pod (with­out spend­ing mon­ey) !

    Regards.

Leave a Reply

Your email address will not be published. Required fields are marked *