photosynth app!

As every­one who’s been glanc­ing now and then at my face­book page knows, I’ve been post­ing a lot of panora­mas lately from the mys­te­ri­ous “Pho­to­synth pub­lish pho­tos” app.  Well, the app is now finally avail­able on the iPhone— just look in the app store for “pho­to­synth” or “bing” (Apple’s index looks like it’s rebuild­ing now, hope­fully by the time I put this post up it’ll be fully live).

I’m very excited to finally have this app out there.  The team’s done a won­der­ful job on it.  Here’s the offi­cial blog post.  They also shot a very nice release video, which to my cha­grin ends with y.t. pon­tif­i­cat­ing on a posthu­man future.

A few notes on this evo­lu­tion in the Pho­to­synth and Bing Mobile story, in no par­tic­u­lar order.

expe­ri­ence

This app is a lot of fun to use, and the out­put is— I think— com­pelling.  It addresses a fun­da­men­tal lim­i­ta­tion of cam­eras as we now know them: field of view.  A phone’s cam­era is the right size and design for tak­ing a snap­shot of, say, someone’s face at a party.  But as any­one who has tried to put up real estate pho­tos knows well, try­ing to cap­ture a view out the win­dow with an ordi­nary cam­era is an awful expe­ri­ence.  With your eyes, you see a whole for­est; in the cam­era frame, you see a cou­ple of trees.  With Pho­to­synth, you rotate the cam­era to take in the whole view, and the app fuses that view together:

Of course there’s an inher­ent dif­fi­culty in tak­ing these wide fields of view and pro­ject­ing them down into pla­nar images.  It’s the­o­ret­i­cally pos­si­ble to do this with a per­spec­tive pro­jec­tion when the field of view is less than half a sphere, though in prac­tice the image starts to dis­tort unpleas­antly when the larger axis exceeds 60 degrees or so.  Remem­ber that in our own eyes, the retina is hemi­spher­i­cal, not pla­nar.  Our very wide nat­ural field of view doesn’t rely on a rec­tan­gu­lar pro­jec­tion the way film or dig­i­tal cam­eras do.

This is why, when Pho­to­synth pub­lishes pho­tos to a flat medium, like the inline news feed in face­book or the flat image above, it uses a spher­i­cal pro­jec­tion, which results in a dis­torted image.  Straight lines turn into arcs.  This kind of imagery can be quite beau­ti­ful in its own right, though it can get a bit coun­ter­in­tu­itive, espe­cially as the field of view grows all the way to the full sphere.

Pro­ject­ing the sphere down to a rec­tan­gu­lar image is of course just what one does when one makes a flat map of the Earth.  We’ve all seen pla­nar world maps so often that we think of them as “nor­mal”, while the above image looks “dis­torted”, although really they exhibit the same dis­tor­tions.  Yes, Green­land is big— about three times the size of Texas.  Not twice the size of the whole USA.

The best way to expe­ri­ence a panorama is in an immer­sive viewer, which repro­jects the imagery inter­ac­tively into a smaller win­dow, allow­ing you to rotate.  We’re work­ing on view­ers that let these things hap­pen in native Web-ese, though it requires advanced browsers (HTML5, Can­vas or CSS3).  In the mean­time, it can be done with Sil­verlight.  Here’s the panorama of the Neue Nation­al­ga­lerie above, ren­dered this way:

tech­nol­ogy

In the past few years there has been a grow­ing trickle of “smart” cam­eras and phone apps for stitch­ing together pho­tos or video into panora­mas.  Many of them aren’t par­tic­u­larly good, but I do need to give a shout out to our friends at Occip­i­tal, whose 360 panorama app was an inspi­ra­tion to the team.

As far as I know, ours is the first app that goes beyond “strip” panorama mak­ing to allow cov­er­age of the entire visual sphere with real­time track­ing.  This relies on some pretty cutting-edge com­puter vision hack­ing (thank you Georg).  Extract­ing fea­tures from the video stream on the fly, fol­low­ing them from frame to frame, and mod­el­ing the envi­ron­ment in real­time is hard­core stuff, and requires not only high-performance algo­rithms, but also very aggres­sive low-level opti­miza­tion and trick­ery with the cam­era and graph­ics pipelines.  Fusion on the sphere isn’t all the way to full 3D mod­el­ing, but it already involves its share of topo­log­i­cal com­pli­ca­tions and trade­offs between local and global recon­struc­tion.  By all means, dear CS grads, try this at home (and send us your screen­shots)— but be fore­warned, there’s a rea­son why we haven’t seen a mobile app like this before in the mar­ket!  This is a bit like Doom first appear­ing on the PC in 1993.  Now that it’s been shown to be pos­si­ble, we should expect quite a few fol­low­ers, and a rapid evo­lu­tion in real­time com­puter vision on mobile devices.

design

There’s another rea­son I’m very pleased by Pho­to­synth for iPhone: it’s the first rea­son­ably com­plete appli­ca­tion of our design sys­tem to a non-Windows phone.  We’ve been work­ing for more than a year with the very beau­ti­ful design lan­guage cre­ated by the Enter­tain­ment and Devices peo­ple at Microsoft (yes, Microsoft can do beau­ti­ful design!  It’s true!) for Xbox, Zune, and Win­dows Phone 7, code­named Metro.  The “laven­der” map style we released last year is a car­to­graphic embod­i­ment of this lan­guage.  Trans­lat­ing Metro for an envi­ron­ment like iPhone, in which there’s a strong native look, feel and inter­ac­tion model, is risky busi­ness.  Done poorly, the result is con­fus­ing and incon­gru­ous.  Some would argue that an app should always adopt native con­trols, look and feel, tai­lor­ing itself entirely to the host plat­form to min­i­mize cog­ni­tive dis­so­nance.  This is an old argu­ment.  I remem­ber it from the X win­dows days, and ear­lier.  (More recently, Apple’s first release of Safari on the PC gar­nered much crit­i­cism for its dis­so­nant non-PC look and feel.  Today they’ve moved closer to native.)

I think from this per­spec­tive the Pho­to­synth app is a great suc­cess.  Its inter­ac­tion model, look and feel are very Metro, dis­tinc­tively ours, yet it works in the iPhone con­text.  It has a dis­tinct voice, yet remains trans­par­ent and usable.  It isn’t anti­so­cial in its approach to the plat­form.  I was delighted to read this post Adri­enne found on Giz­modo (unchar­ac­ter­is­tic read­ing mate­r­ial for her)—

Also, Microsoft, who as you may know makes their own mobile phone OS these days, has puck­ishly brought the Win­dows Phone 7 aes­thetic to the iPhone app, which, man, is just really really nice. You don’t realise how, I don’t know, corny all these bev­elled but­tons and 3D ani­ma­tions are until you see Microsoft’s flat, geo­met­ric UI on your iPhone’s dis­play... More apps that look like this, please.

One of the things that I think makes this eas­ier to do on mobile devices than on PCs with win­dow­ing sys­tems is the fact that mobile apps are always full-screen.  They can cre­ate self-contained worlds by book­end­ing an expe­ri­ence in time.  This works for Web pages too, because the Web has evolved from the world of “con­tent” instead of “appli­ca­tion”— one wouldn’t com­plain about dis­tinc­tive design lan­guage on a web­site, any more than one would com­plain about incon­sis­tent fonts on the cov­ers of dif­fer­ent mag­a­zines lined up on a rack.  For­tu­nately, between web­sites and mobile apps, we have the next decade pretty much covered.

Here’s one of my favorite screens from the Pho­to­synth app:

The typog­ra­phy is beau­ti­ful and clean, the visual bal­ance rem­i­nis­cent of mid-20th cen­tury Swiss design.  Sur­faces are flat, cor­ners square, and any con­tent shown is authen­tic, not just rep­re­sen­ta­tive.  That unstitched pano is a lovely arti­fact in its own right, a bit of a nod to David Hockney’s join­ers.  I hope that with this app we prove that we can have our cake and eat it too— usabil­ity, beauty, a dis­tinc­tive voice.

bloop­ers

There are a cou­ple of sim­ple rules to fol­low in order to make a great panorama.  The first is to rotate the phone in place instead of hold­ing it out at arm’s length and sweep­ing it.  This is espe­cially impor­tant in indoor envi­ron­ments, where many sur­faces are nearby and any move­ment of the focal point will result in images with dif­fer­ing per­spec­tives— which are much harder to stitch.  Admit­tedly it’s a bit awk­ward to do this.  In prac­tice it means doing a lit­tle dance around the phone— you orbit around it while it stays in place.  It helps to iden­tify a land­mark on the ground and make sure the phone stays right over it.  If you want to do a full sphere, you’ll also have to point the phone down at the ground at some point, and if you don’t want your dis­em­bod­ied feet in there, you’ll have to back away from the phone and point it down care­fully to avoid them.  You’ll look silly, but the beau­ti­ful immer­sive pano will be worth it, right?

The other rule is that when there are peo­ple in your pano, you want to try to get them to stay very still while you’re shoot­ing near them, or man­age your cap­ture in such a way that they only appear in a sin­gle photo.  Oth­er­wise, between the graph cut algo­rithm and the Helmholtz blend­ing, you’ll splinch your friends:

(The pano on the right is espe­cially inter­est­ing.  Mike is wear­ing Heather’s legs.)

One final tip.  If you can, turn on “expo­sure lock” in the set­tings screen.  This will help the blend­ing.  With expo­sure lock off, the algo­rithms must do their best to blend shots taken with very dif­fer­ent expo­sure set­tings and color bal­ances, which will some­times leave arti­facts in spite of our best efforts.  It’s not always pos­si­ble to lock expo­sure, because in some panora­mas you’ll be shoot­ing both straight at the sun and into deep shadow.

what about win­dows phone 7?

I’m sure over the com­ing days and weeks we’ll be answer­ing, over and over, the “why didn’t this ship first on Microsoft’s own phone” ques­tion.  Our approach to the design of the Pho­to­synth app hope­fully pro­vides some evi­dence that we very much think of Win­dows Phone 7 as brethren and inspi­ra­tion, not to men­tion proof that Microsoft can make beau­ti­ful things.  (Such a joy and a relief, after the pre­vi­ous gen­er­a­tion of Win­dows phones!)  If we could have shipped first on these devices, we would have.  But the level of cam­era and low-level algo­rith­mic hack­ing needed to make Pho­to­synth work meant that, if we wanted to get this out as quickly as pos­si­ble— and we surely did— we needed to do so on a plat­form that pro­vided the nec­es­sary low-level device access.  Win­dows Phone 7 doesn’t yet allow this for apps.  It will soon.  It’s worth keep­ing in mind that the first sev­eral gen­er­a­tions of iPhone device and OS wouldn’t have allowed us to build this app either.  For now, iPhone’s plat­form matu­rity— and of course the large num­ber of peo­ple with iPhones out there— meant that it made sense for us to go for it.

At Bing we’re always inter­ested in reach­ing as many peo­ple as pos­si­ble, which means we’ll always develop for mul­ti­ple plat­forms.  But over time, we’ll be doing more and more of our early inno­va­tion on the Win­dows Phone.

future

We hope that in addi­tion to being a good party trick, the Pho­to­synth app will have lots of peo­ple record­ing places and events they want to remem­ber and share with friends.  And share with the world.  A major ele­ment in the larger vision of Pho­to­synth is to let many dif­fer­ent types of media con­nect together into a kind of shared “world tapes­try”.  I’ll be talk­ing about this, shar­ing both our think­ing and some of our lat­est work, at Where 2.0 on Wednes­day.  That might require another post.  It’s excit­ing, in any event, to be fur­ther­ing this story again, after a year spent mostly on incu­bat­ing other aspects of Bing Mobile.


This entry was posted in maps, mobile and tagged , , , , , , . Bookmark the permalink.

27 Responses to photosynth app!

  1. Pingback: Photosynth for WP7 « Go Microsoft !!!!!!!

  2. fred says:

    For the next ver­sion you should check out the work that Microsoft Research has done. They’ve also worked in this area.

  3. Pingback: My Microsoft Life » Photosynth: coming next to Windows Phone [Update]

  4. Great to see this app released — I’m look­ing for­ward to play­ing with it (as soon as I get some free time!)

    And con­grat­u­la­tions for mak­ing the deci­sion to release the app now on iPhone rather than wait for WP7. Microsoft makes great soft­ware. In this case, the right plat­form for that soft­ware hap­pens to be made by a dif­fer­ent com­pany. Some would say a brave deci­sion, but it’s one that is based on sound tech­ni­cal rea­son­ing, rather than blind cor­po­rate loy­alty or mar­ket­ing hype. That demon­strates integrity, and can only increase respect for Microsoft.

  5. Pingback: Photosynth team explain their iPhone first decision: “ we needed … low-level device access”

  6. Pingback: Microsoft explains lack of Windows Phone 7 Photosynth app, promises one in future | WinRumors

  7. Pingback: Microsoft Photosynth promesso per il WP7 in un prossimo futuro | Windows Phone Magazine Italia il blog Italiano sul Microsoft Windows Phone 7

  8. Pingback: Don’t worry, Windows Phone will get the Photosynth app too, eventually

  9. jonny says:

    Do you really think we (WP7 sup­port­ers) will see this kind of app in the NEAR future? As far as I know, there will be no Native Code avail­able in Mango. If this is solely because of the direct video stream... I would be dis­ap­pointed. Why not take pic­tures and then stich them together...ofcourse, “live” is bet­ter but atleast give us something?

    Btw. Isn’t Bing a part of Microsoft? How comes that not even you have proper access to the tools to make this hap­pen? It’s really sad :(

  10. @Blaise, thanks for tak­ing the time to post pub­licly on a bit of the future.

    I’ve wanted to ask: Have you con­sid­ered mak­ing Pho­to­synth apps for mobile gam­ing sys­tems like Nintendo’s 3DS or Sony’s NGP? I would also love to see Pho­to­synth view­ers (that load both panora­mas and synths) for the home con­soles: XBOX 360, Playsta­tion 3, and Wii. To be hon­est, I’d love Pho­to­synth viewer apps to be as per­va­sive as Netflix.

    I’d also smile at see­ing stereo­scopic ads for you guys before 3D movies at theatres.

    Lastly, if you could give us an update on cam­era at Where 2.0 about where you guys are on synth link­ing and pano link­ing or when dense recon­struc­tion might ship, I will be very thank­ful. Tan­gen­tially, I’d love to see the syn­ther updated so that it reads full 360° panos stitched by ICE and exported as JPGs, so that I could make synths where each photo is actu­ally a 360° or even spher­i­cal pano, dis­played cor­rectly within the synth viewer.

    @Jonny, this will be a Mango app.

    If you watch the MIX11 Day 2 Keynote, you can hear Joe Belfiore make the announce­ment about the sen­sor API that will be ship­ping in the Mango devel­op­ment tools in May 2011.

    Here’s the keynote: http://channel9.msdn.com/Events/MIX/MIX11/KEY02
    (Pay atten­tion around 00:34:58)

    He also did a short recap with Laura Foy after he got off stage, which you can watch here: http://t.co/Fqlf7VP (Sen­sor stuff from 00:07:50 — 00:10:50)

    You can watch for news of the avail­abil­ity of the Mango devel­oper tools at http://developer.windowsphone.com

    What this means, as I said at the begin­ning, is that we won’t see the Pho­to­synth app for Win­dows Phone before the Mango update ships this fall.

    If you want an extra lit­tle inside con­fir­ma­tion, check this out:
    http://bit.ly/wp7synthmango

    • blaise says:

      Nate, great ques­tions, great sug­ges­tions (and use­ful info!). At Where 2.0 I’ll be talk­ing about some upcom­ing things very rel­e­vant to your points. In par­tic­u­lar with respect to view­ers, the emer­gence of next-generation browser tech­nolo­gies will make panos and other arti­facts acces­si­ble on many devices (they’ll all adopt HTML5 sooner or later). On the syn­thing side, we do need to make some of these things work as pure ser­vices, which opens up many devices/applications, though at the cost of real­time feed­back and dis­tri­b­u­tion of com­pute. (Real­time feed­back is pretty much essen­tial for mak­ing panos, unless one is stick­ing to “strip” or “loop” only, or using a fish­eye lens or motor­ized mount.) You make a num­ber of other sug­ges­tions here too, which I’d be very happy to talk with you about on or off cam­era at Where 2.0.

    • Just a heads-up for any of my peers who read my last com­ment: The beta devel­oper tools for cre­at­ing Mango apps are up for any­one to down­load now, so I’m sure that the inter­nal teams have had them for a while now.

      Here are a few per­ti­nent links:
      Chan­nel 9 Video: http://t.co/w9dKizF
      App Hub: http://create.msdn.com/
      Blog: http://bit.ly/betamangotoolsblog
      Announce­ment: http://create.msdn.com/en-US/news/WPDT_7.1_Beta

  11. Sam Choi says:

    Hi Blaise,

    I still remem­ber you from the TED pre­sen­ta­tion, and I came across this post of yours from another Blog. Great job with the app! I tried it, and it is really amazing.

    To be bru­tally hon­est, I think the most log­i­cal next-step for your team is to bring this app to the Android, not Win­dows Phone. The Android crowd is in the order of mag­ni­tudes larger than the Win­dows Phone crowd, and will con­tinue to remain so for a long, long time. It’s time to reach out to the mil­lions of Android users and show them what Bing is, and what it is capa­ble of.

    Bring­ing this app to Win­dows Phone is only preach­ing to the converted.

    • blaise says:

      Thanks Sam! Re. Android, if our deci­sion were purely based on mar­ket reach today, I’d agree. (Though if things go as we hope and expect, the cal­cu­lus will look dif­fer­ent!) That can’t be the only fac­tor, though. In addi­tion to lots of praise for the app, we’ve got­ten a cer­tain amount of trolling about not keep­ing our pow­der totally dry to release first on Win­Phone, and act­ing like “One Microsoft”. I’ll reit­er­ate that mar­ket reach or no mar­ket reach, we would have devel­oped first for Win­Phone if that had been tech­ni­cally fea­si­ble with­out hold­ing the ship for months. It’s a del­i­cate trade­off, and one that causes angst no mat­ter which way we choose. It’ll be great not to have to keep mak­ing this par­tic­u­lar trade­off. We’re work­ing closely with Win­Phone to get the enabling capa­bil­i­ties into the platform.

  12. Pingback: Sites Thursday, 21 April 2011-Bing Read / Write World-Photosynth app-iPhone tracker map-Switch to Hotmail-Shift 2 Unleased tips and DLC « webDotWiz talks Windows Live

  13. Pingback: Windows Phone Dev Podcast » Episode 013 – “Ahead of Schedule”

  14. ricardo says:

    Hi Blaise,

    Con­grat­u­la­tions on this release. I’m a fan of Pho­to­synth. The appli­ca­tion for iOS looks great. One ques­tion, do you have any plans to release a REST­ful API to inte­grate with plat­forms such as HTML5 MCV3?

  15. mark k. says:

    I really enjoy the app, now I’m wait­ing for the html5 viewer to actu­ally use it in order to embed the results on my blog ;)

    If the app could inte­grate some xml-rpc magic to pub­lish directly to blogs it will be even cooler (just take some code from the win­dows live writer ....)

  16. Pingback: Photosynth goes mobile » learn | you | good - we're chained to the world and we've all gotta pull…

  17. Seth says:

    Blaise,

    Bravo! This is an amaz­ing app that puts the power to the peo­ple. I man­age a grass­roots com­mu­nity orga­ni­za­tion that pro­motes one of America’s largest urban his­toric dis­tricts which is under­go­ing some major rein­vest­ment (hun­dreds of mil­lions of dol­lars, thou­sands of new res­i­dents, dozens of busi­nesses, etc.) and have been wait­ing for Google Streetview to update its images of the neigh­bor­hood because it’s old imagery is out of date. With Pho­to­synth and its con­nec­tions to Bing, I no longer have to wait. And, tourists to the neighborhood’s vast under­ground net­work of brew­ery tun­nels and lager­ing cel­lars can upload their panora­mas to the map too. Very excit­ing stuff!

    One ques­tion though, on the app could you add a way to con­nect to Twitter?

    Thanks again and bravo!

    • blaise says:

      Thank you Seth, very kind words! We’re for sure adding sup­port for more forms of shar­ing.. Twit­ter is on our list, but not sure if it’ll make v1.1. Which urban environment?

  18. Seth says:

    It’s Over-the-Rhine in Cincin­nati. Fea­tured in the NYT today on TR 11.

    Is there a way to part­ner with Bing/Photosynth like the Google Streetview Part­ner Project?

  19. Pingback: Quora

  20. Daniel says:

    Great app, been using it for a while and I love it.

    My expe­ri­ence is that in order to get excel­lent results in low light indoor con­di­tions a stan­dard tri­pod is a must and makes the process faster cause there’s almost no need for man­ual shots.

    Of course I had to fig­ure out how to attach my iPhone firmly to the tri­pod (with­out spend­ing money) !

    Regards.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>