photosynth app!

As everyone who’s been glancing now and then at my facebook page knows, I’ve been posting a lot of panoramas lately from the mysterious “Photosynth publish photos” app.  Well, the app is now finally available on the iPhone— just look in the app store for “photosynth” or “bing” (Apple’s index looks like it’s rebuilding now, hopefully by the time I put this post up it’ll be fully live).

I’m very excited to finally have this app out there.  The team’s done a wonderful job on it.  Here’s the official blog post.  They also shot a very nice release video, which to my chagrin ends with y.t. pontificating on a posthuman future.

A few notes on this evolution in the Photosynth and Bing Mobile story, in no particular order.


This app is a lot of fun to use, and the output is— I think— compelling.  It addresses a fundamental limitation of cameras as we now know them: field of view.  A phone’s camera is the right size and design for taking a snapshot of, say, someone’s face at a party.  But as anyone who has tried to put up real estate photos knows well, trying to capture a view out the window with an ordinary camera is an awful experience.  With your eyes, you see a whole forest; in the camera frame, you see a couple of trees.  With Photosynth, you rotate the camera to take in the whole view, and the app fuses that view together:

Of course there’s an inherent difficulty in taking these wide fields of view and projecting them down into planar images.  It’s theoretically possible to do this with a perspective projection when the field of view is less than half a sphere, though in practice the image starts to distort unpleasantly when the larger axis exceeds 60 degrees or so.  Remember that in our own eyes, the retina is hemispherical, not planar.  Our very wide natural field of view doesn’t rely on a rectangular projection the way film or digital cameras do.

This is why, when Photosynth publishes photos to a flat medium, like the inline news feed in facebook or the flat image above, it uses a spherical projection, which results in a distorted image.  Straight lines turn into arcs.  This kind of imagery can be quite beautiful in its own right, though it can get a bit counterintuitive, especially as the field of view grows all the way to the full sphere.

Projecting the sphere down to a rectangular image is of course just what one does when one makes a flat map of the Earth.  We’ve all seen planar world maps so often that we think of them as “normal”, while the above image looks “distorted”, although really they exhibit the same distortions.  Yes, Greenland is big— about three times the size of Texas.  Not twice the size of the whole USA.

The best way to experience a panorama is in an immersive viewer, which reprojects the imagery interactively into a smaller window, allowing you to rotate.  We’re working on viewers that let these things happen in native Web-ese, though it requires advanced browsers (HTML5, Canvas or CSS3).  In the meantime, it can be done with Silverlight.  Here’s the panorama of the Neue Nationalgalerie above, rendered this way:


In the past few years there has been a growing trickle of “smart” cameras and phone apps for stitching together photos or video into panoramas.  Many of them aren’t particularly good, but I do need to give a shout out to our friends at Occipital, whose 360 panorama app was an inspiration to the team.

As far as I know, ours is the first app that goes beyond “strip” panorama making to allow coverage of the entire visual sphere with realtime tracking.  This relies on some pretty cutting-edge computer vision hacking (thank you Georg).  Extracting features from the video stream on the fly, following them from frame to frame, and modeling the environment in realtime is hardcore stuff, and requires not only high-performance algorithms, but also very aggressive low-level optimization and trickery with the camera and graphics pipelines.  Fusion on the sphere isn’t all the way to full 3D modeling, but it already involves its share of topological complications and tradeoffs between local and global reconstruction.  By all means, dear CS grads, try this at home (and send us your screenshots)— but be forewarned, there’s a reason why we haven’t seen a mobile app like this before in the market!  This is a bit like Doom first appearing on the PC in 1993.  Now that it’s been shown to be possible, we should expect quite a few followers, and a rapid evolution in realtime computer vision on mobile devices.


There’s another reason I’m very pleased by Photosynth for iPhone: it’s the first reasonably complete application of our design system to a non-Windows phone.  We’ve been working for more than a year with the very beautiful design language created by the Entertainment and Devices people at Microsoft (yes, Microsoft can do beautiful design!  It’s true!) for Xbox, Zune, and Windows Phone 7, codenamed Metro.  The “lavender” map style we released last year is a cartographic embodiment of this language.  Translating Metro for an environment like iPhone, in which there’s a strong native look, feel and interaction model, is risky business.  Done poorly, the result is confusing and incongruous.  Some would argue that an app should always adopt native controls, look and feel, tailoring itself entirely to the host platform to minimize cognitive dissonance.  This is an old argument.  I remember it from the X windows days, and earlier.  (More recently, Apple’s first release of Safari on the PC garnered much criticism for its dissonant non-PC look and feel.  Today they’ve moved closer to native.)

I think from this perspective the Photosynth app is a great success.  Its interaction model, look and feel are very Metro, distinctively ours, yet it works in the iPhone context.  It has a distinct voice, yet remains transparent and usable.  It isn’t antisocial in its approach to the platform.  I was delighted to read this post Adrienne found on Gizmodo (uncharacteristic reading material for her)—

Also, Microsoft, who as you may know makes their own mobile phone OS these days, has puckishly brought the Windows Phone 7 aesthetic to the iPhone app, which, man, is just really really nice. You don’t realise how, I don’t know, corny all these bevelled buttons and 3D animations are until you see Microsoft’s flat, geometric UI on your iPhone’s display… More apps that look like this, please.

One of the things that I think makes this easier to do on mobile devices than on PCs with windowing systems is the fact that mobile apps are always full-screen.  They can create self-contained worlds by bookending an experience in time.  This works for Web pages too, because the Web has evolved from the world of “content” instead of “application”— one wouldn’t complain about distinctive design language on a website, any more than one would complain about inconsistent fonts on the covers of different magazines lined up on a rack.  Fortunately, between websites and mobile apps, we have the next decade pretty much covered.

Here’s one of my favorite screens from the Photosynth app:

The typography is beautiful and clean, the visual balance reminiscent of mid-20th century Swiss design.  Surfaces are flat, corners square, and any content shown is authentic, not just representative.  That unstitched pano is a lovely artifact in its own right, a bit of a nod to David Hockney’s joiners.  I hope that with this app we prove that we can have our cake and eat it too— usability, beauty, a distinctive voice.


There are a couple of simple rules to follow in order to make a great panorama.  The first is to rotate the phone in place instead of holding it out at arm’s length and sweeping it.  This is especially important in indoor environments, where many surfaces are nearby and any movement of the focal point will result in images with differing perspectives— which are much harder to stitch.  Admittedly it’s a bit awkward to do this.  In practice it means doing a little dance around the phone— you orbit around it while it stays in place.  It helps to identify a landmark on the ground and make sure the phone stays right over it.  If you want to do a full sphere, you’ll also have to point the phone down at the ground at some point, and if you don’t want your disembodied feet in there, you’ll have to back away from the phone and point it down carefully to avoid them.  You’ll look silly, but the beautiful immersive pano will be worth it, right?

The other rule is that when there are people in your pano, you want to try to get them to stay very still while you’re shooting near them, or manage your capture in such a way that they only appear in a single photo.  Otherwise, between the graph cut algorithm and the Helmholtz blending, you’ll splinch your friends:

(The pano on the right is especially interesting.  Mike is wearing Heather’s legs.)

One final tip.  If you can, turn on “exposure lock” in the settings screen.  This will help the blending.  With exposure lock off, the algorithms must do their best to blend shots taken with very different exposure settings and color balances, which will sometimes leave artifacts in spite of our best efforts.  It’s not always possible to lock exposure, because in some panoramas you’ll be shooting both straight at the sun and into deep shadow.

what about windows phone 7?

I’m sure over the coming days and weeks we’ll be answering, over and over, the “why didn’t this ship first on Microsoft’s own phone” question.  Our approach to the design of the Photosynth app hopefully provides some evidence that we very much think of Windows Phone 7 as brethren and inspiration, not to mention proof that Microsoft can make beautiful things.  (Such a joy and a relief, after the previous generation of Windows phones!)  If we could have shipped first on these devices, we would have.  But the level of camera and low-level algorithmic hacking needed to make Photosynth work meant that, if we wanted to get this out as quickly as possible— and we surely did— we needed to do so on a platform that provided the necessary low-level device access.  Windows Phone 7 doesn’t yet allow this for apps.  It will soon.  It’s worth keeping in mind that the first several generations of iPhone device and OS wouldn’t have allowed us to build this app either.  For now, iPhone’s platform maturity— and of course the large number of people with iPhones out there— meant that it made sense for us to go for it.

At Bing we’re always interested in reaching as many people as possible, which means we’ll always develop for multiple platforms.  But over time, we’ll be doing more and more of our early innovation on the Windows Phone.


We hope that in addition to being a good party trick, the Photosynth app will have lots of people recording places and events they want to remember and share with friends.  And share with the world.  A major element in the larger vision of Photosynth is to let many different types of media connect together into a kind of shared “world tapestry”.  I’ll be talking about this, sharing both our thinking and some of our latest work, at Where 2.0 on Wednesday.  That might require another post.  It’s exciting, in any event, to be furthering this story again, after a year spent mostly on incubating other aspects of Bing Mobile.

This entry was posted in maps, mobile and tagged , , , , , , . Bookmark the permalink.

27 Responses to photosynth app!

  1. Pingback: Photosynth for WP7 « Go Microsoft !!!!!!!

  2. fred says:

    For the next version you should check out the work that Microsoft Research has done. They’ve also worked in this area.

  3. Pingback: My Microsoft Life » Photosynth: coming next to Windows Phone [Update]

  4. Great to see this app released – I’m looking forward to playing with it (as soon as I get some free time!)

    And congratulations for making the decision to release the app now on iPhone rather than wait for WP7. Microsoft makes great software. In this case, the right platform for that software happens to be made by a different company. Some would say a brave decision, but it’s one that is based on sound technical reasoning, rather than blind corporate loyalty or marketing hype. That demonstrates integrity, and can only increase respect for Microsoft.

  5. Pingback: Photosynth team explain their iPhone first decision: “ we needed … low-level device access”

  6. Pingback: Microsoft explains lack of Windows Phone 7 Photosynth app, promises one in future | WinRumors

  7. Pingback: Microsoft Photosynth promesso per il WP7 in un prossimo futuro | Windows Phone Magazine Italia il blog Italiano sul Microsoft Windows Phone 7

  8. Pingback: Don’t worry, Windows Phone will get the Photosynth app too, eventually

  9. jonny says:

    Do you really think we (WP7 supporters) will see this kind of app in the NEAR future? As far as I know, there will be no Native Code available in Mango. If this is solely because of the direct video stream… I would be disappointed. Why not take pictures and then stich them together…ofcourse, “live” is better but atleast give us something?

    Btw. Isn’t Bing a part of Microsoft? How comes that not even you have proper access to the tools to make this happen? It’s really sad :(

  10. @Blaise, thanks for taking the time to post publicly on a bit of the future.

    I’ve wanted to ask: Have you considered making Photosynth apps for mobile gaming systems like Nintendo’s 3DS or Sony’s NGP? I would also love to see Photosynth viewers (that load both panoramas and synths) for the home consoles: XBOX 360, Playstation 3, and Wii. To be honest, I’d love Photosynth viewer apps to be as pervasive as Netflix.

    I’d also smile at seeing stereoscopic ads for you guys before 3D movies at theatres.

    Lastly, if you could give us an update on camera at Where 2.0 about where you guys are on synth linking and pano linking or when dense reconstruction might ship, I will be very thankful. Tangentially, I’d love to see the synther updated so that it reads full 360° panos stitched by ICE and exported as JPGs, so that I could make synths where each photo is actually a 360° or even spherical pano, displayed correctly within the synth viewer.

    @Jonny, this will be a Mango app.

    If you watch the MIX11 Day 2 Keynote, you can hear Joe Belfiore make the announcement about the sensor API that will be shipping in the Mango development tools in May 2011.

    Here’s the keynote:
    (Pay attention around 00:34:58)

    He also did a short recap with Laura Foy after he got off stage, which you can watch here: (Sensor stuff from 00:07:50 – 00:10:50)

    You can watch for news of the availability of the Mango developer tools at

    What this means, as I said at the beginning, is that we won’t see the Photosynth app for Windows Phone before the Mango update ships this fall.

    If you want an extra little inside confirmation, check this out:

    • blaise says:

      Nate, great questions, great suggestions (and useful info!). At Where 2.0 I’ll be talking about some upcoming things very relevant to your points. In particular with respect to viewers, the emergence of next-generation browser technologies will make panos and other artifacts accessible on many devices (they’ll all adopt HTML5 sooner or later). On the synthing side, we do need to make some of these things work as pure services, which opens up many devices/applications, though at the cost of realtime feedback and distribution of compute. (Realtime feedback is pretty much essential for making panos, unless one is sticking to “strip” or “loop” only, or using a fisheye lens or motorized mount.) You make a number of other suggestions here too, which I’d be very happy to talk with you about on or off camera at Where 2.0.

    • Just a heads-up for any of my peers who read my last comment: The beta developer tools for creating Mango apps are up for anyone to download now, so I’m sure that the internal teams have had them for a while now.

      Here are a few pertinent links:
      Channel 9 Video:
      App Hub:

  11. Sam Choi says:

    Hi Blaise,

    I still remember you from the TED presentation, and I came across this post of yours from another Blog. Great job with the app! I tried it, and it is really amazing.

    To be brutally honest, I think the most logical next-step for your team is to bring this app to the Android, not Windows Phone. The Android crowd is in the order of magnitudes larger than the Windows Phone crowd, and will continue to remain so for a long, long time. It’s time to reach out to the millions of Android users and show them what Bing is, and what it is capable of.

    Bringing this app to Windows Phone is only preaching to the converted.

    • blaise says:

      Thanks Sam! Re. Android, if our decision were purely based on market reach today, I’d agree. (Though if things go as we hope and expect, the calculus will look different!) That can’t be the only factor, though. In addition to lots of praise for the app, we’ve gotten a certain amount of trolling about not keeping our powder totally dry to release first on WinPhone, and acting like “One Microsoft”. I’ll reiterate that market reach or no market reach, we would have developed first for WinPhone if that had been technically feasible without holding the ship for months. It’s a delicate tradeoff, and one that causes angst no matter which way we choose. It’ll be great not to have to keep making this particular tradeoff. We’re working closely with WinPhone to get the enabling capabilities into the platform.

  12. Pingback: Sites Thursday, 21 April 2011-Bing Read / Write World-Photosynth app-iPhone tracker map-Switch to Hotmail-Shift 2 Unleased tips and DLC « webDotWiz talks Windows Live

  13. Pingback: Windows Phone Dev Podcast » Episode 013 – “Ahead of Schedule”

  14. ricardo says:

    Hi Blaise,

    Congratulations on this release. I’m a fan of Photosynth. The application for iOS looks great. One question, do you have any plans to release a RESTful API to integrate with platforms such as HTML5 MCV3?

  15. mark k. says:

    I really enjoy the app, now I’m waiting for the html5 viewer to actually use it in order to embed the results on my blog ;)

    If the app could integrate some xml-rpc magic to publish directly to blogs it will be even cooler (just take some code from the windows live writer ….)

  16. Pingback: Photosynth goes mobile » learn | you | good - we're chained to the world and we've all gotta pull…

  17. Seth says:


    Bravo! This is an amazing app that puts the power to the people. I manage a grassroots community organization that promotes one of America’s largest urban historic districts which is undergoing some major reinvestment (hundreds of millions of dollars, thousands of new residents, dozens of businesses, etc.) and have been waiting for Google Streetview to update its images of the neighborhood because it’s old imagery is out of date. With Photosynth and its connections to Bing, I no longer have to wait. And, tourists to the neighborhood’s vast underground network of brewery tunnels and lagering cellars can upload their panoramas to the map too. Very exciting stuff!

    One question though, on the app could you add a way to connect to Twitter?

    Thanks again and bravo!

    • blaise says:

      Thank you Seth, very kind words! We’re for sure adding support for more forms of sharing.. Twitter is on our list, but not sure if it’ll make v1.1. Which urban environment?

  18. Seth says:

    It’s Over-the-Rhine in Cincinnati. Featured in the NYT today on TR 11.

    Is there a way to partner with Bing/Photosynth like the Google Streetview Partner Project?

  19. Pingback: Quora

  20. Daniel says:

    Great app, been using it for a while and I love it.

    My experience is that in order to get excellent results in low light indoor conditions a standard tripod is a must and makes the process faster cause there’s almost no need for manual shots.

    Of course I had to figure out how to attach my iPhone firmly to the tripod (without spending money) !


Leave a Reply

Your email address will not be published. Required fields are marked *