Rebooting history.
April 15th, 2008I gave the keynote address yesterday at the annual meeting of the International Oracle Users’ Group (IOUG), part of the big enterprise database tech-fest that is Collaborate ‘08.
The topic of my talk was how the decreasing cost of storage and bandwidth, plus ubiquitous, simple search, are going to change the world. I made good use of Wired editor-in-chief Chris Anderson’s feature story Free! Why $0.00 Is the Future of Business, plus science fiction author Charlie Stross’s notion that history is just about to begin.
I’ve pasted my notes for the talk below (they’re rough). Also, you can get a list of links to my sources and related stories or download my slides: Rebooting history - IOUG keynote (PPT).
Rebooting History
How the coming wave of cheap storage, high bandwidth and infinite searchability will make us forget everything that went before
Notes for a talk April 14, 2008 at Collaborate ‘08 / IOUG Annual Meeting
by Dylan Tweney, senior editor, Wired.com
The Argentinean writer Borges has a story called “Funes the Memorious.” In this story, he describes a young man thrown from a horse who then discovers he has perfect memory — he can’t forget anything. Here’s what the man named Funes says about his experience:
“For nineteen years, he said, he had lived like a person in a dream: he looked without seeing, heard without hearing, forgot everything - almost everything. On falling from the horse, he lost consciousness; when he recovered it, the present was almost intolerable it was so rich and bright; the same was true of the most ancient and most trivial memories. …
“He remembered the shapes of the clouds in the south at dawn on the 30th of April of 1882, and he could compare them in his recollection with the marbled grain in the design of a leather-bound book which he had seen only once, and with the lines in the spray which an oar raised in the Rio Negro on the eve of the battle of the Quebracho.”
I think Borges knew how alien that kind of perception would seem to his readers, and it seems alien to me now. That’s why it’s a fun, strange story to read.
Now, in the next 20 minutes or so I’m going to explain how technology is going to make this weird experience of perfect memory seem quite commonplace, probably within the next 10 years.
But first: A little bit about where I’m coming from.
Wired’s perspective
Technology as a social, political and cultural force
“Lifestyle publication for geeks” — Vogue for programmers. Who go to Burning Man.
News for techies — but not tech news. My job is to try and present the bigger picture — Where we’re headed and what it all means.
Today’s talk
My focus is on technology and business. The big trends I’m tracking:
Dropping cost of storage
Increasing bandwidth at stable cost
Ubiquitous searchability
I’m going to talk about each of these trends, then spell out some of the implications for why they’re interesting.
Audience survey
How many people here are involved in with or work on systems that support public websites?
That get more than 1 million visitors/month?
More than 10 million visitors/month?
Have a webmail account?
Have used more than 1 GB of its storage capacity?
Maxed out a Gmail account?
Have a weblog?
Have a Twitter account or Facebook account?
Update your status daily?
Broadcast a live stream?
Storage
[slide: terabytes/$100]
Storage has been dropping in price by about 45% per year. 1/10 of the price in 4 years. What’s this mean?
Pundits like to talk about how many Libraries of Congress you can store in a Blu-ray Profile 3 disc or whatever but who really knows how big the Library of Congress is?
A more relevant measure: How many episodes of Battlestar Galactica?
HD video: Compressed 720i video requires a data stream of about 100 Mbps, which means you can store an hour of video — one BSG — in 44GB.
(Using the HD-DVCPRO format developed by Panasonic.)
That means you could store HD video of a 24 hour period — 2 seasons of BSG — in one terabyte.
Right now a terabyte hard drive costs about $200. Add another $25 if you want it to be portable, not an internal drive.
OK, BSG is one thing. What about recording HD video of your own life?
Skip the 8 hours/day you sleep, and the current cost to record your entire life in HD video is about $50,000 per year, using magnetic spinning media.
In four years, if trends hold, that cost will be $5,000; and another four years after that, in 2016, it will be $500. That’s assuming there are no advances in compression or new storage technologies that make terabytes even cheaper.
Suppose you only want 480p, standard-def video: Cut your storage requirements to 1/4 these estimates.
Or: suppose you want YouTube-quality video. YouTube’s current “high quality” setting gives you 480×360 pixel video at roughly 1 MB per second, or 360MB per hour. (If you could upload hourlong videos to YouTube).
At that rate, you’d need less than 10GB to store video of all 24 hours in the day, which works out to 3.6 terabytes per year — $720 in today’s storage technology.
In four years, you’ll be able to buy that much storage for $70 at the drugstore. Another $99 will get you the camera to record all that video. Or the camcorder might even be free (but I’ll get to that in a minute.)
I think there’s even more promise to flash media, which is more suited to being carried around, knocked about, and generally abused. However it’s still a lot more expensive.
[slide: Flash memory cost]
Right now a 4GB USB thumb drive is about $20-25.
Next year or the year after you should be able to get a 10GB thumb drive for the same amount, which means you could store a day’s worth of middling-quality video on your keychain. You can swap keys or upload the data to your multiterabyte hard drive at night.
Bandwidth
[slide: number of websites]
My first computer was an Apple II +. Great for playing games and learning how to program. I miss it. But if I still had it today, it would be nearly useless because it lacked an internet connection.
Used to be that personal computers were primarily standalone machines, used for computing, playing games, and the like. Only occasionally connected to each other to exchange information — but default condition was disconnected.
Now computers are always on, always connected, to the point where a standalone PC — a computer without an internet connection — is practically useless.
The first major impact of broadband is not the speed: it’s that it switched the default state for computers from disconnected to connected.
Of course there’s a lot more for us to look at and do online: more than 100 million websites, nearly 60 million active sites, as shown on this slide
Next step is extending this default state to other devices, such as cameras and camcorders.
[slide: Eye-Fi card]
We’re starting to see this with Wi-Fi add-ons for cameras, like the Eye-Fi, a 2GB SD card / Wi-Fi adapter. Plug it into your camera and it not only stores pictures but uploads them to your computer — or via that to your Flickr account — as fast as you take them.
Samsung was showing a Wi-Fi enabled HD camcorder at CES this year that can stream high-def video to your TV or computer, wirelessly. It’s called the SC-HMXC10 WiFi HD
Amount of bandwidth available to wireless networks is still a limiting factor: theoretically up to 540Mbps for Wi-Fi N, but in reality you’re lucky if you get anywhere close to 100Mbps.
At that rate you can transfer standard-def and low-quality video, assuming nothing goes wrong. But HD is still basically out of reach.
Companies working on transmitting HD wirelessly: UWB (Ultra Wideband) seems to be the key here
Second broadband trend: Increasing bandwidth.
Broadband 2.0: First wave of broadband was DSL and cable modems, and gave us one-third to 5Mbps.
Second wave will bring us 50-100Mbps via next-generation cable tech (Docsis 3.0), or fiber to the home.
With 100Mbps broadband you can download HD movies as fast as you can watch them. Or download a full-length, 2-hour standard def movie in 30 minutes.
All this bandwidth means people will not just be downloading, they’ll increasingly be moving their own stuff online: video, photos, and more.
[slide: YouTube growth]
Not only that, they’re going to expect that facilities will be available to help them do this.
YouTube? As many videos as you want to upload, as long as they’re smaller than 100MB apiece. As we move to Broadband 2.0, YouTube will go high-def.
[slide: Flickr growth]
Google Mail? 6GB of storage. Yahoo Mail? Unlimited storage.
Google Docs, Flickr, Odeo, Twitter, Blogger.com, Wordpress.com — you name it, there are services offering people essentially unlimited personal bandwidth and vast amounts of storage. For free.
You think your company’s website is going to be able to avoid this trend? Guess again.
[upcoming Wired.com story on Broadband 2.0. look for it next week]
Searchability
For decades, the model for information access was the library catalog.
In grade school, my teachers taught me how to use the subject, title, or author catalogs depending on what I was looking for. Each card had a bunch of structured data on it that had to be entered just so, and it took some work to learn how to use it properly.
That mentality gave us search forms like this one.
[slide: Worldcat]
Turns out, what people really want is this.
[slide: I'm feeling lucky button]
Google has taught us that what people want is a super-simple interface to data.
They don’t want multiple fields, checkboxes and radio buttons. They want one text field and a “Search” button. Plus and “I’m Feeling Lucky” button to make them smile.
Never mind that it takes hundreds of Ph.Ds and billions of dollars to create the complex algorithms needed to make that happen.
Surface simplicity — deep complexity
The problem: Unstructured data search is running into a wall: Machine intelligence can only go so far in determining relevance & organizing results. Even with Google, you sometimes have to page through dozens of results to find what you want. And you have to be savvy about constructing queries.
Startups like Brijit, Mahalo, Squidoo, ChaCha, and Boxxet aim to solve this problem by recruiting human help.
Problem is that the data is just too unstructured.
This problem is only going to get worse as you mix in huge amounts of visual and video data: MP3s, FLVs, MOVs, etc
One solution: paste on a layer of structure with microformats. Technorati pioneers this; other companies adopting them including Google, Microsoft and others
Another solution: Embrace structured data behind the scenes. And figure out how to keep the interface super simple
Experiments with popular crowdsourced structured data — eg Google Base and Freebase — suggest most people don’t really want to think about fields and database normalization. EVEN IF THEY NEED THIS
[slide: Nokia 95 and iPhone]
Some other reasons to keep things simple: Mobile devices, like the Nokia N95 and iPhone.
[Slide: Sony's flexible OLED]
Coming generation of flexible displays embedded into jewelry, backpacks, bags, hats, clothing, and more
All of these are going to demand extremely simple search interfaces
[15 min mark]
The Free Economy
Now I’m going to explain some of the implications of these trendlines in storage, broadband and search technology.
One is that many things will become free. Not virtually free — actually free.
Chris Anderson’s article in the March issue of Wired starts from the indisputable point that you can get more free stuff now than at any time in history. Free mail, free music, free DVRs, even free plane tickets.
Of course this is not a new idea, and Chris recognizes this. He talks about King Gillette, a former salesman of cork-lined bottle caps, who invented the disposable safety razor in 1895. By the early 1900s, he realized he could make far more money by essentially giving away his razors through high-profile promotions, then making all his profit from the high-margin blades.
It’s a model that has worked in many other industries, such as inkjet printers. And Ari Kaplan even told me last night that he used to give away BlackBerrys just to get people to use his mobile application software.
But now, thanks to the plummeting costs of information technology, “free” is becoming the foundation of many companies’ strategies.
[slide: Google 411]
For example, Google gives away directory assistance at 1-800-GOOG-411. Most phone companies charge 50 cents to $1.50 for the same service. How can Google afford to throw that money away? Simple: they’re just putting a stake in the ground for mobile information services. That market that could make them billions in a few years.
[slide: Comcast DVR]
Comcast can give away $250 DVRs because they’re going to make the money back from other services they’ll sell you. For example, the $20 installation fee, the $14 a month for cable service, $43 per month for high-speed internet, and so on.
The upshot: They make back their $250 in 18 months, and everything after that is profit.
[slide: webmail windfall]
The same thing is happening in webmail. As storage prices fall, companies can offer more and more mail storage for little or no money.
2002: Yahoo’s premium email service cost $30/year for 25MB of storage
2004: Google introduces Gmail with 1GB for free
2007: Yahoo mail offers unlimited storage.
At the same time, revenues for both companies are rising because they can show more ads to each user.
Anderson: “we are entering an era when free will be seen as the norm, not an anomaly. …
“In 1954, at the dawn of nuclear power, Lewis Strauss, head of the Atomic Energy Commission, promised that we were entering an age when electricity would be ‘too cheap to meter.’”
What would the world look like with free electricity?
Electric heating
Electric cars
Massive electric desalinization plants “would turn seawater into all the freshwater anyone could want”
“fossil fuels would be seen as ludicrously expensive and dirty, and so carbon emissions would plummet. The phrase ‘global warming’ would have never entered the language.”
A similar thing is about to happen with digital technologies.
Not free for IT people, of course. “Bandwidth will be too cheap to meter” is a joke.
Ditto storage.
Only free from the customer’s perspective. But this is so important from a marketing and branding perspective that no company can afford to ignore it.
Plan to spend a lot of money giving away what once would have seemed like enormous quantities of computing resources.
The Beginning of History
[slide: Justin Kan]
Justin.tv story: started with one guy, so unusual Justin had to keep changing his phone # and moving his apt. to keep people from harassing him. Now has 28,106 video channels with 73,754 user uploaded video clips, and is delivering 3.6Gbps of video traffic at its peak.
[slide: Gordon Bell]
This is Gordon Bell, a Microsoft researcher who is probably the best-documented human being on Earth right now. He saves all his email, of course. He also has scanned all his documents, manuals, receipts, paper of any kind.
Plus he wears a Microsoft SenseCam, a camera around his neck that snaps a picture every few seconds or whenever the light levels change. And a voice-activated recorder that records everything Bell says or hears.
He’s not just liveblogging or livecasting, he’s lifelogging — making a digital record of everything he can in his life.
A lot of the pictures wind up looking like this.
[slide: Gordon Bell in an elevator]
Rather boring, isn’t it. But that’s ok, because this project is really about putting everything together, and even seemingly mundane images like this can take on significance when they’ve been inserted into a long stream of images, with context, notes, tags, and connections to other pictures.
Science fiction writer Charlie Stross:
“This century we’re going to learn a lesson about what it means to be unable to forget anything. … our descendants’ relationship with their history is going to be very different from our own, because they will be able to see it with a level of depth and clarity that nobody has ever experienced before.
“Meet your descendants. They don’t know what it’s like to be involuntarily lost, don’t understand what we mean by the word ‘privacy’, and will have access (sooner or later) to a historical representation of our species that defies understanding. They live in a world where history has a sharply-drawn start line, and everything they individually do or say will sooner or later be visible to everyone who comes after them, forever. They are incredibly alien to us.”
[slide: Venetus A]
This is the Venetus A, a 10th century manuscript of the Iliad. I’m showing you this because I think it looks really cool.
But it’s also a great example of extreme digitization. Wired.com wrote about this project last year: Scholars photographed every page of this manuscript using a 39-megapixel Hasselblad.
They also used an industrial 3D laser scanner to create a high-resolution 3D image of every page, with every wrinkle, crease and undulation in the parchment recorded.
Scholars can view the page in 3D, as it actually is, or they can digitally “flatten” it.
Searchable, XML-encoded transcriptions and translations are being added to the scans. And the whole thing is being placed online with a Creative Commons license so anyone can use or reuse it.
Robot arms, high-bandwidth networking and serious processing power involved in scanning process too. Cool stuff!
It’s a cool project, but it also suggests the vast amounts of detail that could be scanned, and will be scanned, once storage is cheap enough.
That’s another reason why “Libraries of Congress” are a poor measure of storage capacity — it depends on how high resolution your digitized Libraries of Congress are!
And this brings us to the problem with total memory. In that Borges story I mentioned, the man with perfect memory becomes overwhelmed by his memories because there are so many of them and they are so specific. “… he was disturbed by the fact that a dog at three-fourteen (seen in profile) should have the same name as the dog at three-fifteen (seen from the front).”
Obsessed with the specific details and unable to generalize
“Without effort, he had learned English, French, Portuguese, Latin. I suspect, nevertheless, that he was not very capable of thought. To think is to forget a difference, to generalize, to abstract. In the overly replete world of Funes there were nothing but details, almost contiguous details.”
This story was written in 1942, just about the same time as Vannevar Bush’s prescient essay “As We May Think,” which imagined a world in which machines would augment human memory and intellect by storing huge amounts of visual and textual data — including the minutiae of our daily lives.
Bush, who was a mathematician and computer scientist, realized how important it would be for the memory machine to include tools for making connections between different bits of information.
In the next ten or so years, we are going to see a lot of people “waking up” to infinite memory the way Borges and Bush, 60 years ago, could only imagine.
Whether that awakening is immobilizing or empowering will depend on how easily we can use our new memories.
Will these memories just sit immobilized in the multi-terabyte vaults in our living rooms, or on Google’s servers?
Will we waste our present days playing and replaying highlight reels from of our past?
Or will we have the capacity to search those memories, making connections, discovering things, and sharing that information with others?
I don’t know the answers to these questions. But I do know that I’m really excited to be living in this time, at the beginning of known history. And I know that you, in this room, will play an important role in designing that future and then making it happen.
I can’t wait to see what you do.