Ask, and Understand

I didn't watch the recent man vs. machine series on Jeopardy! this past week, so all the information I'm getting about how IBM's Watson crushed its human competitors has been filtered through other sources - but that's okay. The most fundamental thing that we should take from this is that clear-language interaction with computers - something that's been a staple of science fiction for decades - has taken a huge step toward reality, despite speedbumps such as answering "Toronto" in a Final Jeopardy category about United States cities. Watson wasn't fooled by the wordplay and the back-and-forth; it was able to to pierce through the doubletalk and understand what was being asked of it.

That bears repeating: a computer program that understands something. There are plenty of people who think that this is just a dim shadow of what's waiting for us in the very near future. As potentially world-shaking as the Singularity could be, supposing that we ever agree on how to define the damn thing and then it actually happens, that's not one of the implications of Watson's victory that I've been thinking about much.

Specifically, the social implications of people having access to individual question-answerer programs. What if, in ten years, every smartphone literally is a smart phone, and comes equipped with a descendant of Watson that can answer any question you ask of it so long as it has sufficient time to poke around for answers on the internet? Considering the pace of development over the last ten years, I would not be surprised for the phones of 2021 to function as oracles as well.

But what does this do to people? Consider the modern day - how often do you encounter stories of people driving themselves into ditches, or into farmers' fields, or into an illegal turn across multiple lanes of traffic because their GPS told them to, and they obeyed it without question? Let's be honest... critical thinking isn't exactly part of the school curriculum in North America at least, and there are plenty of people out there who don't know how to think think. People have always wanted easy answers. What's going to happen when it's as easy as asking your phone when you've got a burning question, and uncritically accepting whatever it flings back at you as the truth?

One thing I'll always remember about school is that research is difficult. It's hard to find good and appropriate sources, to sort through those sources, to figure out what material you need and to identify biases in the text and distill it down so you can answer the question in front of you. For me, the research phase of an essay took weeks - and that was even without waiting for worthwhile sources to get there through interlibrary loan. Consider how many people would leap at the opportunity to skip all that effort - hell, it's practically what we see now, with some students shamelessly cribbing from Wikipedia articles, or going whole hog and buying essays from mills. Who cares about understanding the answer, so long as you have it?

Understanding is the key. An answered question is worthless unless you know what the answer means. As our computers become more and more sophisticated and capable of interacting with us on more meaningful levels, we need to take care that we avoid falling into the trap of complacency - of working toward a world where our computers understand its workings better than we do.

A Robot Must Obey

Yesterday I lost a multiplayer match of StarCraft II because my computer turned itself off to install automatic updates. To be honest, it ultimately was my fault - it had been needling me to do so for days, but I'd kept on just clicking "postpone." The installation of automatic updates requires that the computer be rebooted a few times, and I dislike rebooting unnecessarily. The problem is that when it happened, it came completely by surprise. The computer restarted after a fifteen-minute countdown, but a countdown that popped under. Since I was busy trying desperately to find a tactic that would end with my opponent not standing atop the broken remnants of my base, my first indication of its presence was when programs started shutting down of their own accord.

Without giving me a chance to save, either.

Over the last few decades our civilization has been computerized to a great degree, and this is a trend that's only going to gain strength in the coming years. We're at the point now where armed military robots are no longer science fiction, but are being used in theatres of operation at this moment. If that's going to continue to be the case, we can't just let our current operating procedures stand. My issue was minor - it meant my opponent could find someone else to smash sooner, and I lost a few points on the division ladder - but it arose from the computer's programming telling it to embark on an action unless that action was specifically countermanded.

The attention of a human in a serious issue is important, but humans can only divide their attention so much - and sometimes, be it through information overload or programming defect, the computer's message to the operator may be missed. If we're not careful, this could lead to a future where automated systems put dangerous wheels in motion because no human contradicted them in time. Things are stressful enough without a ticking clock, especially one that has no reason to be there.

This is the sort of thing that creeps in, a bit at a time. No one's ever going to sit down and say "you know what? Let's make it so that computers figure out what we're going to do." No, they'll automate control of low-priority systems, always intending to keep humans in the loop, but as things go on and as the organization paying the bills realizes that automated decision loops mean that they can get by with fewer humans on the payroll, people may gradually but steadily get crowded out. It happens by degrees - the sort of transition that humans are the worst at perceiving until it's too late, or nearly so, to do anything about. I've heard stories of automated stock trading that's caused a company's share price to crash utterly because of the decisions some computer program made - I can't recall now if it actually happened or is just an urban legend, but the danger is there just the same. It would certainly make it harder to make those systems Three Laws-compliant.

Freedom of choice is one of the most valuable things in the universe. If we invest the course-choosing mechanisms of our civilization in machines that will act unless specifically instructed otherwise, well - at that point, whose civilization is it _really_?

A Singular Complaint

Pretty much since I first learned about the concept, I've taken a rather dim view of the Singularity. If you've never heard of it before, let me summarize - the Singularity is a concept of the future, first explored in detail by science fiction writer Vernor Vinge in the 1980s but now greatly associated with futurist Ray Kurzweil, which holds that the pace of technological development will continue to accelerate and soon come to a point wherein it's impossible to speculate on what things might be like on the far side of it - a cultural version of a black hole's event horizon.

The reason I take a dim view of it is because, to me, pursuing this ideal is crazy. Look around and evaluate how well we've managed the world and ourselves, how optimally our societies are running as technology and methodology becomes progressively more and more advanced. As it is we're just muddling through in the best British tradition, even though we've had the benefit of decades of speculation about what prospects the future holds. How well do you think we'd do when we couldn't even hope to see what's coming?

Artificial intelligence is a common component of Singularitarian speculation. The idea there is that once the first AI is built, it will redesign itself to become more intelligent and capable of faster thought, onward and upward until it ascends to some kind of technological godhood. There's never any explanation as to why this is the case - the assumption is that it's inevitable, just because, and that humanity would of course be brought along for the ride. Because, really, aren't you just dying to shed your fleshy prison by uploading your consciousness into a computer, and thus becoming a true citizen of the Singularity Age?

The big problem with this, though, is that as far as I'm concerned it's all a scam. There's no such thing as digital immortality. If you upload your consciousness to a machine, you're not saving yourself. You're not cheating death. All you're doing is creating another perspective, a sophisticated computer program that has been trained to think just like you and has all your memories. Meanwhile, the original "you" is dead on the operating table, as the general idea right now is that uploading a consciousness would, of necessity, include a destructive scan of the brain - and the brain's important. At least for me. It's where I keep all my me-ness.

This is not to say that I think the idea of consciousness uploading, as a technology, is ridiculous. I think it's quite plausible, and the presence of consciousness uploading and artificial intelligence in a future culture does not validate the Singularity. That's one reason I can stomach Transhuman Space - sure, it's got AIs and ghosts flitting around the networks, but it's still a fundamentally comprehensible setting.

What really gets me is the general assumption of how easy it is. Maybe with the right tools and skills the process of uploading would be straightforward, but then there's everything after that. If you're looking for digital immortality, you're going to want the computer program that thinks it's you to survive and prosper for ages, right? One problem - it's damned expensive, and difficult besides, to ensure survival as a ghost in the machine.

Come on, computer! If you're so awesome, then defend yourself!

Think about it - how often have you had a hard drive fail, or a USB drive go kaput? A computer running an uploaded consciousness would have to not only be of extremely rugged construction, but proof against electromagnetic pulses, viruses, and so on, in addition to whatever properties might be required for a conscious program - maybe it would have to be a quantum computer. I can't imagine anything but large, dedicated machines running these self-aware programs. Maybe they're installed in lunar lava tubes, and rely on secrecy for security - maybe they're maintained by teams of technicians and guarded by an army of security in some Earthbound hideaway. Either way, it doesn't strike me as something that would be generally accessible to the average person.

But there have got to be interesting stories in that - human conflicts against a digital aristocracy.