Monday, September 20, 2004

Ask me a question that only someone who had seen a real raccoon would be able to answer

One day in the early nineties, when I was playing outside after school, I saw a raccoon, cautious and deliberate, making its way across the lawn in measured, calculated steps. I was completely entranced; this was the first and only raccoon I have ever seen in the wild, if the suburbs of D.C. count as the “wild.” When my parents came home from work, I insisted that I be questioned rigorously. I was very concerned, I recall, with not being believed, and I wanted to dispel all doubt that I could be making this up. Besides, as an only child, it often happened that I was the only witness to the various things that strike children as remarkable, worthy of reporting, and I certainly did not want to be thought an impostor. What did the raccoon look like? How was it walking? What did its face look like? Where was it going? Did it have paws or not? These were the kinds of amusing questions I demanded be asked of me, rather incessantly I'm told, until I was satisfied that my parents believed me.

My parents seem to enjoy trotting out this story at dinner parties, and it struck me as an interesting demonstration of the difficulties that arise from attempts to determine authenticity with only text at our disposal. It also reminds me of the hilarious questions that the Leisure Suit Larry games (a seedy role-playing game for adults) would pose of the user, intended to ascertain age and maturity level. If you missed several of these questions, you would be deemed under-age and denied access to the game, even though some of the questions were culturally quite specific, involving seemingly arbitrary references to pop culture and the like (so even my mom, who had spent the last 12 years in Japan, could not always answer all of them).

Can questions alone ever unfailingly determine the authenticity of information, or how authoritative a given information source is? How good can they get? Was there any set of questions that could have established “for sure” whether I had in fact seen a real raccoon? What strategies are we developing online to ferret out impostors? When we evaluate web sites for their quality, or receive advice from people on message boards or in chat rooms, what questions are we asking? Are they the same ones we would ask off-line to establish expertise in a court, for example, or is there a new breed of interrogation evolving to address informal online environments?


Blogger Joseph Lorenzo Hall said...

When I read something (especially scholarly articles), I unconsciously ask myself a few things: Is the information in this piece consistent? Is it rational? What are the rhetorical techniques being used? Through a series of "always on" interrogations, I feel I can usually spot rotten information.

However, nothing is a better example of this than law review articles or legal opinions. I *love footnotes*. Moreover, I love the feeling and the ability to be able to track down information supposedly spurting forth from footnoted sources. I wish everything had footnoted sources to the level of which law reviews do... even newspapers and magazines. That doesn't mean that anything *would* be more authoritative... but at least its logical pedigree would be disclosed.

12:54 PM  
Blogger Judd said...

Although Callie will certainly be on my case for failing to answer her question, I have a related thought:

What is an example of a situation where we must "unfailingly determine the authenticity of information"? I suggest in the context of daily life (and in most other contexts as well) coming up with an answer might be difficult. The question for me isn't what situation (or decision) is so grave that it requires absolute information (which may or may not exist), it's more about identifying the proper threshold. So really maybe we ought to be asking, how do we determine whether the authenticity of information is good enough?

1:35 PM  
Blogger yardi said...

Judd asked - What is an example of a situation where we must "unfailingly determine the authenticity of information"?

I would argue that an example of this is locating actual weapons of mass destruction before taking an action on the assumption that they are there.

7:15 PM  
Blogger yardi said...

We could also amend Judd's last question to be:
how do we determine whether the authenticity of information is good enough FOR ME?

A few months ago, sadly, my dentist told me I had a cavity. I couldn't see it or feel it and I tend to resist doctors and dentists if possible, but figured he knew what he was talking about. My dad, however, resists the advice of doctors on a whole different level and was convinced it was idiotic for me to let him drill a hole in my teeth for no reason at all, given that I couldn't feel it. (Yes, my dad actually took a flashlight in my mouth to look for it himself, but that's a different story). My mom, on the other hand, thought that I should absolutely have it filled. Whatever the dentist says goes, in her opinion. I ended up getting it filled, if only because the wrath of my mom is worse than the wrath of my dad (but again, different story).
Regardless, it did become an interesting discussion of who to trust and why.

7:33 PM  
Blogger Judd said...

Sarita made fun of me earlier for answering a question with a question (which is fair enough), so I thought I would chime in with a response to my own quesiton and Sarita's responses:

We might agree that confidence in information should be high for the purposes of policy making or military matters. Nothing in my life has such profound consequences. Perhaps the situations where we must "unfailingly determine the authenticity of information" are where the consequences of an action or decision are (or are perceived to be) severe and irreversable.

7:52 PM  
Blogger yardi said...

In my defense, I was also making fun of myself for having posted questions as well. It is so much easier to come up with questions than answers.

I think Judd is on to something with the qualification that the "consequences of an action or decision are severe and irreversable."

The one example where something in each of our lives has profound consequences is personal health. I would choose to have a surgical procedure only if I was as confident as could be that I had been given good information about what they were going to do and why I needed it. A related debate is with medical malpractice. Doctors have to pay absurd insurance premiums because of fears (with good reason) of being sued. I'm not sure what my opinion is of how responsible they should be for information and procedures they have performed, but it is an engaging topic.

10:27 PM  
Blogger Calvert Jones said...

On your case, Judd? Why, never!

I think you are probably right that it’s more practical to ask what questions will bring us to the proper threshold, where we can sit back, sigh with relief, and declare, “Fine, I believe you. This is legitimate.” But that word, “threshold,” brings me to another thought, one I find extremely distasteful, like Chesapeake Bay scallops so overcooked by a negligent chef that they taste rough and rubbery, and that is this: do you all believe that authenticity of information is simply a matter of personal satisfaction, of reaching our own personal thresholds? You ask questions until you are satisfied that X is authentic, and I ask questions until I am satisfied – by the time each of us is finished with our interrogation, we are at completely different points, having learned possibly different, even incompatible things about X, yet each is “satisfied” that X is authentic.

This brings us back, I think, to our earlier discussion of whether bad information is relative. In that context, I argued that determining whether information was bad was not an inherently subjective process, which is what someone had suggested. I said I thought that there are certain properties of information that can be objectively considered good or bad, based on logic and principles of reasoning, though we may not be in a position to see these properties (because we have been deceived, or we don’t know enough about physics, or the situation has become too complicated to break down logically, etc) and not all information may have these properties. So I don’t think that authenticity of information is simply a matter of personal satisfaction, because we have these objective tools – which may or may not bring us satisfaction, but do, more or less reliably, shed light on authenticity. And I think Joe made a great point that footnotes disclose logical pedigree, in that they point readers to evidence for the premises on which an argument is built.

Returning to Judd’s question: “How do we determine whether the authenticity of information is good enough?” I think we have established, time-tested, and certainly culturally specific ways of doing this – which do involve objective quality tests – but what do we have for assessing information on the Web? Are there any implicitly standardized methods?

I wonder if the over-reliance on textual exchange in determining authenticity of online information has the potential to sharpen, reinforce, or even encourage dangerous stereotypes, especially in informal settings. Suppose my parents are anti-Semites, but I’m not, at least not yet – I go into a chat room, start talking to people, and then it occurs to me. What if these people are Jewish? How can I find out if they are Jewish? They might lie and say they’re not, so what questions will help me determine this for sure? Won’t these questions rely on Jewish stereotypes, or assumptions about what a Jew would know, etc? The problem reminds me of a scene in an Eddie Murphy movie called “The Distinguished Gentleman,” in which he is on the phone masquerading as a white man. It’s supposed to be a comedy; he’s asked a series of questions, which embrace racial stereotypes and some of which seemed to make no sense at all (“What’s your favorite kind of pie?” Eddie Murphy hesitates, about to say something about the pie his mother would make, then “corrects” himself and says, “Oh, apple pie. Apple pie is certainly my favorite kind. Most definitely.” – the white men on the other end nod with approval; do white people prefer apple pie?).

So the questions we ask to ascertain authenticity of information are culturally, nationally, and personally quite telling; they may reveal more about the questioner than the questioned party. In purely textual environments, they might reflect stereotypes even more, since stereotypes can be easily broken down into leading questions that fit well into textual environments (a “real” Serb, white man, Jew, raccoon). The Leisure Suit Larry questions revealed a particular, definitively American conception of maturity, the knowledge and set of experiences a “mature” person ought to have, according to the designers; in short, a stereotype. Not only could authenticity of information online, in these sorts of cases, be a matter of personal satisfaction, it could also encourage the proliferation of stereotypes, guiding the questions we think to ask to find out if X is authentic.

10:35 PM  
Blogger Joseph Lorenzo Hall said...

You've got to love how Callie brings in Leisure Suit Larry and pie.

To Judd, what's a case where you need absolute information? Patents. That is, if you're making something that is high-tech (or even low-tech these days) you have to do an exhaustive search of the USPTO patent database to make sure you aren't infringing any of the 6 billion patents granted in the U.S. If you don't do such a search, you might get sued for patent infringement, which is easily the most expensive kind of litigation out there. (Granted, if you do a search and miss the one patent that you infringe, that can be used as evidence that you "should have known" of the other patent... so it might be better to not search at all!). Needless to say, the patent system in this country is broken (especially for software).

9:17 AM  
Blogger laura said...

Going back to this:
yardi said...
Judd asked - What is an example of a situation where we must "unfailingly determine the authenticity of information"?

I would argue that an example of this is locating actual weapons of mass destruction before taking an action on the assumption that they are there.

It's a little scary to think, then, that the people who are making decisions regarding such things as war and weapons of mass destruction are doing their research on the Internet. Here's Colin Powell on why he doesn't need a library anymore, because everything's on the Internet:

"I told my staff: 'I no longer have any encyclopedias, any dictionaries, or any reference materials anywhere in my office, whatsoever, I don't need them. I've stopped using all reference materials because you don't need it. All you need is a search engine.'"

Here's the link to the whole speech:

Many of us skeptics take the things we find on the Web with a grain of salt, or at least -- being information professionals -- check a variety of sources before we consider ourselves informed. But if the collective authority of the Internet is good enough for our Secretary of State, shouldn't it be good enough for the rest of us? Or are we shocked (as I am) that Colin Powell does all of his research on the Internet because we have this implicit distrust of it? I used to be a reference administrator for the Internet Public Library, where one of the goals was to build a website full of free and freely available web resources that could be trusted, and we had a long list of criteria that websites had to meet in order to be included. And despite this experience, despite my training in finding good resources -- and they do exist -- I still find myself shaking my head when I hear that Colin Powell doesn't need a library anymore.

Taking off from what Judd said above, how do we know when we're done, when what we've found is good enough? Is it when we've found enough sources that generally concur with one another and with our however vaguely-formulated original thesis? Is it when we find the one source that somehow both looks authoritative and agrees with our ideological perspective? For example, even though the Hoover Institution is a well-funded think tank with a lot of big names behind it, I'd still question any "authoritative" information that came out of it. I know a lot of other people would do the same for the Council on Foreign Relations, for the same reason.

11:28 AM  
Blogger Jonathan Traupman said...

Not that I want to be known as the guy who defends Secretary Powell, but he wasn't saying that he didn't need libraries, rather that he didn't need reference books -- dictionaries and encyclopedias. I have to say, I agree. Reference books are dinosaurs -- I can't even remember the last time I used a bound paper dictionary or encyclopedia. has a greater lexicon than any reasonable sized dictionary I've ever had on my shelf, and looking up words takes far less time than with a paper dictionary.

Likewise, search engines are a very convenient way of getting general information on a topic. While the information returned from Google may not be as authoritative as a traditional encyclopedia, one can argue that it is more useful because it isn't subject to the biases of a single editor. Even if you'd prefer traditional encyclopedias, they're mostly online now.

So I'm not the least bit concerned that our Secretary of State uses the web instead of Webster's. Now, had he actually said that all of the State Department's research is now being done exclusively using online resources and that he's torn up his staff's library cards, I'd be worried.

Changing topic somewhat...

When evaluating authenticity online, we're not limited to our own line of questioning, but can use other's judgments as a basis for our decisions. One way to decide someone or something is trustworthy is to ask a large number of other users, "is resource X trustworthy?"

Of course, some respondants may have poor judgment and others may lie, either to forward their own agenda or simply to be difficult. But if we ask enough people, we'll get a pretty good idea of the reliability of a resource.

This is exactly the process that happens behind the scenes to allow us to use services like with confidence. There are enough other people using it that errors or biases will likely be detected and publicized. The more people that rely on a service, the more likely the service is reliable.

I think this is related to the comment Joe made in another article about blogs being more accurate because they are a two-way communication. The information source itself doesn't need to be two-way, but if you have a large enough readership who are able to communicate, it will be much harder to pass off bad information than if your audience is disconnected and unable to efficiently disseminate their reactions.

12:34 PM  
Blogger Judd said...

Laura wrote:

"how do we know when we're done, when what we've found is good enough? Is it when we've found enough sources that generally concur with one another and with our however vaguely-formulated original thesis? Is it when we find the one source that somehow both looks authoritative and agrees with our ideological perspective?"

I would say that searching on the web takes several different forms that require several different answers to this question. While the internet is a nearly bottomless source of information, I suspect that most people only engage it on the most superficial level. So in order to evaluate how I feel about Colin Powell's statements, I need to know not just whether he's using the internet as his only resource, but how he's using the internet. The danger of one-stop-shops like Google is that they provide a quick but shallow response, and they don't necessarily promote using the capacity of the internet as a 'deep' information source.

So many people probably use the web as a way of satisficing. But even in situations where search is deeper and more directed, I think people often submit to their preconcieved ideas about the information they're looking for and settle on the information which affords them the least amount of cognitive dissonance. In other words, you might just keep on looking until you find information which supports the conclusion you were implicitly or explicitly hoping to find to begin with.

11:32 AM  

Post a Comment

<< Home