From the December 1996 Issue of PC World
Searching is My Business: A Gumshoe’s Guide to the Web
It takes more than a browser to search the Web. It takes smarts,the right tools, and tips for tracking down leads, no matter where they take you.
By Dylan Tweney
Chapter 1: The Big Seek
It was a cold November morning. A fog weighed on the city like a badhangover. I was working hard on a case of cigarettes and not much else when hewalked in the door. A real mess, from his cheap business suit all the way to the386 notebook. I could tell he was in trouble.
“I don’t know what to do…I’ve looked absolutely everywhere and I can’t findanything!” he blurted.
“Easy, kid,” I soothed. “Tell me all about it.”
“It all started last week. My boss called me in and asked me to do someresearch on Megasoft, our main competitor. So I went online and starteddowsing…”
“Browsing,” I corrected. But the kid had a point even if he didn’t know it.Trying to find out about Megasoft by browsing is like looking for water in theMojave using a forked stick.
“Yeah, whatever. Anyhow, I spent all night clicking on links, and all I foundwas a bunch of silly home pages and a lot of advertisements.”
He was tangled in the Web and tangled bad. I had to show him how to getaround, and quick. Fortunately, he came to the right place. I’m a Web searcher. Ifind things. That’s my job.
Chapter 2:
To Browse And Browse Not
I tried to break the bad news to him gently. “Listen, kid,” I said. “Browsingis a waste of time. If you want to find anything, you gotta search the Web, notbrowse it.”
Looking at him, I realized the poor sap didn’t know a Web directory from asearch engine, let alone how to use metasearch tools, offline browsers, andpersonal Web assistants. He had a lot to learn. Good thing he had me to teachhim.
“Pull up a chair, kid, and I’ll tell you the smart way to search the Web.” Ilit a cigarette, leaned back, and started in…
Directories Versus Search Engines
Web directories, like Yahoo’s and Magellan’s, are like card catalogs: Theyfile everything in categories such as Sports, Entertainment, Computers…you getthe picture. Once you find your topic in a Web directory, you’ve got a handful ofkey sites to start with. And that might be all you need. Most directories includebrief descriptions of each site and lead you to a site’s home page, but not tospecific pages within a site.
In short, your average Web directory is a no-nonsense tool–it gets the jobdone, and quick. When you’re looking for information on a general topic, adirectory is the best place to start, particularly if you already know your wayaround it.
But Web directories cover only a small fraction of the pages available on theWorld Wide Web. That’s where search engines like those found on AltaVista,HotBot, and Lycos come in. You tell the engine what you’re interested in, and itfinds all the Web pages it knows about that match. To keep their records current,the search engines use programs called spiders or bots that follow links frompage to page, recording all or part of the contents of each page as they go,eventually casing much of the Web.
Because no human intervention is required, search engines can cover much moreof the Web than directories can. But you need to know how to use them if youdon’t want to waste your time sifting through thousands of possibilities.
Chapter 3: The Searcher Always Clicks Twice
“Thanks for your help,” the kid said. “Well, I gotta get going. How much do Iowe you?”
A little knowledge is a dangerous thing, I thought. “Sit down, kid,” Igrowled. “We’ve barely cracked the ice here.
“The search engines and directories on the Web are powerful, but they’reblunt instruments–without a good search strategy and carefully constructedqueries, you can waste a lot of time. Here’s how you go about it.”
Stake Out the Subject
If you’re looking for general information, start with a Web directory. I useYahoo, because it’s got a lot of material and I’m familiar with the way itscategories are organized–but you might prefer Magellan, or maybe the A2Zdirectory in Lycos.
On the other hand, if you’re looking for a specific concept or a phrase, likeMaltese Falcon, or something that’s not easily categorized, like bulletproofvests, the Web directories won’t be much help. You’ve got to use a search engine.
You’ll likely need more than one, since the same query gives differentresults from engine to engine. I like to start with AltaVista, since it covers alot of the Web, and then try Infoseek Ultra (http://ultra.infoseek.com)–a newversion of Infoseek–because it gives results right away. See “The Simple Art ofSearching” for details on the major search engines and Web directories.
Zero In on Your Target
With search engines, take your time and do the job right. If you give somethought to your queries, you’ll get better results. Try entering a handful ofwords or a phrase related to your topic, like San Francisco Hotel if you’relooking for a place to flop when you’re in the City by the Bay.
The more specific you can be, the better. Don’t worry aboutredundancy–synonyms can help narrow the field of your search. Leave outnonessential words like prepositions and articles (of, to, and, the, and soon)–most search engines ignore them anyway.
Say you’re looking for the inside skinny on the Whitewater scandal. Don’tenter the query whitewater–you’ll find yourself swamped with ads for whitewaterrafting expeditions. The query whitewater clinton bill hillary real estate getsyou what you want.
Work the Combinations
If a search site returns no hits or too few, your query may be too narrow–orit may use the wrong terms. Try another query with fewer words, or one withdifferent, less specific words. Covert surveillance takes you nowhere? Try spy orstakeout.
If the search engine returns hundreds or thousands of hits, your query isprobably too broad. If you don’t find what you want in the first two or threepages of results, stop. Try again with more specific words or more restrictivequery options. Instead of just detective, try sam spade humphrey bogart; see”Smooth Operators” for tips on creating sophisticated queries.
Don’t be afraid to try different search engines. Often the same query onother engines will turn up completely different hits.
Chapter 4:
Farewell, My Browser
“You mean I have to enter queries again and again?” The kid winced. “Isn’tthere a better way?”
He was catching on. “Sure,” I said. “You don’t have to do all the footworkyourself. That’s why they invented computers.” I poured a shot and filled him in.
Once you’ve tapped out the directories and search engines, there’s a wholelineup of Web pages and software tools that can help sniff out what you need.They fall into three groups: metasearch tools, which submit the same querysimultaneously to several engines; search assistants, which help you managesearches more efficiently; and offline browsers, which check Web pages anddownload them if anything’s new.
Metasearch Tools
Tired of pounding the pavement? Instead of sending a query repeatedly todifferent sites, use a metasearch tool–a Web site or a program that submits yourquery to several engines simultaneously, then rounds up the most likely suspectson a single page.
One of the best Web-based search consolidators is SavvySearch (see “TheSimple Art of Searching”). With SavvySearch, you enter your query and specifysearch options, such as how many hits to return from each search engine and howto display the results.
Once you’ve entered your query, SavvySearch submits it to three searchengines that contain the kind of information you told it you’re looking for. Thenit collects all the results on one page. At the bottom of the page is a clevertoolbar that allows you to submit the same query to different sets of searchengines. It’s a quick and convenient way to run searches on many engines. Best ofall, it’s free.
On the downside, SavvySearch limits you to 50 hits per search engine, and itcan often be slowed by network traffic. For comprehensive searching, you may wanta PC-based assistant, such as FastFind (read on).
An alternative to SavvySearch and similar sites is a metasearch program, suchas Bitsafe Computer Services’ Arf (http://www.execpc.com/~bitsafe/arf). Arf is astripped-down, inexpensive ($20) utility for automating searches andconsolidating the results. Its straightforward interface lets you specify aquery, a maximum number of hits (up to 10,000), and a search engine (you’relimited to AltaVista, DejaNews, Infoseek, Lycos, and WebCrawler). Click on theStart button, and Arf returns a simple, unsorted list of hits; in addition, itdownloads the pages that each hit links to. To perform more than one querysimultaneously, you can run several copies of Arf at once.
For a few dollars more, the ForeFront Group’s $70 WebSeeker(http://www.ffg.com) offers many additional options, such as sorting the list ofhits alphabetically. WebSeeker queries as many as 20 Web and Usenet searchengines simultaneously and compiles a list of hits that you can sort or search.WebSeeker can be somewhat slow, but it has a substantial amount of querying powerto assist the Internet sleuth.
Search Assistants
A metasearch tool can save you the trouble of visiting every search siteyourself, but you still face the problem of sorting through too many (or notenough) hits. The solution is search assistants–programs that enhance, augment,and help you keep track of your searches. Some of these tools stake out the Netfor you, running the same queries again and again and notifying you only whensomething new turns up.
Between you, me, and the rats in the wall, the world’s full of these searchassistants, and most of them aren’t worth a damn. Each takes some time to learn,so choose with care. I’ll clue you in to one of the easiest, and two of the mostpowerful.
One of the dandiest search assistants is the $35 More Like This, fromKnowledge Discovery
(http://www.morelikethis.com). It’s a simple toolbar thatfloats on top of your browser. You use it to run sophisticated queries quicklyand easily: Just click on the More Like Query button, enter the word you’relooking for, and click OK. More Like This submits the query to the search engineyou choose and opens the results page in your browser.
Better yet, you don’t need to learn any search syntax. More Like Thistranslates your query into something the search engine can use. You can alsoclick on the More Like Title button to submit a query based on the title of theWeb page you’re currently looking at (much like the More Like This option offeredby the unrelated Excite search engine). If the page’s title is ‘SurveillanceTools,’ for example, More Like This may take you to other sites where you canfind information on snoop merchandise. You can also copy text from anyapplication to the Clipboard, and click on More Like Clipboard to submit thattext as a query.
More Like This also offers special concept searches, which automaticallyamplify your queries with related words to help zero in on relevant sites. If youenter the word bogart and select a concept search, More Like This may augmentyour query with words such as humphrey, maltese, bacall, and so on.
More Like This is a minimalist tool that stays out of the way until you needit. For a heavier-caliber weapon, try Symantec’s $49.95 Internet FastFind(http://www.symantec.com), a collection of several useful utilities for findingand managing Internet information. As a metasearch tool, FastFind collects allthe results on a single page, which you can view in your Web browser, sortingthem by relevance or by Web site. Other utilities from FastFind’s main menu canfind files and download them from FTP sites, notify you when Web sites change,and search the Web for updates to your system’s drivers and installed programs.
Quarterdeck’s new $49.95 all-in-one search tool, WebCompass 2.0(http://www.quarterdeck.com), is in the same league. In fact, WebCompass has morefeatures than this town has crooked cops. The beta version I checked out lookspretty good. WebCompass 2.0 gives you folders for organizing your searches bytopic and for managing the Web pages you’re interested in. It monitors thosepages for changes, and its agents run your queries for you at scheduled times,notifying you if anything’s new.
Offline Browsers
Once you’ve found the pages you’re interested in, you’ll probably want tokeep an eye on them. But only a sucker would visit every one of those bookmarksevery day, looking for changes to each site. That’s where offline browsers comein.
Offline browsers download and sort entire Web sites, including their links,or just certain pages. You can also use them like search assistants to runqueries again and again. Just set a standing query (such as the weather inMalta), and that’s it.
Some offline readers let you read news or other announcements; othersfunction as research tools. Like search assistants, some are more trouble thanthey’re worth.
If you’re short on scratch, you’ll love FreeLoader 2.0 (http://www.freeloader.com). It’s a free program that integrates with Netscape Navigatoror Internet Explorer to download sites and topics. It lets you schedule thedownloads so you can avoid the Net’s peak hours. In return, you have to put upwith the ad banners alongside its toolbar, but that’s a small price to pay.
In the same vein, but offering less flexibility, is PointCast(http://www.pointcast.com), which delivers news on your favorite topics, using aspecial Windows screen saver to show you the headlines. While you can choose whatkind of news you want to see (business, political, financial, sports) and howfrequently you want that news updated, you’re limited to the menu of news choicesthat PointCast provides. It’s mainly just a nice way to scan the headlines, not atool for finding specific information quickly. PointCast is free.
FirstFloor’s $24.95 Smart Bookmarks 2.02 (http://www.firstfloor.com) isbilled as an enhancement to your browser’s bookmarking feature; however, it’sreally a separate program that not only manages bookmarks but also downloads Webpages for offline browsing. It takes a little time to learn how to use it, butthe effort pays off. Once you’ve created a bookmark for a site or a query thatyou’re interested in, Smart Bookmarks monitors the site for changes. When thereis one, the program tells you–and you can tell it to download that page and thepages it links to.
If you need a more powerful offline browsing utility, try Open Market’s$29.95 OM-Express
(http://www.openmarket.com). Among these utilities, thisoffline browser is the easiest to use, and it comes with a simple, cleaninterface and a straightforward quick-start guide. OM-Express shows Web pages’original URLs in your browser even when they reside on your system.
Want to test out the latest agent technology for free? Empirical Media’sWiseWire (see “The Simple Art of Searching”) is a Web-based agent that learnsyour preferences as you use it, presenting increasingly well honed selectionseach time you return. However, it’s slow and awkward to use. It’s an interestingglimpse of the future of online searching, but it’s not an effective search toolyet.
If you’re serious about online research, take a look at Folio Retriever 2.1($39.95, http://www.folio.com), an offline program that builds on the Folio Viewsinfobase format, dumping the contents of downloaded sites into fully searchableinfobases that you can annotate and share with others. Retriever includes a fullyfunctional version of Folio Views 3.1, though it’s limited to importing HTMLpages. FolioRetriever’s power lies in letting you handle the data once you’vedownloaded it.
Chapter 5:
This Database for Hire
The kid got out of his chair. “Well, now I really better get going.” he said.”Thank you–you’ve been a great help.”
“Not so fast,” I growled. “There’s one more angle to go into.”
“But you’ve told me all I need to know!” he said. “Now I’ll be able to findall that information my boss wanted.”
It was time to break the news to him. “Ever heard of Sturgeon’s law?” Heshook his head. “‘Ninety percent of everything is crap.’ If that’s true ofanything, it’s true of the Web. Ninety percent of everything on it isn’t evenworth the time it takes to download.
“Sure, real gems of information are buried there, and everything I’ve toldyou today will help you zero in on these. But the really important stuff thatwill give your business a competitive edge–like high-quality market informationor financial data–costs money.” I paused. “It all depends on whether or notyou’re willing to pay the price.”
The Best Info–for a Price
IBM’s InfoSage (http://www.infosage.ibm.com) is my secret weapon forstaying on top of my business. Twice a day, infoSage delivers a customizedselection of business news, stock quotes, and other information to an e-mailbox or personal Web page. To set up or modify your infoSage profile, whichspecifies what topics and stories you get, you must use IBM’s quirky and awkwardWindows software, but a browser and e-mail are all you need to read the news.
Subscriptions are $24.95 per month; the first month is free. There areadditional pay-per-view charges for specific types of information, such asStandard & Poor’s company profiles.
If it’s financial data you’re after, you can hardly do better than Quote.com(at–no surprise–http://www.quote.com). Quote.com has an abundance of free stockquotes and financial information for the savvy investor; you only need toregister at the site to gain access to a wealth of information. For yet morewealth, you can pay subscription fees (starting at $9.95 per month), andQuote.com will throw in additional news stories, customizable stock charts, aportfolio tracking service, and historical stock data.
To get real depth on a company, an industry, a legal matter–almost anythingon record–the best source is Knight-Ridder’s venerable Lexis-Nexis service(http://www.lexis-nexis.com). Lexis, a huge archive of laws, cases, and publicrecords, has been around since 1973. Its companion service, Nexis, providesaccess to the full-text archives of thousands of newspapers and magazines, a newsclipping service, and more.
The catch? Lexis-Nexis costs more than a big night on the town. Subscriptionsare $100 or more per month, with additional fees for using many databases. But ifyour business needs access to this kind of information, it’s well worth theexpense.
For general background information on almost any topic, EncyclopediaBritannica’s Britannica Online (http://www.eb.com) provides well-researched,well-written articles that include full bibliographies–a handy feature if yournext stop is the local public library. Individual subscriptions aren’t cheap:It’ll cost you $14.95 per month, or $150 per year plus a one-time $25registration fee, but that still beats the two-ton book set handily.
Chapter 6:
Sorry, Wrong Detective
“You’ve been a big help to me,” the kid said. “How can I ever thank you?”
“Don’t mention it,” I replied. “Just leave your money on the table.”
This is a good business, I thought as I fingered the C-notes. But it couldn’tlast. Someday–maybe not today, maybe not tomorrow–I knew I’d be out of a job,replaced by computerized agents and artificial intelligence.
But until then, I was going to be living in Fat City–and I needed somebodyto celebrate with. I fired up my browser and looked for the e-mail address of anold flame.
Unfortunately, there were several Jessica Fletchers in the Four11 directory(http://www.four11.com), so I picked one at random and sent her e-mail. Shereplied within the hour. Sounds like a great lady, and a soul mate to boot: She’san amateur detective. This could be the beginning of a beautiful friendship.
Find files mentioned in this article, including a customizable search page,on PC World Online (http://www. pcworld.com/dec96/bigseek). Dylan Tweney(dft@pobox.com) is an associate reviews editor for InfoWorld. Yael Li-Ron issenior online editor of PC World Online.
Link: Searching is My Business: A Gumshoe’s Guide to the Web
Link broken? Try the Wayback Machine.
