~ The amazing flying wizards ~
         to basic   

Version November 2002

The amazing flying wizards 
 (How a bunch of leet seekers enters databases)
by VVAA (Various Authors)

(edited by fravia+)


Great stuff for seekers, truncation galore, read and take note of the many tricks...

Everything began with a simple and interesting question...
site specific searches/filenames ?

the-scientist.com publishes some articles that are "hot":

Data derived from the Science Watch/ Hot Papers database and the Web of Science (ISI, Philadelphia) [these are paid services] show that Hot Papers are cited 50 to 100 times more often than the average paper of the same type and age.

The URL for one of them is
http://www.the-scientist.com/yr2002/oct/hot_021028.html

How could we search for a list of "hot" articles in that site (because we / cannot /will not / should not / pay for the original service from Science Watch)?.

I tried in Google:
site:www.the-scientist.com inurl:hot
but it fails, I think because hot is not in the domain but in the filename
Thanks

digital gaucho
You will admit that this is an interesting question for a seeker. A typical case of 'closed database', where information is hoarded instead of being freely spread. But the fundamental law of the web is on the side of the spreaders, and very strong web-winds blow against the commercial hoarders...
Re: site specific searches/filenames ?

I think your problem is truncation... I only know to main search engines that perform this kind o search: altavista and hotbot

Here are the queries you can use:

(host:the-scientist.com OR host:www.the-scientist.com) AND url:hot_** on altavista

+"the-scientist.com" +domain:com +hot_* on hotbot

the second query brings the folowing url: http://www.the-scientist.com/hotpapersarchive.htm, maybe that's what you want.

Nemo

Truncation! How important! Moreover, those among you that have visited the first link given above by Nemo will have noticed some URLs: www.the-scientist.com/yr2000/jan/hot_000110.html, www.the-scientist.com/yr2000/feb/hot_000221.html, www.the-scientist.com/yr2002/mar/hot_020318.html, etc... mmmm :-)
Re: Re: Re: site specific searches/filenames ?

well... heres a couple my tricks (guess thats what you'd call them :)

archive org of course will give you lots of info ... however even with many url names (including nemos above) you still get delivered TOO the registration page

(I am unclear as to the problem though --- it seems to indicate it is a free reg (?) and then u can proceed to read free of charge ?) thats hat i read

I didn't register so am not clear or sure about this (butt you indicate that you have to pay)?

butt to a trick that works sometimes ... sometimes...

lets take Your posted url: http://www.the-scientist.com/yr2002/oct/hot_021028.html

now lets feed it to google ...

now lets click on googles cache of (your url... or any url at the site that you have brought up in the search engine that produces the doc that you want to read)

google s cache wisks you off to their Registration page ...
why?
because somewhere it is reading and if and or else statement telling it you are a bad guy
yes?
could it be in googles cache?
yup ...

so, back space now back to the google page
and click again on the cache link
now VERY QUICKLY before it can transfer you to the registration page HIT YOUR STOP BUTTON (if you are not quick enough u may have to try several times...

you will be staring at a blank googles cache page...
butt lets not stop there ... lets look at view source... [i tried pasting the source here but it didn't want to work correctly] ... make a copy of your 'source' and put it in an editor and view the html page :) :) works fine

not only is the code for transfering you to the bad guy page there (i guess ---im not a coder)... butt WhalaH --- also THERE you will see is the article that you wanted to read in the source :) :)



what i find even more interesting in this little project is the scientists robots txt
User-agent: *    # applies to all robots
Disallow: /surveys
Disallow: /webreports
Disallow: /eugene_garfield
now why in the world would they block out a specific name???
google shows some rather nice returns for that name :) [although i don't have time to figure out if he is listing and giving the articles essays away for free at his sites --- the pdf files seem to work --- or why his name is disallowed

??? ohoooooooooo well geeeeeeeeeeesh -- you know the clocks were rolled BACK yesterday and its only 11:20 here --- but really it should be 12:20 and by 12:20 I have had at least one beer so i guess my overactive mind isn't slowed down enough because of the lack of beeeeeeeeeeeeeeeeeeeeeeeeer ... ok all it means is this i guess --- http://www.the-scientist.com/eugene_garfield/


so lets re-evaluate
if you try nemo's page with your url on it ... and click it it brings you to the registration or login page ... they want an email address to let you in --- yes?

don't u supposse that someone who works there has an email address that lets him in???

lets ask google:

well sonofabeehive ... google lists a number of members
lets snatch the very first one's email address and try it



lets disguise this emailaddress a little here, so that the harvesting bots and others nasties wont index it:
hcohen
at the-scientist
dot com

now let's paste that above (corrected of course, with @ and everything) into the email addy into the login page ... annnnnnnnnnnd

what was that quote?? oh yes 'That's funny ...'" Isaac Asimov

sunofabee there's your article with pictures ...

(DO NOT --- mess with the guys account info should you try this --- thanks)

beertime

jeff

He did it! Among many other useful information in the snippet above (like the checking of the robots.txt), Jeff shows you an incredibly powerful access-shortcut whenever someone dares to stop seekers asking for a registered email address: they want an email address? Let's give them proper emailaddresses a-plenty! :-)
Re: Re: Re: Re: Re: on second thought

Haha, jeff, you rock! (again and as always)

My 2c: since the redirection is handled by javascript, you can just disable js in opera or bypass it with a proxomitron filter
Mordred

The voice of the rational! Google cache gives for certain that visitors have a javascript enabled browser! Indeed nowadays we all give all too often per aquired that everyone uses javascript enabled browsers per default (or flash enabled). Try browsing the wide web with lynx (see the tools page) and be prepared for some surprises! :-)
Re: Re: Re: Re: Re: Re: on second thought and third thought

hiya mor!!!
(I knew one of you js guys would know what to do!! :) :)

on third thought, as i was driving to the store and re-thinking, i feel i did something wrong in this thread

digital was trying very hard to understand google and proceed at abcdefg

I jumped him all the way to google - xyz

i should not have done that ... because his specific efforts and google-questions were not really answered by my tricks

i apologize digital ... please proceed with your questions

PS:heheh
i just finally figured out what you meant
turn my javascript, in netscape, OFF
and then click on googles cache
oh yeeees ... so much easier :) thanks!
you rock! :)
jeff

This true he went to google "xyz", even if we did learn a lot by his digression :-)
Re: Re: Re: Re: Re: Re: Re: on second thought and third thought

jeff you post was excellent, also Nemo´s that showed how to master booleans on the engines capable of them (not google unfortunately).

strangely, your findings where due to a misunderstanding: the URL is freely accesible directly, I wonder why google´s cache redirects to a sign page! will have to check that. I use proxomitron so there are no automatic redirects :-)

but sure I will apply your steps in the future, thanks
digital gaucho

The registration policy has probably changed over time, and as Nemo pointed out: "it's not so much a question of booleans, because since late 2000 you can do that on google as well: Boolean Searching on Google. You can even use the operator OR inside phrases: "advertising OR advertisement statistics". Its a problem of truncation.    You can use * on google, but it doesn't work in the same way... on google it replaces an entire word, exemple: "ad augusta * angusta".
On altavista you can use * or **, here is an explanation how they work: truncation on altavista.
You can read some more information about * on google here".

Re: Re: Re: site specific searches/filenames ?

>I wonder why Google does not allow wildcards, it is true that most often you
>get the results, but the * is very useful sometimes.

Yes, truncation is very useful sometimes... the best search engine for truncation was northernlight where you could use queries like this one: +"*lempicka*.jpg" hehehe... here is the url:
http://www.northernlight.com/nlquery.fcg?view=extend&function=search&qr=%2B%22*lempicka*.jpg%22

>Lets see if I understand your logic:
>(host:the-scientist.com OR host:www.the-scientist.com) AND url:hot_** on >altavista
>is the OR here just to get all the domains if the URL changes?
>won´t host:the-scientist.com catch all?

You're right host:www.the-scientist.com is contained in host:the-scientist.com, but I joined the two just in case...

>+"the-scientist.com" +domain:com +hot_* on hotbot
>why the +domain:com here?

Because you said that the pages are in a comercial site... and you showed a working page...
Nemo

the long way

looked at the example above and ran
url: www.the-scientist.com/yr2002/jan/hot**
at AV and shifted the months to get the following
www.the-scientist.com/yr2002/jan/hot_020121.html
www.the-scientist.com/yr2002/feb/hot_020204.html
www.the-scientist.com/yr2002/feb/hot_020218.html
www.the-scientist.com/yr2002/mar/hot_020304.html
www.the-scientist.com/yr2002/mar/hot_020318.html
www.the-scientist.com/yr2002/apr/hot_020401.html
www.the-scientist.com/yr2002/apr/hot_020415.html
www.the-scientist.com/yr2002/apr/hot_020429.html
www.the-scientist.com/yr2002/may/hot_020513.html
seems to stop there so that may be the last time AV spidered it.
did a little induction (deduction? guessing?) and came up with
www.the-scientist.com/yr2002/jan/hot_020107.html
www.the-scientist.com/yr2002/may/hot_020527.html
www.the-scientist.com/yr2002/jun/hot_020610.html
www.the-scientist.com/yr2002/jun/hot_020624.html
www.the-scientist.com/yr2002/jul/hot_020708.html
www.the-scientist.com/yr2002/jul/hot_020722.html
www.the-scientist.com/yr2002/aug/hot_020805.html
www.the-scientist.com/yr2002/aug/hot_020819.html
www.the-scientist.com/yr2002/sep/hot_020902.html
www.the-scientist.com/yr2002/sep/hot_020916.html
www.the-scientist.com/yr2002/sep/hot_020930.html
www.the-scientist.com/yr2002/oct/hot_021014.html
www.the-scientist.com/yr2002/oct/hot_021028.html
www.the-scientist.com/yr2002/nov/hot_021111.html
www.the-scientist.com/yr2002/nov/hot_021125.html
www.the-scientist.com/yr2002/dec/hot_021209.html
www.the-scientist.com/yr2002/dec/hot_021223.html
checked a few, seem to be good. except for the ones not out yet.

enjoy
dan

Once again, the power of guessing...hehe :-)
Let's just imagine that there exist on the web sites similar to the-scientist, this one, but that are not as open and freely accessible as this one is, and require a fee in order to access information (yes, it happens, alas). Let's imagine that they too have a subdirectory structure similar to the one explained above. Well, you could do the same, and 'guess' them. See, for an example, the bottom part of my flange page.











Petit image

(c) III Millennium: [fravia+], all rights reserved