Google is now officially useless.

A subtle attack may be by making bots stop indexing it or using SEO practices to lower it enough so it would become unsearchable, and therefore, non-existent.

Or just crack into Google...

Answering people's individual questions doesn't scale to the entire Internet, so Google really has no choice but to address problems on a case-by-case basis. In this case, Matt reads HN and personally wants to solve the problem. That's the only way Google could possibly work, so that's how they do it.

Yet here, a website owner is purposely blocking the crawler and they jump with solutions to try to fix the problem. Sigh.

I can usually get the hacker news by just googling for "hn"

If you have a whole bunch queries that are somehow related, like say we do a bad job at Python function names, then you can collect them and post them on our forums in one go.

And please give us details as Matt said. In particular, If your refer to a specific result, please say its URL not "5th result" because rankings fluctuate.

Lots of folks here like to give Matt Cutts this aura of an angel, or some sort of saint. Those who have known his actions over the past 5-10 years know better. Just ask guys like Aaron Wall and Rand. Matt Cutts is just a pawn that is there to do damage control for Google. That's all he is.

Seriously, one thing about Google is that they seem to really like ensuring people are logged on, preferably at all times. Fortunately recent changes to Google Apps (promoting apps user accounts to full Google accounts) has made this more complex on my side and probably degraded the level of actionable info they can get out of it.

I don't mean the ranking but other aspects - like you guys blacklisted some domains which produce low quality content in wholesale. (I don't know if the algorithm was tweaked to detect and filter such sources or if it was a manual thing.)

Clearly that's not the case, as advertising makes 99% of their revenues.

Another thing that might help google is for them to announce and support some meta tag that would allow site owners (or web app devs) to declare how likely a page is to change in the future. Google could store that with the page metadata and when crawling a site for updates, particularly when rate limited via webmaster tools, it could first crawl those pages most likely to have changed. Forum/discussion sites could add the meta tags to older threads (particularly once they're no longer open for comments) announcing to google that those thread pages are unlikely to change in the future. For sites with lots of old threads (or lots of pages generated from data stored in a DB and not all of which can be cached), that sort of feature would help the site during google crawls and would help google keep more recent pages up to date without crawling entire sites.

Giving occasional personalized help is a great way to avoid a total black hole while still creating an escalation path. It's like having EXPLAIN QUERY stack traces on NewRelic. You don't need a deep trace of every transaction; you need a deep trace from a few representative squeaky wheels, and you use that information to improve the underlying self-serve process. This is something Google, with their "1/3 of the company parses logs", excels at.

You'll note neither Matt nor Pierre offered to do anything special for PG. They didn't say "Oh! I love HN, and we want it to succeed; I'll go push out a manual override that makes sure we use only the one crawler you unblocked." They gave him exactly the info he'd get from Webmaster Tools plus general knowledge, and told him how to use Webmaster Tools to fix it. Maybe they'll go figure out how to implement that "you're blocked!" hover-over in SERPs, but again, that would benefit everyone, not just HN. There's no special treatment here.

[1] http://blog.wordtothewise.com/2008/04/troubleshooting-a-post...

The point is if you can build relationships with Google employees rather than with Google itself then you too can get this sort of amazing help.

There's nothing sinister or crazy about this. Rather than being one in a million, become one in a circle of friends or acquaintances.

PG knows his shit. He knows how to block crawlers, he knows how get SEO done, he's no dummy. "So what?" you say. Well I work in web development and I meet with people all the time and explain to them how to run a site and get rankings up, etc. I tell them I'll give them the tools, show them how to use them and even leave detailed manuals on how to work everything. These people are so into it at our first meeting and they're excited to get into it and work this stuff.

Then the job is done, the tools are handed over, and the lessons are learned. What do they do? Nothing. Then they do nothing. Then even more nothing. The site is never updated and they do no maintenance whatsoever and eventually it becomes outdated and dead for all intents and purposes. A year later they want to know why they're not ranking in Google. I ask if they've checked or even once used any of the tools I gave them and the answer is always no. Then they demand I fix it. I kindly remind them they're the ones who wanted to have Larry in accounting run the site because he's "good at computers" and to save money by not hiring an IT guy.

So the point I'm making is that I'm sure there are a ton of people out there who are trying to get customer service without trying to help themselves or using the resources readily available. I think it's great that Google is showing they're willing to help those who help themselves.

I think google isn't treating everyone equally but they are treating people fairly. There's a difference. I'd prefer fairness. Not all "customers" are created equal so you compensate by treating them with fairness instead.

the listing only shows up if you type the exact name of the hospital into the search bar, which is useless.

(I used to telnet to port 80 for testing, and type GET / HTTP/1.0 <enter> <enter>, and that should be LF on Linux & Mac)

::cough::

A few years ago, I did pretty much the same thing myself. Thankfully the late summer was our slow season and the site recovered pretty quickly from my bone-headed move, but the split second after I realized what I've done was bone-chilling.

I think just about everyone has thought at some point that they understood how something worked, only to have had things go pear-shaped on them.

The lesson: people are not fully knowledgeable about everything, even the smart and talented ones.

"You would not believe the sort of weird, random, ill-formed stuff that some people put up on the web: everything from tables nested to infinity and beyond, to web documents with a filetype of exe, to executables returned as text documents. In a 1996 paper titled "An Investigation of Documents from the World Wide Web," Inktomi Eric Brewer and colleagues discovered that over 40% of web pages had at least one syntax error".

We can often figure out the intent of the site owner, but mistakes do happen.

So, do you want to pay the price upfront when you can plan for it or afterwards when the fix must be done immediately because customers are complaining?

Obviously, that's not a problem if you already know exactly how different browsers will treat your code, or you're using parsing errors so elemental that they must be patched up identically for the page to work. For example, on the Google homepage, they don't escape ampersands that appear in URLs (like href="http://example.com/?foo=bar&baz=qux — the & should be &amp;). That's a syntax error, but one that maybe 80% of the web commits, so any browser that couldn't handle it wouldn't be very useful.

Anyhow, one downside to having syntax errors might be that parsers which aren't as clever as those in web browsers, and which haven't caught up with the HTML5 parser standard, might choke on your page. This means that crawlers and other software that might try to extract semantic information (like microformat/microdata parsers) might not be able to parse your page. Google probably doesn't need to worry about this too much; there's no real benefit they get from having anyone crawl or extract information from their home page, and there is significant benefit from reducing the number of bytes as much as possible while still remaining compatible with all common web browsers.

I really wish that HTML5 would stop calling many of these problems "errors." They are really more like warnings in any other compiler. There is well-defined, sensible behavior for them specified in the standard. There is no real guesswork being made on the part of the parser, in which the user's intentions are unclear and the parser just needs to make an arbitrary choice and keep going (except for the unclosed center tag, because unclosed tags for anything but the few valid ones can indicate that someone made a mistake in authoring). Many of the "errors" are stylistic warnings, saying that you should use CSS instead of the older presentational attributes, but all of the presentational attributes are still defined and still will be indefinitely, as no one can remove support for them without breaking the web.

There is no reason to allow most of these errors other than coding sloppiness.

http://validator.w3.org/check?uri=http%3A%2F%2Fwww.google.co...