How do you deal with User Generated Content (UGC) SEO spam?

3 points by hyperknot 1 year ago | 9 comments

I think anyone who allows UGC on a web app realizes that they will be targeted by SEO spam at some point. Nofollow definitely doesn't deter SEO spam efforts.

My question is, how do big platforms handle this? I thought it would make sense to put things on a separate domain, so bad reputation won't destroy the main domain.

But how can chatgpt .com/share exist then? It's on robots/allow and it's using the exact same domain that ChatGPT runs on. As I see, their search results are simply "noopener" which is super strange.

My question is, as a website owner, what strategies would you recommend for handling UGC? So far, my best idea is to buy a separate domain and let it handle it.

mooreds 1 year ago |

We had a big SEO spam problem with our forum, even with verified email addresses. I guess it is pretty easy to get a gmail address.

We fixed it by:

* moderating everyone's first post

* banning spam accounts liberally

* only allowing folks with a certain level of karma to post without moderation

* deleting any spam posts that sneak through, then warning or banning the poster

I think you need to spend some time managing your UGC.

hyperknot 1 year ago | |

Luckily I'm still at the planning stage, no UGC problem yet. I'm thinking of launching a new app where a single click share would be a feature worth implementing, but definitely not part of the core functionality. I'm trying to figure out if this feature is worth implementing at all, or it's just asking for trouble.

mooreds 1 year ago | | |

What are the benefits? The costs are pretty clear (time spent moderating), but I'm not sure what the benefits of the feature would be.

That is probably worth spending time thinking about.

solardev 1 year ago |

Do you do any moderation at all? Usually some combination of Cloudflare + Akismet + manual review (or LLMs these days) can drastically cut down on spam. Much more expensive than just adding HTML attributes, of course.

And unmoderated share links should probably be noindex and nofollow anyway, no? Why would you want search engines to crawl that?

hyperknot 1 year ago | |

I'm thinking of launching a new app where a single click share would be a feature worth implementing, but definitely not part of the core functionality. I'm trying to figure out if this feature is worth implementing at all, or it's just asking for trouble.

solardev 1 year ago | | |

I always appreciate shareable permalinks (if that's what you mean), but I think that's a different question than SEO.

Personally I would expect share links to be unlisted (not indexed by search engines, not publicly discoverable, with a long enough uuid to be effectively unguessable). "Publish publicly" could be a separate function you add later, with moderation?

PaulHoule 1 year ago | |

Personally I don't think "nofollow" and "noindex" are effective the way many webmasters think. I think they're a scam that Matt Butts came up to gaslight webmasters, right up there with all the other Matt Butts schemes to gaslight webmasters.

You're right though that you have to moderate the stuff. It's typical backwards Google Economy thinking that have to block spam to please Google (that's how an SEO spammer thinks, it's the only thing they think) as opposed to block spam so that your users will keep coming back. Letting your web site get spammed is like fertilizing your flowers with herbicide.

solardev 1 year ago | | |

I don't know who Matt Butts is, but what do you mean? Those are just standard tools, along with robots.txt and a siremap, for telling well-behaved crawlers what to index and not. I've used them effectively across many sites. They won't help with rude crawlers (LLM training etc.) that ignore those signals, but should work with all major search engines. What am I missing?

But yeah, I think spam should be deleted altogether, not merely hidden from search engines.