Turn entire websites into LLM-ready data

altdataseller 2 years ago |

What user agent does it use and do you obey robots.txt?

* apparently not as I tested it with LinkedIn.com which blocks most non-google/bing bots and it crawls it OK

super256 2 years ago | |

> FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and caching

They probably ignore robots.txt

padolsey 2 years ago |

Do you have pay-as-you-go pricing? I feel that’s always missing from things like this. Cool otherwise.

jshchnz 2 years ago |

This is cool, seems like a nice way to easily add context for ChatGPT at the least

cpeffer 2 years ago | |

* Creator here - Thats the goal!

donohoe 2 years ago | | |

And you honor or ignore robots.txt?

cpeffer 2 years ago | | |

It wasn't in our initial version (we didn't plan on launching today), but we are pushing an update to do so now.

pryelluw 2 years ago |

Does this respect any anti AI related scraping rules set forth by website owners?

duskwuff 2 years ago | |

It doesn't even respect general robots.txt exclusions, so... I doubt it.

nyolfen 2 years ago |

cpeffer 2 years ago | |

It crawls webpages (finds subdirectories), handles JS blocking with fallbacks to headless browsers, and does this all concurrently.

If only that script worked for every website. But, alas, it does not.