Ask HN: How to scrape and is it legal?

1 points by newyorker 15 years ago | 0 comments

I want to put together a website with a collection of data from other websites. I'm simply putting all the info in one place instead of many.

What's the best way to scrape this data? UPDATES: What if the data changes, will I be able to set it to rescrape or do I have to do it manually? Is it legal? I know its someone else's data, but is it the same as copy and paste?

*I have no background in scraping and only a little to make the web app, so any advice would be very helpful!

Thanks

apedley 15 years ago |

Most websites have terms and conditions saying that automated retrieval of their content is prohibited.

It is generally all up to the Terms and Conditions of the site.

If a site wanted to share it's data it would normally give you an API.

You could email them and ask for permission if they do prohibit it, depending on your purpose.

And scraping does have its drawbacks such as changing site design and having to change your code with it.

The best and only way to scrape is just to retrieve the webpage in your application and parse out the results you want. You could do things such as XSS but it all depends on the complexitiy of the data. Normally the split and substring and indexof functions are what you will use.