Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.
Not once have I had someone I work for in this capacity even ask about robots.txt
> All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.
For AI, wasn't the book first "reproduced or transmitted" into a training system?
How would we reformulate it today?
Make of that what you will.
Whilst I assume you meant to say “scrape”, I can’t help but appreciate the poetry of that typo.
I suggest doing it, actually. Could be a fun project. I suggest not making the script make any extra requests on your main account just in case - anything already on the page is free for scraping.
News at never.