Archiving Web Sites (2018)(lwn.net) |
Archiving Web Sites (2018)(lwn.net) |
The below code does the job of saving the page as a single file.
```
const page = await this.browser.newPage()
const response = await page.goto(url, { timeout: 50000 })
if (response.status() === 404) {
await page.close()
throw new Error('not found')
}
// credit: https://stackoverflow.com/questions/54814323/puppeteer-how-to-download-entire-web-page-for-offline-use
const cdp = await page.target().createCDPSession();
const { data } = await cdp.send('Page.captureSnapshot', { format: 'mhtml' });
const htmlFilename = "./data/" + slugify(url)+'.mhtml';
fs.writeFileSync(htmlFilename, data);
```So far, SingleFile looks like a perfect fit, thanks!
produces repayable archives in a format that doesn't have the issues of WARC. It's a work in progress tho and currently archives everything, instead of just what you select.
You can of course edit the archive by hand. it's very simple to do, but not as simple as being able to select only what you want.
I think there's a whitelist Domain option that works per archive.