The pieces exist, you can string them together with a Perl one-liner. You are interested in the set intersection of the following two topics:

Full indexing: <https://lesbonscomptes.com/recoll>, <https://userbase.kde.org/Akonadi>, <https://addons.mozilla.org/firefox/addon/falcon_extension> (If you're not content with a piece, then research substitutes on <https://alternativeto.net>.)

Recent: `.local/share/recently-used.xbel`

This does not help with the email part because email programs do not register opened messages in recently used. Work-around: install a DBus or AT-SPI hook and write your own database of recently opened messages.

Happy hacking!

I've seen this been called a 'personal search engine' before. One person who is well known for their personal search setup is thesephist[0]. My friend is also working on an extension that uses NLP to semantically index your browsing history so that any text on the internet can be turned into a hyperlink to something else you've read[1].

[0]: https://thesephist.com/posts/monocle/

[1]: (WIP) http://espial.uzpg.me

I've read comments from people (don't remember the forum, perhaps HN) where people have said that they did this. No idea if there's a public project for this that you can use, but people have definitely done it. I agree that it'd be useful to have, though you probably need a good way to filter out irrelevant stuff.

Try looking at karilicoss' promnesia and it's background for similar ideas and tools.

I’ve been working on exactly that! [0]

My info is in my hn profile, if you (or anyone reading) would like to chat about it.

[0] https://apse.io

For Windows, Google used to have something like that. Because it's Google, it has since been discontinued [1].

Mac's finder is close to what you have described, and works reasonably well for me.

On Unix, this sounds like something a grep one-liner (maybe with some document depacking/packing pipe for Office documents) would do.

[1] https://en.wikipedia.org/wiki/Google_Desktop

I considered doing this - take screenshots of your screen constantly, OCR them and index them. It's fairly simple. However, there are some problems

- OCR constantly running in the background is power consuming - What granularity do you take your screenshots? Imagine each screenshot is 500 Kb and you take one each second. This'd result in 40 gigs of data per day. How are we gonna store it? How many days data do you want to keep?

There was once a thread here on hacker news about missing features of operating systems, as such a thing could only be achieved on OS level... can't find it anymore, unfortunately. It was mentioned that passwords etc. could be a nightmare. A feature like that would be a big dream of mine: Some kind of an individualized semantic archiving processing and a vector search engine to search through it.

Open-text question answering. Just make your own; index all paragraphs of all documents using TF-IDF as you access them, then when trying to search for something, use this index to get a set of candidate paragraphs and run them through BERT-QA trained on SQuAD v2. You can extend this to the content of images - first run image captioning using CNN and transformers, then index the resulting paragraphs the same way (in both cases, include a link to the original in the metadata). You might need to write some browser plugin/system driver to do it automatically as you access documents/images.

There used to be a project that kind if did this: https://beagle-project.org/. It's long since defunct and I'm always surprised that nothing emerged to fill the gap?

EDIT: I did a bit of Wikipedia rabbit holing only to discover that tracker [1] is currently running on my computer and indexing my files

[1]: https://en.wikipedia.org/wiki/Tracker_(search_software)

I think windows and Mac both do this by default, no? Just disable the web search and your local full text is what you're left with.

For the web part, there's a tool called Recoll, and a browser plugin Recoll-we.

this has been built before. the problem is that it also needs attention for ranking

you know,,,, there was a april fools day annoucement on torrentfreak years ago, maybe a decade, it was describing this behaviour. that was nice

Browser extension OCR: From the homepage "Project Naptha automatically applies state-of-the-art computer vision algorithms on every image you see while browsing the web. The result is a seamless and intuitive experience, where you can highlight as well as copy and paste and even edit and translate the text formerly trapped within an image. "

They probably put a descriptive title and then had a mod edit it to use the non-descriptive page title instead, as per HN's usual policy and practice.

I never heard of it and I clicked because the name intrigued me. You might benefit from a link preview extension to reduce the friction to explore links before comments.

Adding the "subtitle" from the site itself would provide at least a bit of helpful context: "highlight, copy, and translate text from any image."

True, that said, it made it to the front page... and while not everything on the front page is super interesting, it does give it a bit weight on the dice roll fun.

Macs and iPhones have been doing selectable text for a while now. Especially if you use Safari instead of Chrome. All images have selectable text since Monterey and iOS 15. It existed to a lesser extent in previous years.

Some of the features Naptha aims to do are not part of it though, like translating, removing the text from the image, and some other right click behavior. On the other hand, Apple is using this without any connection, which Napha says it is availble but with degraded quality and slower speed. Apple probably uses their optimized ML silicon, so I would imagine is is more battery efficient for using on the go.

The Mac built-in tool does not work on Youtube videos, which is one of my use cases for the Copyfish OCR tool. This, and the integrated translation.

Android does this too, though you have to go into the app switcher for it to work. Super useful.

Yes I think you can do it in preview too.

Copyfish is a good alternative to Project Naptha and works in Firefox:

https://ocr.space/copyfish

Yeah, I wonder if there's a more modern version of this somewhere. On-device OCR is probably good enough now that a fully-offline-by-default version of this might make sense, and that's something I might actually use.

hehe, I was thinking the same, then I read your comment.

I wish they give support to Firefox in the future, getting text (and even modifying it!) is something I need to do often.

Ahh, I did an Algolia search just now and found the 2019 post[0] but thought that felt a bit recent for what I remember. I can't find that 2014 post, however - do you have a link?

[0] https://news.ycombinator.com/item?id=20919147

i remember the last time i saw his name was on https://news.ycombinator.com/item?id=14894653 (which i guess lost against tensorflow.js?) in around 2017 or so.

There are probably some hacking angles too, e.g. I wonder if the API will helpfully tell me the contents of https://mybank.com/user/latest-statement.jpg or whatever

It works on all images in Safari on macOS as well. It's actually been helpful but was very odd when all of a sudden the functionality appeared out of the blue.

I want something similar for digitization of photos from whiteboarding sessions.

I think it would be an awesome way to capture notes from design sessions while still allowing the fluidity of a real whiteboard.

OneNote's recognition of my handwriting--once described by a tutor as 'dogs dribble'--is nothing short of miraculous.

OneNote only does recognition for search and by text-block, you can't select as in the OP; would love that feature.