I made an open source Windows app to rewind and search everything on screen(tonoko.notion.site) |
I made an open source Windows app to rewind and search everything on screen(tonoko.notion.site) |
This is tremendously helpful for people like me who forget stuff often.
I would propose you are both absolutely correct.
> Please refresh and try again or.
Copying ffmpeg to C:\Windows\System32 doesn't seem to be the correct way to install it.
This is the windows way. In the early days (Windows 95/98) it was indeed common practice to put shared dlls and exes in C:\Windows or C:\Windows\System32, but without central oversight this was a nightmare: version conflicts were rampant, and uninstallers didn't know what could be safely removed. Everyone switched to installing everything into the program's install dir, and the world was much better off.
Some poweruser tooling gets fancy by checking if ffmpeg is installed systemwide, and if not downloads it to the programs appdata folder. But that exposes you to versioning issues if that version is too old or too new
Anyway, in my lab environment where I had this running .. over a decade ago now I guess .. I set up scripts to automatically cryofreeze the daemon processes I was writing, so I could indeed reset to the state in time, of interest during testing. This lab got re-comissioned for other things, and a needed kernel upgrade broke cryofreeze, and I never got back to it - but I have often wondered about it, as a temporal interface to some things would make for a whole new world of useful UI and other interation paradigms to explore.
I remember there was something like this possible on WANG and Tandem systems back in the 80's, and it always has kind of piqued my interest as to why this isn't still considered a thing. Well, I guess syscall complexity has vastly changed since WANG and Tandem were around, lol.
Anyway, I'd love to be able to do this on Linux again - I keep thinking to catch up with the state of the art of process freezing/unfreezing, so this has motivated me to do this - although I do confess that, as principally a Lua/C developer in my chosen market (embedded), I attain this re-playability in other ways (state sync'ing) which also allows me to move processes to other systems, relatively smoothly, also.
Still, would love to have this for normal Linux processes, out of the box. Am I ignoring an obvious way to do this?
http://www.cs.columbia.edu/~orenl/papers/sosp07-dejaview.pdf
Abstract: As users interact with the world and their peers through their computers, it is becoming important to archive and later search the information that they have viewed. We present DejaView, a personal virtual computer recorder that provides a complete record of a desktop computing experience that a user can playback, browse, search, and revive seamlessly. DejaView records visual output, checkpoints corresponding application and file system state, and captures displayed text with contextual information to index the record. A user can then browse and search the record for any visual information that has been displayed on the desktop, and revive and interact with the desktop computing state corresponding to any point in the record. DejaView combines display, operating system, and file system virtualization to provide its functionality transparently without any modifications to applications, window systems, or operating system kernels. We have implemented DejaView and evaluated its performance on real-world desktop applications. Our results demonstrate that DejaView can provide continuous low-overhead recording without any user noticeable performance degradation, and allows browsing, search and playback of records fast enough for interactive use.
TL;DR: It's an old PhD project that requires custom patches to an ancient kernel version. So I guess you can't download it, and even if you could it wouldn't work on any system you'd want to use today:
> compsciphd on Oct 11, 2022 | parent | context | favorite | on: Linux NILFS file system: automatic continuous snap...
> we used NILFS 15 years ago in dejaview - https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf
> We combined nilfs + our process snapshotting tech (we tried to mainline it, but it didn't go, but many of the concepts ended up in CRIU though) + our remote display + screen reading tech (i.e. normal APIs) to create an environment that could record everything you ever saw visually and textually. enable you to search it and enable you to recreate the state as it was at that time with non noticeable interruption to the user (processes downtime was like 0.02s).
https://news.ycombinator.com/item?id=33165519
> compsciphd on Oct 12, 2022 | parent | next [–]
> sadly (as with much work form phd students like I was), the closest one could get to it today is trying to duplicate it. i.e. combining criu with nilfs (but a lot of the work that we did to get process downtime to minimal numbers requires being in kernel, as described in paper) and unsure criu can do it.
> In addition our screenrecording mechanism was our own "proprietary" (not really proprietary as fully described in research papers, but also not a standard) and something that was built as an X display driver 15 years ago (so not directly usable today even if code is available). Could probably duplicate it with vnc based screencasting. vnc didn't work for us as we needed better performance (i.e. it was built to demonstrate remote display of video and games and there was no real remote audio setup back then so we had to create our own).
> the "text" search just used gnome's accessible API much like a screenreader would do (with a bit of per application optimizations as can filter out things like menus and the like, primarily was to dump text out of terminals, firefox and perhaps open office and maybe even a pdf reader if memory serves me correctly, but a long time ago).
I've been looking into it myself though, mostly for forensic concerns (capturing state/evidence in a way that's harder to forge than screenshots). Hopefully, running the desktop environment through VNC and then running `criu dump --leave-stopped --prev-images-dir …` immediately followed by `mkcp --snapshot …` (or `btrfs subvolume snapshot …` would also work, I guess) would be enough for basic functionality.
This one works : https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf
This one’s broken: https://www.cs.columbia.edu/~orenl/papers/sosp07-dejaview.pd...
When I search for “dejaview” in the main index, I get the same broken link in the search results[0]. At first I thought they’d changed the URL structure (/pubs/ to /papers/), but if you visit [1] it works, but [2] doesn’t work. I guess “orenl” isn’t a member of the faculty anymore, so they tore down their page and removed all the associated resources.
[0] https://www.cs.columbia.edu/g-search/?q=dejaview#gsc.tab=0&g...
https://github.com/jasonjmcghee/rem
Previously submitted on HN by the dev with some decent discussion 3 months ago here:
1) Make a Meta Smart Glasses take a photo every 2 seconds.
2) Send images to some server in the cloud.
3) Run OCR and object detection/labeling on these images.
4) Then present an app that allows searching (and chatting with) your past.
I could then ask the LLM things like:
1) Where did I left my wallet?
2) Did I get my credit card back after paying the restaurant yesterday? (ADHD things, don't ask)
3) What was written in my daughter's new tshirt today?
Bonus point if the app also records and transcribes audio so you could ask the LLM things like:
1) In the last meeting, what was the deadline that we settled on?
2) What was the phone number of that person I met in the park earlier today?
3) What was the name of the investor I met today?
Bonus bonus point if it has access to your phone calls to so it can transcribe and index what others said.
See Black Mirror episode "The Entire History of You": https://en.wikipedia.org/wiki/The_Entire_History_of_You
I concluded that in case this is a viable approach, Microsoft or Apple will built that into their OS natively as part of a copilot that remembers everything and comes to assist the user with the knowlede.
My screen history was not as advanced as the app mentioned here though. And I didn't use it myself.
> Please enable JavaScript to continue.
I run NoScript, and the way this block is constructed makes it impossible to actually enable JavaScript to run the site.
Please don't redirect to a static page, because that page doesn't have any of the JavaScript present to enable on a case-by-case basis, the the redirect happens so fast I have no chance to enable JavaScript on the main page.
Just enabling for notion.so itself is not sufficient to bypass the block.
Also I'm surprised html even allows for non-JS redirects.
Original Savant source? https://bitbucket.org/theluxury/savant/src/master/
Videos of how the later Atlas Recall worked: https://www.youtube.com/@atlasinformatics3316/videos
I know my screen can be checked at my current client, but I also know the product owners don't really care as long as the work is getting done on time (or at least they haven't bugged me about it since I've starting working with them a year ago and have given very positive reviews about me to my company).
I think employers would be more interested in basic things like “did this person login their computer today, between what times were they active, how many hours in the day did they type at least 100 words?” … or things like “here’s a list of people on PTO today, which people not on this list haven’t typed on their keyboard today?”
That sort of data would be easy to collect - keyloggers have been around forever, doesn’t trigger MacOS to show the “recording screen” notification, and the data is easier to aggregate and view across a large number of computers.
I think the fact that keylogging didn’t explode as a method of tracking productivity during Covid WFH probably means we’re safe from screen recording for the foreseeable future.
It seems to create a huge risk for your data though. Any ideas about how you might go about securing against the application itself doing something untoward?
I’ve long thought that ideally an auto-archiving transparent proxy would be a more elegant approach.
There's a huge miss in this implementation though: 95% of the time what is on screen is web-browsing. So to go from nicely formatted markup with title tags -> video -> OCR is clearly missing am ore obvious path (either a proxy or browser extension).
I've often thought that screen readers might be better off now by attempting this visual approach. It used to not be possible, now it may be easier than relying on accessible markup.
And yes, most of my screen time is also web-browsing. Currently, I can use the web page's Windows title as a reference, which provides quite a bit of information. To enhance this feature, it might consider using like Chromedriver to gather more details, such as web links and page text, similar to what 'Rewind' offers.
I distinctly remember seeing it somewhere (I think it was here), but I can't find what the app was called, despite all teh gooling.
I made one called flyonthewall, you can upload a m4a file and it gives back transcription from openAI whisper model. Basic setup (no accounts right now) that I use to transcribe plus summarize my audio notes from my dictaphone.
I made it, for full disclosure.
Come join in on development!
It's MIT Licensed.
I also kicked off a cross-platform version in rust https://github.com/jasonjmcghee/xrem which is earlier in development and could use even more help.
My main and daily use case it to look back at the previous day - this is helpful for standups, retros, and so on. I skim through my days (sometimes weeks) to pick up on what I was working on - it's incredible how much "untracked" work is performed that you pick up on. Sometimes I forget who exactly I talked to about something, but just knowing the rough date I can usually find something to jog my memory.
Obviously I am biased, but still!
You can do things like find the point in history where you fixed a bug and go watch yourself debug and put in the fix.
Pretty wild. It makes a lot more sense once you experience getting the value.
Personally I hope Apple adds the feature natively to the OS at some point. They're plenty experienced to build it themselves over there, but maybe having a reference implementation will encourage them to give it a shot.
I'd love to contribute but I know nothing about Swift, mine was all in bash scripts with launchd to run them.
https://twitter.com/AlexBlechman/status/1457842724128833538
> @AlexBlechman
> Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale
> Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus
Remove cloud from the equation, replacing it with a properly safeguarded fully owned hardware located on your own premises - basically make it privacy-respecting, minimizing abuse potential, do a proper audit, and it becomes a super desirable system for people with neurological differences (or just stressed folks - attention and memory suffer under the stress too), a glasses that enhance memory rather than vision. Sure, not without caveats and gotchas (don’t bring it to a poker night lol), but not that cyberpunky-bad either.
a local model that can analyze the videostream/files and answer questions would be great.
A restaurant should never be taking your credit card out of your view, ever. Really, they should not even be handling it: you should be swiping it yourself at the terminal at the cash register when you pay before leaving.
I'd never seen a restaurant do that in the UK.
I think it's a relic of the times before handheld payment terminals existed.
Also, having to decide on a tip percentage directly in front of the waiter is awkward sometimes, doing it the US way avoids that. Tipping is very common in the US.
I’ve preordered them.
Excellent product for a consultant like myself to have on hand to document everything happening on their machine. The ability to change the screenshot interval and add notes along the timeline are real power tools for someone involved in multiple projects every day. You can secure your time record snapshots with a password. Playback at multiple speeds so you can jog your memory by browsing the timeline to the exact time when you flipped to something else. Classify by project, client, etc.
Really a great bit of software.
I once found one of my invoiced time periods being challenged during a regular audit of one of my client's operations. I frequently billed long hours since I worked long hours on 24-hour call (remote oil and gas data processing and support of international operations). The auditor called and requested anything I had that could support a particular time period where I billed "excessive" hours (>20 consecutive). I told them I would get them everything that I had and asked whether they were interested in any other dates or time periods or work for specific projects. They narrowed it to one single invoice entry so that made it simple for me.
Thanks to TimeSnapper I was able to produce a GIF movie of the screenshots (I always used a 1 minute snap interval to allow fine-grained operations review) during the requested interval that documented everything happening on screen and I provided a PDF of the notes dumped from that interval documenting phone calls taken, etc. I also sent copies of my handwritten notes I always maintain as primary memory-jogging tool. In the return note to the auditor I asked if they were interested in any other dates or time periods and offered to send an archive on their request.
The auditor called me up and we talked for a while and they thanked me for the detailed records and told me they wouldn't need anything else.
It really helps that my wife is a CPA/Auditor/Fraud Examiner. Tools like TimeSnapper Pro helped me in my business to maintain excellent records about day-to-day operations that made it simple for me to invoice clients for all the time that I spent on their behalf.
Thanks /u/LeonB
Prior to TimeSnapper for Mac I ran my own little script, and a companion one for Linux. Now I have TimeSnapper on my Macbook Pro too. And if you ever create a Linux version (or want someone to create a Linux version), let me know. I'd be happy to beta test.
I've lost count of the number of times that TimeSnapper has saved my arse, either with accidentally deleted data, finding an obscure web page, proving I did something on a specific date/time, or that something happened in the way I say it did, or recovering the keys to a small amount of Ethereum and Dogecoin stored in a wallet.
Thank you for TimeSnapper. It has been interesting to use it for that length of time.
"Four weeks work in 9 minutes" captured by TimeSnapper where I create a video game (or three) in October of 2012 http://www.otakunozoku.com/video/working.flv
Sorry, it's an flv, along with an onion on your belt, it was the fashion back in the day
Surprised to see that "Maintain aspect ratio" is only there in the pro version. Maintaining the aspect ratio should be a basic feature.
It looks like the first version was available ~ mid November 2005 (via https://secretgeek.net/snapper_wins) and available from the TimeSnapper.com domain from ~ November 28 2005. (https://secretgeek.net/TimeSnapper_ready_at_last)
So you got in very early!
The professional version was a year or two later.
cheers
It's possible to not record specific window IDs (at the code level) - so you'd need to be able to detect whether the window is "private". I am not familiar with such a flag.
Alternatively you might be able to just have a lookup or regex of all the most popular private browsing / applications...
Or let a user specify a pattern to look for and skip.
But at the same time, it's your private data and it's not going anywhere. There's no telemetry or network access of any kind.
Anyway- doesn't exist as a feature today. Maybe someone (you?) will build it!
15 years is a long time, performance wise VNC would be adequate and we're at the point now where OCRing a render is probably fast enough for most use cases (it's actually the most reasonable choice a lot of the time, e.g.: PDF parsing).
(But how's VNC support looking on Wayland based window managers?)
For what kinds of surveillance? Why is the forgery-proofing important for the playbacks?
another project I created on the forensic side (steve bellovin asked the Q and I was like, yeah, I know exactly how to build thta) that might then interest you was something we called ISE-T (I See Everything Twice - Catch 22).
https://academiccommons.columbia.edu/doi/10.7916/D8HQ45MK
Two-Person Control Administration: Preventing Administration Faults through Duplication
Modern computing systems are complex and difficult to administer, making them more prone to system administration faults. Faults can occur simply due to mistakes in the process of administering a complex system. These mistakes can make the system insecure or unavailable. Faults can also occur due to a malicious act of the system administrator. Systems provide little protection against system administrators who install a backdoor or otherwise hide their actions. To prevent these types of system administration faults, we created ISE-T (I See Everything Twice), a system that applies the two-person control model to system administration. ISE-T requires two separate system administrators to perform each administration task. ISE-T then compares the results of the two administrators’ actions for equivalence. ISE-T only applies the results of the actions to the real system if they are equivalent. This provides a higher level of assurance that administration tasks are completed in a manner that will not introduce faults into the system. While the two-person control model is expensive, it is a natural fit for many financial, government, and military systems that require higher levels of assurance. We implemented a prototype ISE-T system for Linux using virtual machines and a unioning file system. Using this system, we conducted a real user study to test its ability to capture changes performed by separate system administrators and compare them for equivalence. Our results show that ISE-T is effective at determining equivalence for many common administration tasks, even when administrators perform those tasks in different ways.
I should note that the paper also discusses that 2 people might be expensive, so the same mechanism can be used by a single admin but in a manner that maintains an audit trail.
The above project wouldn't require any kernel modifications as the work was all about using unionfs (using normal vfs loadable module interface hooks) to capture changes and user spaces to log and compare them.
All this work led to what can be viewed as a proto-docker - https://www.usenix.org/legacy/events/atc10/tech/full_papers/... and https://www.usenix.org/legacy/events/lisa11/tech/full_papers...
Your hard work and thoughtful programming not only made it possible for me to document my efforts no matter what I was doing on my machines, but it made it easy.
That is the true measure of success with any programming effort.
You created a product that 1) I could install on my machines, read through the documentation and understand how to make it work for me without any friction since it is well-documented, 2) I could parameterize my record-keeping (even allowing changing things on the fly) to fit my own unique situation so that I could get compensated fairly for all of my time that I spent dealing with my client's problems, 3) I could archive and export in common formats to form the basis for reports that I needed to satisfy client requirements and help them determine how much value as a consultant I added to their projects.
Bravo! It's a great piece of work.
Before choosing it I ran side by side comparisons with other similar apps including some that stored all my usage data in a cloud or in a proprietary format with hoops to jump through to be able to prepare invoices and reports. The pure simplicity of using TimeSnapper Pro and the fact that I had all the usage data locally stored where I didn't have to depend on someone else's cloud availability when an invoice was due and all the other well-designed features including the ability to lock it all behind a password made my choice simple.
For most of the shitty projects I worked on I billed by the minute hoping that some asshole would challenge that, knowing that I had everything I needed to support that granularity. I really wanted them to come back and challenge some of the other invoiced time. Those ducks are still in a row if I ever need them thanks to my usage of your software.
I frequently worked really stupid hours, sometimes billing more than 40 hours in a 48 hour period when they hit me with a time-sensitive task that had to be completed now. I remember too many times getting a call late on Friday telling me about new data that needed to be processed for a meeting on Monday morning. I got it done and documented all my time and got paid.
You're the bestest!
I have to say -- I'd rather you worked sane hours, and pushed back a bit. You only get one life, and giving too much of your time to (for example -- as I've spent years in that industry too) the oil and gas industry isn't everything in life.
"You work sixteen hours and what do you get? Another day older and deeper in debt" --some old blues legend probably
Yet the end too was quite predictable and logical -- frequently destroying those of this species simply because your own life and mind ends up becoming more tempting and destructive than even the most enticing of drugs. One could simply lose themselves in your own memories. What need is there for the rest of your life when you can simply endlessly relive, in perfect clarity, the best moments of your life - ones that you, in many cases, will likely never surpass?
Even without perfect memory, this can still happen, especially if something like PTSD is involved. I myself have a dissociative disorder and sometimes it can be difficult not to just relive the past over and over. (It's not like this is a daily struggle, but during low points it comes up.)
- the science fiction novella "The Rememberers" by A.E. van Vogt, originally published as "The Book of Ptath" in 1943.
- the species known as "Trills" from the "Star Trek" universe, particularly in the series "Star Trek: Deep Space Nine."
Edit: the first suggestion seems to be a hallucination
Also you might not realize how much the hard of hearing, deaf, and other neurodiverse ADHD types would find a memory prosthetic like that very useful unless you are in their very own shoes, for possibly years.
Anyway, posting this comment so I remember to send it to her tomorrow.
So there's nothing fundamentally wrong in taking inspiration from sci-fi.
Sure the presented perspective is negative but that is because the producer explored local maximums to the benefit of the audience. They are obliged to paint technology in a certain way to captivate viewers.
Does that mean that the presented reality is the only possible one? Certainly not, that would be reductive.
Many popular technologies today are used to scam a defraud people. Does that mean said technologies are fundamentally bad? Not at all.
I use technology as a force multiplier.
If you’ve ever attempted Travis picking, and you haven’t spent a heck of a lot of time practicing it, you may recoil in horror at the name.
Looking deeper into it - “Travis” picking really goes back to Arnold Schultz (https://en.m.wikipedia.org/wiki/Arnold_Shultz) — who is practically lost to history, having never been recorded.
I can understand it being hard to reckon with when one’s primary understanding of government stems from a familiarity with countries founded in accordance with the desires of a handful of pre-existing, regional groups (as opposed to a country founded with explicit mechanisms by which new groups can achieve representation).
It was a terrible idea.