Zotero: Personal Research Assistant(zotero.org) |
Zotero: Personal Research Assistant(zotero.org) |
But it's also just a huge ecosystem with parts that are used and developed independently. E.g., "translators" for save/import/export are used in both Zotero and in the Node-based translation-server (used by Wikipedia and others), and we can give commit access to those separately from the core code.
(Disclosure: Zotero developer)
ps- fellow iPad user: check out Liquid Text. It's the bee's knees: https://www.youtube.com/watch?v=akEMuL4_9sk
https://daily.jstor.org/how-to-use-zotero-and-scrivener-for-...
https://github.com/mtekman/ZoteroGoogleDrive-PDFLinker-Cloud
It's lightweight enough to not get in my way but still syncs everywhere and is searchable. There's even a third party app I can use on my phone if I need quick access to PDFs.
It's basically my personal library
I believe there is a browser extension too, but the main program is (was) a non-electron stand-alone deal.
It's most direct competition, Mendeley, has always been closed-source, and then got acquired by Elsevier.
To make it clear: I love that Zotero is open-source. And I am happy it is growing (I remember the early versions, well over a decade ago, when I was writing my Master's thesis). I am just curious if they are based mostly on the storage payments, grants, or voluntary work.
The free storage is pretty generous but I exceeded it after about 6 months of use.
My setup: writing in Emacs with markdown, export with pandoc, which grabs Zotero's .bib file made with the Better BibTex plugin. Works really smoothly, though takes a little tinkering to get it set up on a fresh rig.
I'd highly recommend it!
Synced bibtex and pdfs folder in linux and windows with dropbox, worked like a charm.The best part was configuring it based on this blog post (http://griechenzicken.blogspot.com/2011/10/configuring-jabre...) and making pdf links work on both machines. I wrote a blog post about why Jabref, but unfortunately it is in portuguese (https://gtpedrosa.github.io/blog/gerenciador-de-refer%C3%AAn...).
It's up there with calibre in terms of "meh" UI but just excellent functionality.
I'm using my own nextcloud storage to get more than the 300Mb free storage on zotero cloud.
* free software
* Linux and multi-platform support
* browser extension "that just works" for ingesting items and magic lookup tool for DOIs and arXiv IDs (and I hardly ever have problems with the metadata)
* shared group libraries for collaboration with students
* offline only as well as sync
* the ability to add notes, tags, and relational links between items.
After reading about Luhmann's Zettelkasten[1] system, I've also had a great productivity boost by implementing a similar scheme in Zotero. After reading an article, I write up a summary of my ideas and thoughts and attach it to the article as a literature note. I then keep a primary repository of notes in a flat folder with links between them and the literature notes as my makeshift Zettelkasten. While not as stream-lined as some special purpose note taking tools, Zotero can do a pretty decent job at this while also having all the advantages of it's bibliographic system, file syncing, etc.
Something I wish it had for this purpose was an "auto-complete" for other entries and a graphical tree viewer of relations. However, these aren't so bad not to have, part of the genius of Luhmann's original paper card notes system seems to have been the critical thinking required to determine which handful (1 to 3) of notes are most related and the serendipitous discovery process from having to manually walk the note files when you need to find something.
As Luhmann did, I'm trying to more frequently write summaries of articles—both actual academic articles and things like blog posts, news articles, even recipes. I prefer to handwrite these notes.
For web links, I was thinking of using pinboard's caching feature which assigns a url like https://pinboard.in/cached/01234567890a/ and recording down the 48-bit identifier.
Alas, what happens when my online service of choice fails? So, maybe the Zotero citation key?
I'm wondering what others' experiences are with hybrid written/digital research workflows, and cross-referencing. Anyone have a "personal DOI" that works really well?
I use it for exactly that, especially since he's quite upfront (and funny) about how hooped everyone could be if he's ever hit by a bus.
I had never heard of it, but I found this article about it: https://medium.com/emvi/luhmanns-zettelkasten-a-productivity...
This post is quite good aswell: https://www.lesswrong.com/posts/NfdHG6oHBJ8Qxc26s/the-zettel...
I'm currently contemplating how to use it as functionally as possible (categorise articles by subject? By hypothesis? Both?), so would love to chat about it if you'd like
What I do is an interaction between the following:
Article <-> Tags <-> Hypotheses <-> Synthesis <-> Prediction <-> Research
So I might read an article, then tag the article/figures with the relevant tags. If it's for a specific project, I keep a list of tags on a separate page. This ensure that I use the tags consistently, so that I can find all the relevant articles later.
I purposefully keep the tags general, like "psychiatric conditions" rather than "depression, mania, schizophrenia" etc. When I reach the hypotheses page, this means that I can form relatively general hypotheses, and then nest them into gradually more specific hypothesis that are specifically supported by the articles.
Then, I use those hypotheses to form new predictions. These predictions turn into new research questions, which I either explore via the literature or doing my own experiments.
An example of this might be: "Wood 2018" <-> #Procedural Memory #Habits <-> Habits are part of procedural memory [[H]] <-> Habits are part of procedural memory [[H]] & Procedural memory is stored in the basal ganglia [[H]] <-> Damage to the basal ganglia disrupts habits [[H]] <-> Seger 2011
This affords me a lot of flexibility in total processing time; I don't spend a lot of time forming hypotheses that aren't interesting/necessary for me, I can quickly tag a lot of sources, and I can quickly collate those sources on a specific topic when necessary.
You might say that the first half of the workflow is "induction", the second half "deduction and confirmation".
I would love it if you had any questions or comments, but just writing this up has been useful for me as well.
Could you give more details about this ? Do you sync the notes to a folder ? How do you do this ? And how to you browse your notes ?
With some plugins (I don't remember exactly) I had a very nice pipeline of "find paper on the interntet" -> Zotero -> automatically updated .bib -> trigger rebuild of Latex document to PDF -> automatic reload in PDF viewer.
The UI is somewhat dated but the functionality is great. Nowadays I would probably choose Citationsy, maybe only because I find the UI more aesthetically pleasing.
1. Come across paper pdf on the web.
2. Use Zotero firefox plugin to import it into Zotero. Zotero is able get citation data, and automatically exports it to a bib file.
3. Use emacs helm, which reads the bib file, to cite papers in my documents.
I would have really loved to have this workflow during my Phd, but I was doing everything manually back then. My only complaint is this recent silent change in Zotero, where the exported bib file has entries in alphabetical order, rather than in last-added order. With the last-added order, when I popped open emacs helm, the last added paper would be on top. Now I have to search for it.
With all due respect to the excellent work you do (many thanks for that), my workflow does not require any manual work most of the time.
By default, there's a persistent connection whenever Zotero is running, a request when you visit a site with a translator (eg NYTimes) the first time since a browser start, when you download a PDF, etc.
I enjoyed using it, but their approach to privacy felt creepy. That [1] is somewhat improved... but only somewhat.
[1] https://www.zotero.org/support/privacy#disabling_automatic_r...
Furthermore I think http://www.docear.org/ deserves to be mentioned as another free open-source solution with some interesting mind-mapping functionality: http://www.docear.org/software/screenshots/
Here's an example from a talk I gave on Variational Autoencoders: https://www.zotero.org/groups/2350257/jszym_presentations/co...
I mostly just used it for keeping lists of books with tags (to read, which library has them, on my kindle etc) and some notes. The bibliography report is nice. I know its is targeted toward research, but it really is good for people who just like to read books if for nothing else to remember what they have read!
I think the really nice thing about it is how easy it is to add books. When I was using a kindle a lot, just add it from amazon. Recently I've been reading a lot more books from archive.org, and again its really easy to add it, just one click.
I did a few updates recently, but I welcome any additions/PRs anyone wants to send.
One of the other things going for Mendeley is seamless sync between an iPad and a desktop. Their cloud limit is 2GB free storage.
Though Zotero has Zotfile, it’s a hassle to set up tablet sync with an iOS device.
Zotero's built-in BibTeX export is mostly intended as a generic data transfer format to other similar tools. It could surely be improved, and we'd be happy for patches, but contributions tend to go to Better BibTeX, since that's what people who care about BibTeX are using. (Zotero will export ASCII BibTeX if you choose "Western" instead of "Unicode" for the export charset, though.)
(Disclosure: Zotero developer)
> 9500 free CSL citation styles.
Should I ask how it has come to be that there are 9500 citation styles? Or will it make me angry and depressed at the lack of cooperation and widely accepted standards?
Still, I've no regrets in choosing it in 2013, and think it's a fantastic piece of software. Hopefully it keeps on developing, and becomes the de facto standard :-)
Also, Zotero just revamped the Web Library interface. Check it out, it's very nice.
Since I would say 80% of people have some level of cloud storage now (University, employer, personal), there should be a simple setting to pick one, and the location. Make it easy for people and win users!
for me the killer feature is that it makes all the PDFs full text searchable so it is like my own personal google where the links never break and the content is all relevant.
Highly recommend!
I have been quick to recommend Qiqqa whenever it seems like it might be a useful tool but I never really saw those comments get much traction - maybe because everyone was already using zotero?
A little bit from 2008: https://news.ycombinator.com/item?id=319975
Related from last year: https://news.ycombinator.com/item?id=18977461
(Links are for the curious. Reposts are fine after a year: https://news.ycombinator.com/newsfaq.html)
Best feature is the web browser add-on. Can open up a few dozen articles during a literature search and dump all of them into Zotero for reading later.
When I saw Zotero at the top of HN, I suddenly wondered if they published a new tool. So I'm kind disappointed...but still love any attention they can get.
[0] https://dataverse.org/ [1] https://dataverse.org/software-features [2] https://github.com/IQSS/dataverse
Everything I want from a reference manager can be done in a relatively simple command line tool with an interface similar to beets. Only the extraction of DOIs from PDFs looks like it will have to be a bit hacky, but if `ref im article.pdf` works for 90% of articles and asks me to provide the DOI for the other 10%, that’s good enough for me.
[1] http://mackerron.com/zot2bib/
[2] https://groups.google.com/forum/#!topic/zotero-dev/a1IPUJ2m_...
1: Tested with English, Dutch, German, and Japanese novels.
I like Zotero precisely because of its UI. It's efficient, and reminds me of the Firefox bookmarks manager.
In fact, I wish I could replace or merge Firefox's bookmark manager with Zotero, so I'd get the best of both worlds.
If I get good enough with Lisp, I want to write an importer that takes all the tags from my bookmarks sqlite db, write them into org-mode files with each tagged bookmark listed in the relevant file.
> We don’t have to promise to keep your data safe — because we don’t collect it in the first place... Citationsy lives in the cloud and is accessible from anywhere... Your data is saved in the cloud and backed up every 10 minutes
> Nothing to install, update, or patch... Use our iPhone and Android apps to cite books on the go with our barcode scanner and add the Chrome or Firefox extensions to cite websites in 2 clicks.
Perhaps I missed the use case where you retain your data, or there's a version you can self host?
To be clear, Citationsy has no tracking and collects as little personal data as possible.
When you cite something we keep your citation data on our servers, of course. You can download all your citation data at any time in various open formats (BibTeX, CSL-JSON, etc).
Criticizing Zotero for privacy, of all things, is a bit bizarre. Zotero is an open-source project from a nonprofit organization with no financial interest in people's research data. It's designed as a local tool specifically to give people complete control over their data, and it's developed in the open. Most similar tools are proprietary programs owned by major publishers or analytics companies with voracious appetites for data.
The page you linked to explains the reasons for every single network connection that Zotero makes and how to disable it. Every one enables a specific Zotero feature — push-based auto-sync, fast translator updates as sites change to minimize save failures, open-access PDF retrieval. When we implemented retraction notifications, we even did it using k-anonymity to avoid sending up library data from people who don't use syncing.
We're always happy to discuss design decisions in our forums, but I'd argue pretty strongly that privacy is one of the main reasons one should use Zotero, not the other way around.
> designed as a local tool
Nod, a local tool. I have various expectations of my local tools. And if I, say, start Zotero in the morning to read a paper, then exit it for a meeting, then return to it afterward, and then exit for lunch, then at least my own expectations for a local tool are, for example, in tension with those four centralized timestamps. As are the varying tcp routes as I move my laptop among buildings. As is the request when I surf to the NYTimes during lunch.
So what does privacy best practice look like? One comment here suggests the ability to fork and edit the code. Another notes the linked documentation, and being more ethical than Elsevier. The linked page notes the existence of scattered opt-out options. And also "You can avoid these requests by keeping Zotero open while you browse the web."
My own understanding of privacy best practices, includes data exposure being opt-in rather than opt-out, and those privacy preferences being easily seen and changed in one place. My impression is Zotero doesn't do these.
And that's just Microsoft-style privacy practice. It would be even nicer to have knobs, like "check for updates every <start/day/week/...>".
> Criticizing Zotero for privacy, of all things, is a bit bizarre.
I'd be fine with "we have limited resources; know privacy is important; are improving; know we have work to do to implement best practices, are working towards it".
But my own fuzzy long-term impression has been, that such recognition has not been proportional to the potential degree of privacy exposure.
And Zotero can run completely offline, without an account.
Now compare that to Zotero's biggest competitor: https://www.mendeley.com/terms/privacy
Here's a video explaining the new functionality:
https://www.youtube.com/watch?v=6M6jNlairGc
Basically all the documents you read can have tags. So you can manage all your documents via whatever tag you want.
You can then read those documents in Polar directly and highlight parts of text that are interesting.
These highlights, notes, comments, and flashcards that you create can also have tags.
We call these annotations. We then have an annotation manager which you can manage by tag so you can pivot everything around the tags you're working with.
This version is pushed to the web version of Polar now and the new desktop version will make it out this weekend.
We're also working on a new Polar 2.0 which will support Android and tablets and have better pen support too so you can work directly in a tablet rather than a desktop/laptop.
We're also working on a dark mode but first need to get 2.0 out the door.
I'll be giving polarized a look this weekend and compare with zotero!
I love the idea of Polar for document management, and Roam for knowledge management, so I'd love to find a way to use both
Also, while I can see tags being useful, I don't think it'll really be Zettelkasten-like until you can link from one annotation to another.
[0] https://www.lesswrong.com/posts/NfdHG6oHBJ8Qxc26s/the-zettel...
I know Zotero has a lot of other features, but 99% of my workflow with it is what I described above.
It's definitely one of those topics where you could spend a huge amount of energy organizing but not much actually doing :)
Regarding roamresearch.com: How do you find yourself writing/organizing pages? Daily journal with a log of what you did first, then linking out from there?
(don't feel pressured to write a novel in response, I don't want to steal time from your day!)
Once your second brain is searchable, you can quickly find those keywords and the context thanks to Roam bidirectional links. This is one way to approach your notes, when you know what you're looking for.
Another way to approach your notes is by browsing. You can create a page with the structure or outline that you come up with, and then fill them with links to blocks in other notes. In here, you can practice a step of progressive summarization and rephrase your notes.
I'll usually just start putting things into Daily Notes, but if a connected set of notes gets too long, I'll make a separate page and add a [[]] link in the Daily Note.
The journal part of Roam, the page-per-day part, is very useful. You write down what you're doing each day, use tags and links to build up a second brain, so to speak.
Let's say, today I'm working on kubernetes. I add a tag for k8s, which is a link to the k8s page, where I have all sorts of interesting links and notes dumped. When I'm on the k8s page, I also get a list of pages that link to the k8s page, which helps me to remember some other subject that might be relevant.
It uses the Zettelkasten Method: https://zettelkasten.de/posts/overview/ and you can also get a lot of value from reading "How to Take Smart Notes" by Sönke Ahrens.
The value also comes from it mirroring your thoughts and experiences, as I said, the second brain thing.
(I'm terrible at explaining stuff, sorry :( I hope I didn't bore you to bits)
It's a wiki like system that helps you to create a second brain of sorts. You write down what you do, and link between lots of different pages, such as "aws" to "ec2" to "vm" to "vmware" to "vmotion". Roam (and org-roam, an Emacs module that I use) shows you what links back to a page, which is incredibly helpful for remembering knowledge and how things fit together.
Recently they even added a new dark pattern so that when you click the "download PDF" button on their articles it opens up a web app reader loaded with tracking instead of just giving you a PDF. You then have to spend time and click through about three menus to really download the PDF.
It's to the point that I avoid articles published by Elsevier if possible. Easier in my field than others I'm sure.
Too bad they sold out to the dark side. After they were acquired, I switched to Zotero, but I wasn't doing active research anymore by then, so I don't have as much experience of using Zotero. On the surface it looks pretty good though.
At first I preferred the way that Mendeley did this with its built in pdf reader, but now I'm happy Zotero delegates to the pdf software of your choice.
lastly -- how good is their mobile app?
You can shoot me an email on "mr [dot] sh4rpeye" at gmail.
Also, going to work on the ability to link them together with a search and auto-complete system so that you can just start typing tags, or the body of the note, and then they can be linked.
I think consistency across different sections is a reasonable expectation.
(Disclosure: Zotero developer)
The biggest challenge is deletions though so I'm still trying to work out the ideal sync framework.
We're adding support for DOI lookup and APIs like Arxiv so you could just add a DOI to polar and it will fetch the PDF and keep the metadata.
Will also support export to bibtex too.
Our big focus right now is shipping 2.0 so that we have a more modern platform that can scale us moving forward.
>Firefox maintains an active connection to a push service in order to receive push messages as long as it is open. The connection ends when Firefox is closed.
I see what you're getting at, but I think the harshest criticism should be reserved for the worst services: those that actually hold your data hostage, don't provide an export functionality and use your data in all sorts of unethical ways.
Organisations that actually honestly do value privacy and try to make an effort to get it "right" should be given the benefit of the doubt and constructive criticism, as they might actually listen. In many cases the feature may simply be driven by convenience and the competition (e.g. cloud storage, accounts, sync), and having a toggle for those is the best you can do if they want to stay relevant. In other cases the privacy issue may have simply been overlooked and the feature is improved (IIRC Mozilla has had a few of these).
Maybe a big red "offline-only" toggle would be great, but the absence of that button does not in my eyes disqualify Zotero from being a great offline solution.
Thinking about dstillman's reply, I was thought-crawling towards a "local/remote vs online/offline" distinction. So Zotero would be local but always online (if net is available). Versus my expectation that when using only local resources, a local tool will be offline.
I think your expectation is reasonable (a local tool will be offline when using only local resources), and it's definitely possible with Zotero (disable the automatic translator/style updates) but just not the default setting.
On a technical level, I don't think there is a huge difference between polling and maintaining a persistent connection, if the polling interval is short or the keepalives are long. The real question that I'd find interesting is why the translator/style updates must be "instant". For my use, once a day or once every few hours would probably be more than sufficient.
> My own understanding of privacy best practices, includes data exposure being opt-in rather than opt-out
Surely you don't expect software to default to not receiving updates automatically? As the linked section says, if you disable translator/style updates and don't use auto-sync, there won't be a persistent connection. But if a high-profile site breaks and we roll out a fix, the longer the delay the more people will just get an error trying to save.
> those privacy preferences being easily seen and changed in one place
We document every single network request that Zotero makes. Expecting them to all be configurable in one place in the software just isn't reasonable. Normal users think of features, not HTTP requests, and auto-sync doesn't have anything to do with translator update checks.
> I'd be fine with "we have limited resources; know privacy is important; are improving; know we have work to do to implement best practices, are working towards it".
OK, but I'm not saying that. I'm saying we consider privacy in all our decisions and believe we've made the right calls (and, for what it's worth, I can't recall a single complaint about our approach to privacy in many years). If you disagree with a specific decision, that's fine — come to the forums and we can discuss. But let's be clear about the features that would break for users as a result.
> configurable in one place in the software just isn't reasonable [...] auto-sync doesn't have anything to do with translator update checks
Microsoft has in one place (something very vaguely like) toggles to control the uploading of web history, hand writing, voice commands, and more. Different features of different apps. With explanations of the functionality lost if the user doesn't opt-in to each. One place, for privacy preferences.
The Zotero privacy documentation page similarly gathers in one place, recipes for opting-out of network-based features, with descriptions of use.
Software preferences having a privacy section is a thing. Firefox, chromium, etc.
I'm unclear on why it isn't reasonable for Zotero software to have similar.
> we consider privacy in all our decisions and believe we've made the right calls [...] If you disagree with a specific decision, that's fine — come to the forums and we can discuss
I suggest there's currently a shift in privacy best practices, from one-size-fits-all "make the right calls", to having user preferences for privacy.
So that's the sort-of clear disagreement.
But part of it may be a deeper difference in perspectives... perhaps call it network minimalism.
When using Zotero, I'd spend more time grovelling over previously collected papers, than collecting new ones. A task that could be done, without loss of functionality, with the net disconnected. My expectation then is, that this local tool, working with local data, will not then start using the net merely because it becomes available. Or rather, that I can easily dissuade such behavior.
Now perhaps that expectation is becoming "old fashioned", as we switch from desktop, to phone apps with only lightly bridled communication lives of their own.
Which might be an underlying issue. Zotero might be thought of as a phone app which just happens to run on desktop-local data. Or it might be thought of as a traditionally desktop application. Design decisions appropriate to the former, might feel a bit odd in the latter. "Local tool" might mean different things.
> I can't recall a single complaint about our approach
In this thread, there was someone suggesting my short paraphrasing of the linked docs was getting it totally wrong. I'm not sure how widely your users are even aware of the approach. It seems users generally aren't. Which, tying things back around, is one of the motivations for having clearly explained privacy preference options.
Thanks for an interesting conversation. Just in case you haven't seen it, the subthread with jmiserez might also be of interest.
Yes, and Firefox's Privacy & Security section doesn't cover Firefox Sync, the default search engine, search bar suggestions, the new tab pane, the default homepage, or app update checks. Those all make network requests to various services, and they're all controlled in their own sections in the preferences where they make more sense. And you can't turn off loading a website when you enter a URL.
Grouping a few more prefs together in Zotero might make sense, but in a modern, web-connected tool, there's just a lot of functionality where the network connectivity is implicit. The main difference in Zotero is that we document it all and tell you how to turn it off.
Before Zotero had push-based auto-sync, translator/style updates were indeed once a day, but that meant that, if a high-profile site changed and we pushed out an updated translator, we'd continue to get reports of the site being broken for 24 hours. We could say to update translators manually, but that would only help the people who made it to the forums.
When we added WebSocket support for syncing, we decided to send translator/style update notifications over the same connection. For anyone using auto-sync anyway (many/most users), there's no difference. If you don't use syncing/auto-sync, it's more debatable, but it's a choice between trying not to expose IP addresses that are likely already making at least some other anonymous requests (app updates, retraction checks, OA PDF checks) and decreasing the amount of breakage that users encounter after we've already fixed something.