How to write a filesystem in 50 lines of code

How to write a filesystem in 50 lines of code(blog.ksplice.com)

129 points by ebroder 15 years ago | 28 comments

DrJokepu 15 years ago |

While this is a very interesting article, I think it would be great if people stopped using "How to do x in y lines of code" titles - it's just as bad as "x things about y" titles. It's a lot better when title actually describes what the article is really about.

It would be awesome if this was added to the FAQ or something.

TallGuyShort 15 years ago | |

Although I think they're overdone, I think "x in y lines" is avery relevant to the hacker community, while "x things about y" isn't always. Doing "x in y lines" generally implies an unusually elegant solution, that could be of great use.

silentbicycle 15 years ago | | |

It's such an easy metric to game, though ("The first line is 'import library_that_does_everything'") - It's a proxy for succinctness, at best.

That said, "[something non-trivial] in only 250-500 LOC" is often more interesting than "...in only 5 LOC", because generally anything that small is just gluing together libraries. There are exceptions, of course - I've seen impressive stuff in just a few lines of J/K/APL.

kirubakaran 15 years ago | |

"How to do x in y lines of code", where x is a perceived to be something very difficult and y is a small number, encourages people to try things that they would be otherwise be too intimidated to try.

tcdent 15 years ago | |

I'd usually agree with you. But, since this is about building filesystems, something I thought was reserved for kernel hackers and truckload of C, I think the LOC title is appropriate. I would have still clicked, but would have expected something way over my head.

lisper 15 years ago |

Youch! Beware, there are hidden pitfalls here.

I tried the fuse-python xmp example and accidentally mounted the filesystem on top of an existing folder with the result being that the folder vanished. Now if I try to cd into that folder I get an input/output error. It doesn't show up on directory listings. And umount says it's not mounted. I have a backup so this is not an unmitigated disaster, but the folder is huge (it's my development folder) so I'd much prefer not to have to do a restore. This is MacFuse 2.0.3 running on Snow Leopard. Any suggestions on how to proceed would be greatly appreciated.

price 15 years ago | |

Your data is fine -- it's hidden while something else is mounted on top of the folder.

Rebooting will fix it. But you should be able to just unmount the filesystem, which will also fix it. Does 'mount' list something mounted there? Does 'df /path/to/folder/' reflect the FUSE filesystem rather than the underlying filesystem? The input/output error sounds like the FUSE filesystem being mounted indeed, and just being broken.

lisper 15 years ago | | |

> Your data is fine -- it's hidden while something else is mounted on top of the folder.

I thought that was the case. But the truth is I don't really understand how mount points work under the hood. Where is the mount point information actually stored? I'm guessing it's in kernel memory somewhere, but I don't really know, and Googling hasn't been much help. Is there an article somewhere that explains this?

> Rebooting will fix it.

Good to know.

> But you should be able to just unmount the filesystem

Yeah, you'd think. But everything I try to do to that directory results in "Input/output error".

> Does 'mount' list something mounted there?

Yes:

Python@fuse0 on /Users/ron/devel (fusefs, nodev, nosuid, synchronous, mounted by ron)

> Does 'df /path/to/folder/' reflect the FUSE filesystem rather than the underlying filesystem?

  [ron@mickey:~]$ df devel
  df: devel: Input/output error
  [ron@mickey:~]$ umount devel
  umount: devel: not currently mounted
  [ron@mickey:~]$ ls devel
  ls: devel: Input/output error

You can hopefully see why I'm puzzled. I would really like to understand why this is happening.

nelhage 15 years ago | |

Have you tried `fusermount -u` on the mountpoint? That's the fuse-specific command to force an unmount, which might clean up the FUSE mount even if the OS's VFS layer doesn't think there's a mountpoint there.

lisper 15 years ago | | |

I'm running MacFuse. The fusermount command is specific to Linux.

billnapier 15 years ago | |

have you tried rebooting?

lisper 15 years ago | | |

Not yet. I was hoping to find a less drastic (and more educational) fix. I'm also a little worried that rebooting might make the situation worse since I don't really understand how FUSE works under the the hood.

snprbob86 15 years ago |

I know of a few professional console games that where successfully debugged by embedded telnet servers: it is incredibly useful to be able to cd and ls around in your scene graph. I believe the ones I saw implemented their own shells and toolset, but I bet you could get pretty clever with debug builds that expose a decent set of unix tools across a remote shell.

wildmXranat 15 years ago |

Disclosure: author of Dropfuse fs http://github.com/arekzb/dropfuse . I wrote it to solve a personal need for fetching shared files by command line.

Yes, FUSE is indeed fun and poses an easy entry barrier to writing a user-space filesystem. Many projects that are based on it are listed on their wiki page as well and a lot of them have source code available.

ydant 15 years ago |

It's an interesting project. I grabbed it and wrote a quick Flickr FS. It seems that it's a little limited for really laggy webservices in that you specify the type of an entry (directory, folder) by the type of data you return, and it calls the same method for readdir, getattr, and read, with no way to differentiate in your code.

In order to specify the type of an entry, you have to return the data for that entry. So a sub-folder has to return an array. There's no way to differentiate between, say, an ls on the parent folder or on the sub folder (e.g. ls / and ls /foo both return the method mapped to /foo), so you have to query all of the subfolders' contents at once AND cache it so it doesn't have to be re-queried when the user wants to look at the sub-folder.

Hopefully I'm overlooking something, but the source is pretty straightforward. The good part is it'd be easy to modify to ask for types separately. Actually just passing in another argument indicating what mode it's in would help.

ebroder 15 years ago | |

Yeah, that definitely can be a weakness of RouteFS's style.

My target application was things like automounters, or the low-latency database querying sort of thing I mention in the actual blog post. Since I wanted to be able to have the filesystem structure change as it was accessed, I decided to make any sort of caching entirely an application-layer problem, not a RouteFS-layer problem.

I think it would be possible to extend RouteFS to handle this sort of case more gracefully. One option in particular might be to take advantage of python-fuse's stateful I/O feature (which lets you associate a Python object with open file descriptors in your filesystem [1]) so that reads from the same file don't result in the same lookup over and over again, although this certainly doesn't help for directories.

But in any case, I'd certainly love to see ideas for extending RouteFS to make it easier to make it more performant. Submissions in the form of patches are always excellent, but even suggestions for API changes would be welcome - feel free to open an issue on Github either way (http://github.com/ebroder/python-routefs/issues).

[1] See "Filehandles can be objects if you want" in http://fuse.cvs.sourceforge.net/viewvc/fuse/python/README.ne... for more information

djacobs 15 years ago |

Intriguing. I hadn't thought of this take on filesystems before.

I'm a big fan of text-based config files but am stuck on a Windows machine at work. I wonder, would it be possible to map the registry to a virtual filesystem I could access from Explorer?

snprbob86 15 years ago | |

One of the many reasons to prefer PowerShell over CMD: http://powershell.com/cs/blogs/ebook/archive/2009/03/30/chap...

djacobs 15 years ago | | |

Nice. I'm not too into DOS, but this definitely is an easier solution than creating a VFS for the job. Thanks for the pointer.

billnapier 15 years ago |

Clever. But I find the idea of RouteFS more clever than the toy FS he built...

barrkel 15 years ago | |

Applications of the same idea have been around for some time, e.g. Plan 9's file system (e.g. GUI elements are part of the FS, , bash's /dev/tcp/<host>/<port> etc., and indeed /proc's file system seen in Linux and, in a limited way, in Solaris.

Seems like no Linux app framework can be complete without reinventing its own virtual file system, with various syntaxes for paths to e.g. network shares but that are inaccessible when used on the command line, etc.

price 15 years ago | | |

The point is that a FUSE filesystem is available from the command line and anywhere else, because it is an actual filesystem. RouteFS is a way of taking any virtual filesystem-like tree that you might find useful and making it available to the entire system as a normal filesystem, just as easily as you could describe the tree in any other form.