Show HN: WAL Implementation in Golang(github.com) I wrote this simple WAL library in Golang that I use to write data that my kafka producer fails due to errors like Broker going down or some other issue. Took inspiration from etcd/wal |
Show HN: WAL Implementation in Golang(github.com) I wrote this simple WAL library in Golang that I use to write data that my kafka producer fails due to errors like Broker going down or some other issue. Took inspiration from etcd/wal |
1. The rebuf.Init function panics. I almost never want a library to call panic, and when it does, I want the library function to denote that. The convention I’ve seen most often is to start the function name with Must, so MustInit instead of Init. In this case though, I think it’d be safe to be a little more lenient in what you accept as input and trim the trailing slash.
2. I never (not almost, actually never) want library code to call any of the fmt.Print functions unless the library is explicitly for writing output, or that behavior is strictly opt in. If the library really must print things, it should take a user supplied os.Writer and write to that. Let the user control what gets printed or not.
2. Yes, Will integrate a logging library instead of fmt (https://github.com/uber-go/zap)
Ideally you want to add an option via a function or interface:
func New(optOne bool, log func(string, ...any)) {
if log == nil {
log = func(m string, a ...any) { fmt.Printf(m, a...) }
}
log("starting; optOne=%v", optOne)
}
1. syscall.Iovec allows you to build up multiple batches semi independently and then write them all in a single syscall and sync the file with the next one. It is a good basis for allowing multiple pending writes to proceed in independent go routines and have another one have all the responsibility for flushing data.
2. It is better to use larger preallocated files than a bunch of smaller ones, along with batching, fixed size headers and padding write blocks to a known size. 16 megabytes per wal and a 128 byte padding worked well for me.
3. Batching writes until they reach a max buffer size and/or a max buffer age can also massively increase throughput. 1 megabyte max pending write or 50 ms time passed worked pretty well for me for batching and throughput to start with, then dynamically tuning the last bound to the rolling average of the time the last 16 write+sync operations (and a hard upper bound to deal with 99th percentile latency badness) worked better. Bounded channels and a little clever math makes parallelizing all of this pretty seamless.
4. Mmap'ing the wals makes consistency checking and byte level fiddling much easier on replay. No need to seek or use a buffered reader, just use slice math and copy() or append() to pull out what you need.
etcd/wal actually does do preallocations (https://github.com/etcd-io/etcd/blob/24e05998c68f481af2bd567...)
Yet to implement max buffer age! Any references for this would be bomb!
Is mmap() really needed here? Came across a similar project that does this? Really gotta dig deep here! https://github.com/jhunters/bigqueue
1. syscall.Iovec is a struct that the writev() systemcall uses. You build it up something like this:
func b2iov(bs [][]byte) []syscall.Iovec {
res := []syscall.Iovec{}
for i := range bs {
res = append(res, syscall.Iovec{Base: &bs[i][0], Len: uint64(len(bs[i])}
}
return res
}
func write(fi *os.File, iov []syscall.Iovec, at int64) (written int64, err error) {
if _, err = fi.Seek(at, io.SeekStart); err != nil {
return
}
wr, _, errno := syscall.Syscall(syscall.SYS_WRITEV, fi.Fd(), uintptr(unsafe.Pointer(&iov[0])), uintptr(len(iov)))
if errno != 0 {
err = errno
return
}
written = int64(wr)
err = fi.Sync()
return
}
2. Yeah, I just zero-filled a new file using the fallocate as well.
3. I handled max buffer age by feeding writes to the WAL using a channel, then the main reader loop for that channel select on both the main channel and a time.Timer.C channel. Get clever with the Reset() method on that timer and you can implement whatever timeout scheme you like.
4. No, it is not needed, but my WAL implementation boiled down to a bunch of byte buffers protected by a rolling CRC64, and for me just mmap'ing the whole file into a big slice and sanity-checking the rolling crcs along with other metadata was easier and faster that way.
I also think your rotation will delete the wrong segment when you have more than ten segments - imagine you're writing rebuf-1 to rebuf-10 - what's the "oldest file" to delete now? Besides, should you really delete those files?
1. Use fsync for durable writes in case of system crashes
2. Fix log-rotation-purging logic
3. Fix `file already closed` bug on consecutive writes
4. Add CRC checksum
Perhaps it's just me, but I don't trust code that hasn't been tested.
Since you mention etcd/wal:
https://github.com/etcd-io/etcd/blob/v3.3.27/wal/wal.go#L671
https://github.com/etcd-io/etcd/blob/v3.3.27/pkg/fileutil/sy...
Thanks for your comment, I'll definitely check it out. It was my first attempt at this. How can I make it better?
Similar to RocksDB.