How to parse EVE Online chat logs (in Common Lisp)

How to parse EVE Online chat logs (in Common Lisp)(blog.michaeldresser.io)

90 points by michaelmdresser 4 years ago | 23 comments

sen 4 years ago |

I loved my time playing EVE, spent a fair few years running a major (couple hundred players) Corp in one of the big alliances back pre-pearlabyss. We had a whole suite of web apps that integrated with EVE auth and API, and managed various aspects of the game for us. Threat Intel by putting bots into enemy chats then scraping for keywords (mentions of certain system names), tools to monitor the markets for certain prices/shortages we cared about, custom maps that let players submit scan data, etc etc.

I’ve never experienced any other game that gives you the rush of a major alliance battle in EVE.

It’s an incredible game with a lot of insanely smart people playing and some great communities around it, but regretfully I had to “win” as life is life.

xwolfi 4 years ago | |

Same but I have to admit it was akin to addiction. Had to stop or I d just do EvE stuff and nothing else :D

sarsway 4 years ago | | |

That's the problem with EvE, a lot of people famously use spreadsheets, and once you start coding your custom tools and get into automating stuff it just never ends. Very soon you're scraping all the markets storing everything in a database looking for profits. Using multiple accounts to scale up your "ISK operation". There's really no limit on what you can do, expect that there's probably a better use of your time :)

ben0x539 4 years ago |

Oh, hey, I had one of these when I was afk-ratting in an Ishtar in Guristas space, those were the days: https://github.com/ben0x539/eve-log-alert

I'd drop my Ishtar[s] in an anomaly[s], then tab out and probably read hacker news or something, and completely forget about it, usually leading to either dying as I didn't check intel channels or even look for hostiles in the local system, or wasting time as I didn't warp to the next anomaly after clearing the current one. With a simple ruby script, a janky sql query and a pile of ugly heuristics, most of that stopped being a problem, as a very loud noise would remind me to warp away! It hits a "sweet spot" of being totally non-reusable but doing exactly what I need it to while remaining just editable enough to hack in more special cases/heuristics...

It wasn't a super time-efficient way to get space-rich, but at least somewhat effort-efficient, and it netted me enough that I could probably have bought a titan or three. Of course I eventually lost most of my net worth by having it invested in shitty minerals and being unsubscribed as the floor fell out under that market.

Those were the days.

michaelmdresser 4 years ago | |

I had the exact same motivation as you! I've since moved onto other, much less passive, activities but for a brief moment in time my little tool gave me a lot of peace of mind while fully tabbed out of EVE. Optimizing my game experience for minimal effort/attention and maximum profit has historically taken the enjoyment out of games for me, so I'm glad to have moved on.

amerine 4 years ago |

I love love love posts talking about using our skills to make our entertainment ours. Kudos for sharing and much love for sharing how you are making EVE just a little more fun.

michaelmdresser 4 years ago | |

Thanks for the kind words. One of the things I have always loved about EVE is how open it is (relative to other MMOs) with information like this as well as providing APIs [1] and game data dumps [2]. The data isn't always perfect, and the APIs don't always cover everything the playerbase wants, but I'm just happy any of it exists in the first place! I used the data dump in conjunction with this log parsing to get an automatic "distance from my current location" of enemy players. It isn't anything too special—other developers have made much better tools—but it has been a very fun project.

[1] https://wiki.eveuniversity.org/EVE_API_Guide

[2] https://wiki.eveuniversity.org/Static_Data_Export

grimgrin 4 years ago |

I like to enhance games with log parsing too. Been playing muds again lately and writing logs with the mud/telnet client (tintin++) to be parsed/spat out in tmux splits, which works great

(as an aside, I started looking and testing mushclient, which is adored by many. nick gammon has been upkeeping it for years and it offers a ton of features/settings)

laserbeam 4 years ago |

I used to have an eve chat log parsing tool for getting all the kills in a roam. I'd just ask everyone to x-up once, would then grab their names and timestamps from the chat logs, and then ask zkill what happened to those players during the fleet chat. This was perfect for the kind of chaotic, but small roams we did with Eve Uni, and for the fact that we used to write after action reports after most fleets.

It would sort those kills, and add separators between fights based on time.

It's one of the simplest things I've ever implemented, and I think Eve Uni still used it for years after I stopped playing. Eve chat logs are beautiful.

cschmittiey 4 years ago | |

Which campus was that at? I feel like I saw something like that used once, but other than that didn’t see it much when I spent my time with E-UNI. I mostly kept to WHC, though.

laserbeam 4 years ago | | |

Mostly low sec and null sec campuses used it. The tool doesn't make sense for whc, it's not a type of space where you roam, having a bunch of small fights in a row.

zeristor 4 years ago |

Before this used to be Play By Email games one from the 80s being Empire.

Sean Eddy, he’s probably on HNews, wrote some software to process the emails to get a lead on other teams.

As I remember the story this keen interest helped him hone the skills for genetics analysis, but he could just have been a natural.

https://www.the-scientist.com/uncategorized/a-mind-apart-450...

or:

https://archive.md/3IbBK

I’m reminded of Richard Feynman honing is mathematical skills at school competitions.

Suffice to say these skills with the impetus behind them are hugely useful.

azalemeth 4 years ago |

I played EVE for about 4 years around the late 2000s and early 2010s. It's the only game I've ever played where (a) physical units were respected in calculations (e.g. tracking speeds in radians per second; distances in Pc -> Ly -> AU -> km -> m) and (b) other players contained a nonzero number of linux kernel developers...

Yhippa 4 years ago |

In general, what is a good language to parse game logs?

stevekemp 4 years ago | |

Most languages can do most things, so you should pick the one(s) you know best.

That said traditionally perl would be the choice for parsing, and it is still a good candidate. For simple extraction - think "programatic grep" - then awk is also a good choice.

duped 4 years ago | |

I've found for down and dirty parsing, you can almost always parse simple log/structured output using operations like `split`, `substring`, `prefix`, and `suffix` where :

- `split` splits a string at an index or into substrings separated by a pattern - `substring` takes a substring of a string for a range of indices - `prefix` returns true if the start of the string matches a pattern, false otherwise - `suffix` returns true if the end of the string matches a pattern, false otherwise

Python and C++ both have good standard library functions for these tasks. What's less important than a language for your parser is a language to plug the output into to make something useful.

tomjen3 4 years ago | |

Unless the game logs are in json or xml, the easiest language to get started in (assuming a few regexs isn't enough) is whatever language you know, that has a good packrat parser library.

bjoli 4 years ago | |

Anything with parser combinators or PEG support. I had to learn PEG parsing for a project when I was a kid, and that has been one of my most useful skills.

Yhippa 4 years ago | | |

Pardon, what is PEG?

dunefox 4 years ago | |

I'm using F# and FParsec atm, works great.

2OEH8eoCRo0 4 years ago | |

Perl

jhgb 4 years ago | | |

Also known as Practical EVE Re-Engineering Language?