Hi, I wrote this!

To continue a discussion:

  - How does your engineering team track new "debt" after releasing code? (if at all, and why not)
  - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful?
  - Do you feel like your company is good at managing access when hiring / firing people?

Otherwise thanks for any feedback, I enjoy writing these!

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about?

Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want?

Every time I start looking at centralized logging stuff it seems like a rabbit hole of problems we're too small to be worrying about, stuff that's not shipping features on my app.

This is the best security article I've read in a long time. If you're at a startup right now, drop most things and take a few minutes to read it carefully.

Ugh. A good and scary reminder of what's lurking around the corner for any of us at any time - including holidays and vacations (Linode's holiday attack last December comes to mind). IMHO, the emotional impact of breaches on the staff who respond to them is under-discussed. The author touches on it here:

> The discovery of a root cause is an important milestone that dictates the emotional environment an incident will take place in, and whether it becomes unhealthy or not.

> A grey cloud will hover over a team until a guiding root cause is discovered. This can make people bad to one another. I work very hard to avoid this toxicity with teams. I remember close calls when massive blame, panic, and resignations felt like they were just one tough conversation away.

Had the pleasure of working with Ryan when he was at FB--he's one of the best.

It's interesting to see press leaks highlighted here as a pattern for insider threat. I don't doubt the author that this is so for the limited scope of organizations considered (SFBA tech companies), but I've worked on several insider cases, had insight into many more, and it's almost always an employee or ex-employee, with an axe to grind, taking trade secret information to a new job at a competitor. In many instances, the competitor has no idea and is pissed when they find out.

One piece of advice that I'd give out with such cases is to listen to your Spidey Sense. A lot of organizations will say, after the fact, "well... something didn't seem right with Bob...". If you sense something isn't right, prepare to secure evidence and analyze it. Don't put IT assets back into circulation if there's doubt, and don't sit on it.

This is solid advice. To illustrate a little based on my own experiences and goals this year:

- Yes, centralized logging is the biggest thing. What you put into it matters; queriability matters; but nothing matters as much as having that centralized logging pipeline to begin with. Once you have that, you can start adding other relevant metadata, like host config states, API calls, et cetera.

- Giving employees a budget to buy the device they want is probably a better idea than BYOD. Strong password policies still matter. If it's BYOD, you probably still want to bring the device into policy. That can include physical rules (only do work work on the VPN or from the office) and software ones (you can use any device you want but it has to be running our osqueryd or whatever). Unfortunately, visibility becomes a double edged sword: there are good legal and ethical reasons for not wanting to see everything on an employee's laptop. (Overall, I think BYOD is a bad idea for most companies.)

- 2FA is pretty cool. It doesn't just solve the usual "bad/compromised password" model -- it also typically makes it a lot harder for employees to mismanage their credentials (e.g. re-use the same SSH keys and have their personal box be compromised). For some reason, having that around seems to remind developers that you can make users re-authenticate for important/unusual actions -- you don't just have to count on the ambient authority of a session cookie.

- We'd all like to imagine that we're going to be attacked by space alien 0day ninjas. Realistically, the main vector is an employee (rogue or confused deputy). Trainings are boring and don't work. Signature-based detection gets outdated pretty quick. I've done a little work on faster analysis tools -- I'm hoping we get a lot better at unobtrusively protecting people from even spearphishing in the next few years. (The tools we're building at Latacora are ready to beat a lot of attacker tactics right now, but I think we have an arms race ahead of us. Boring domain generation algorithms still aren't detected by most organization, so there's not a lot of evolutionary pressure.)

- I have no idea if we'll get better at quantifying metrics for debt and security risk. I did a little bit of research into this, and it's a wide open field. You can get decent high-level reports with a "DEFCON number", but most of these models are not sophisticated in the sense you'd expect actuarial tables to be. And that's what they should be! It's revenue-at-risk! Step one here is fortunately getting all of that data into that centralized logging pipeline, and security professionals seem to mostly agree that's what you do first, so hopefully we get better here.

Logging everything is a great idea, but only if you read the log data. Target installed a system to monitor for certain kinds of security hacks which wrote to their logs files. The logging was turned off due to a high number of warnings cluttering up the logs. Of course the logging was telling them they were being hacked which they ignored for months, leading to all sorts of business disasters.

Great article! For those interested in security debt and how it relates to startups, I wrote this in 2011: https://www.veracode.com/blog/2011/02/application-security-d... and presented it that year: https://www.youtube.com/watch?v=MKdiiXgvz_U This predates by a year the referenced security debt presentation which has much of the same material uncited.

> I wasn’t roped into a single intrusion this year at any companies with completely role driven environments where secrets were completely managed by a secret store.

> This can either mean one of a few things: These environments don’t exist at all, there aren’t many of them, or they don’t see incidents that would warrant involving IR folks like myself.

What are these secrets store? Do they exist?

Im surprised credential theft is still the lowest hanging fruit.

I thought Banks seem to have solved alot of that.

You have a lot of decent options. You could do a lot worse than ELK. If you're on AWS, you can get hosted Elasticsearch. It comes out of the box with Logstash you can hook up to DynamoDB, and it also does Kibana out of the box. There are a number of other vendors; but there are decent reasons for keeping your logs as close as possible.

CloudWatch works fine too. CloudWatch comes integrated with AWS services out of the box. It can be more annoying to get your logs into it than ELK (the latter seems overall more popular). Its alerting and the AWS CLI integration pretty slick, though.

You should also go turn on CloudTrail right now. It lets you automatically log side-effectful API calls. It is not a replacement for a centralized logging pipeline, but it's great high-signal data to put into one.

I appreciate that your complaint (totally valid!) was "this is a rabbit hole", and I just gave you two options, and that might not help your perception that it's a rabbit hole. If you find yourself paralyzed by choice, either choice is much better than deferring the choice! Just pick one. Heck, if you can't pick, let me help: pick AWS hosted Elasticsearch.

A lot of people (also in the security space) like Splunk. I find it annoying to deploy (I've heard rsyslog-in-front-of-forwarders as a canonical deployment method for just ingesting syslog more than once because reasons) and overpriced. YMMV.

Disclaimer: shameless plug! You're not the only one with your hair on fire. One of the first things we're doing for Latacora customers is setting up a centralized logging pipeline.

It depends a LOT on how many machines and services on the machines you're dealing with. There's a remarkable amount of stuff at the small end which is good, cheap, and fast to deploy.

I've been using Loggly for my personal machines (~8, mostly cloud VPSes). On the plus side, it's free at my scale, and the analysis and reporting tools are nice at least in theory. On the minus side, I can't get my logs past 7 days archived to S3 without paying $150/month, which I really want since my main use-case is longer-term analysis and forensics.

I'm planning to switch to Papertrail, which for the princely sum of $7/mo will give me a simpler UI and a year's archiving to S3.

Loggly and Papertrail both use the same deployment strategy (you hook them up to syslog and/or your app's logging package), and I had Loggly up and running and providing useful feedback in solidly under four hours.

I would use some SaaS solution. ELK can be cheaper, but can take tons of time to configure and maintain if you run it. For early startup paying under $100 / month for some logging solution is no brainer vs. spending time configuring.

Disclaimer: I work at Sumo Logic I would recommend: https://www.sumologic.com On top of grep like searches, you can do analytical searches (SQL on text data).

Here are some SaaS choices:

* Sentry: https://sentry.io/welcome/

* Logentries: https://logentries.com/

* Loggly: https://www.loggly.com/

* Opbeat: https://opbeat.com/

* Papertrail: https://papertrailapp.com/

Sentry is open source and there is even an official up-to-date docker image: https://hub.docker.com/_/sentry/

Loggly published an "Ultimate Guide to Logging": https://www.loggly.com/ultimate-guide/

Please excuse the shameless plug but since you are asking for one, Striim is a good out of the box centralized logging solution. We use Kafka as our messaging layer (you can either install on your own Kafka or use our internal) and we use Elasticsearch as the storage layer.

We also have streaming log parsers to connect your data. That whole thing about 'creating new alerts in minutes' is trivial in our platform since everything is based in SQL.

Unlike Splunk or ELK, our solution is based on in-memory streams so you don't have to wait for data to be indexed to fire off alerts on anomalous activity. Feel free to message me to find out more or simply download the product from http://www.striim.com/

TFA aside, centralized logging is super useful for debugging a variety of issues. There are a number of hosted options, and setting them up isn't too hard. It usually involves configuring you're application's log device to talk to the remote service, or configuring syslog on your app servers to forward logs to said service.

See https://logentries.com/ for an example

If you want a pretty prepackaged solution you could do a lot worse than splunk. They even offer it SaaS

https://www.splunk.com/en_us/cloud.html

I'm in the same boat. Looking for recommendations on strong, sturdy, buckets, for bailing water.

In general, secret stores "manage secrets so that you don't have to". That can mean a few things, depending on who's using the term.

Sometimes, it's as simple as a shared password store (I've used one powered by GPG, for example). This is better than YOLO password policy, but not by much: humans still see individual keys.

If you want to be really fancy, you authenticate the human and then decide what they get to do, in a centralized fashion. This is often tricky to do, because you either don't have the funds to do that if you're small, or you have too many services to interact with if you're big. (Many organizations get pretty close -- I'm told that the DoD pretty much authenticates everything with smart cards, for example.)

Sometimes, it means a more automated system where software authenticates instead of a human, and it gets e.g. a certificate. Usually this is still always the same certificate, though; so the main difference is that it's a human versus a machine authenticating.

Sometimes, it means an HSM (hardware security module). These are secure physical devices that perform cryptographic operations for you, so that the key stays on the device.

For example, Hashicorp Vault[0]

[0] - https://www.vaultproject.io/intro/index.html

There are a number of secret stores. Some more basic ones resemble password managers on steroids, with audit logs of who checked out what and when. Or you can go to a full HSM (hardware security module) that totally isolates secrets (keys) from secret users (actual users, application code etc). HSMs allow you to sign or encrypt without ever having the keys used. It's hard to accidentally leak a secret if you never had it in the first place.