- Consulting for a customer where they were deploying to new hardware with a new processor architecture, I received a report that an application was running slower on the new servers than it was on the old ones. I started out looking at things with strace and ltrace, had to move deeper and pull out perf and systemtap, but found that it looked like memory access was slower than on the old hardware. I did research on the processor, and found that it was due to the 'Intel Scalable Memory Buffers'. Since memory first had to be loaded into the buffer before the CPU could access it, things not in the buffer already had higher latency, but things already in the buffer were much more quickly accessed than they would have been previously. I worked with the developers to make up for this performance decrease in other ways. Their application was well suited for using hugepages, but they were not, and TLB pressure was causing performance bottlenecks in other areas. Switching to hugepages prevented TLB pressure, and the application ended up being even more performant on the new platform due to the increased amount of available memory allowing for a large amount of hugepage allocations.

- I was consulting for a customer that was running instances on a xen platform. They were having performance issues vs. their old bare metal deployment, and had already done some analysis. They gave me a perf report that was showing a massive amount of time being spent with a specific xen hypercall. I had to dig into the xen source code to figure out exactly what that hypercall was doing, as general public documentation about it was somewhat vague. I was able to determine that it bundled up a bunch of different operations, so it wasn't conclusive from that, but it did narrow down the possibilities. It was enough to point me in the right direction, however, and I was able to determine with a little bit of trial and error with some tweaking that it was ultimately related to decisions NUMA was making. It turned out that the customer had thought they were doing NUMA node pinning, and ultimately weren't. Interestingly enough, even with pinning, we still saw some of this, and completely disabling NUMA (all the way - not just balancing) actually ended up being needed to fully reclaim the lost performance. I also learned an important lesson in trusting customers - even the ones that know what they're doing aren't always right, and while I should trust them in general, verifying their answers is important. I discounted investigating NUMA as early on they told me they had their applications pinned to nodes, and I would have otherwise investigated that more quickly and probably solved the issue in less time.

Eventually I just gave up.

Layer 8, ie. human beings. The software side of stuff, I can eventually solve by hammering at the keyboard until it works. But the people using it, and the ever-changing requirements they have - especially since this influences my software design - is definitely the hardest part.

However it just occurred to me that maybe the hardest problem I've had was actually making up an architecture from scratch as the problem was unfolding itself, and then having to maintain it and even bring others aboard. Meaning I had to document as much as I could (even though I had very little time for this) and I also had to sometimes give more priority to a not-so-important bug (vs a very pressing issue for me), not because it was critical to any feature but because it was making it very painful and hard for a teammate to implement one which in turn would later delay some other feature.

And the major reason why there was no actual planning to avoid this as much as possible, was because features were being decided on the go by the top brass on a case by case basis, completely opposite of the original direction I was told we were going to go (which was the information I used to lay down the foundations of the project). I.e. I was told at first that this was going to be just a wrapper script and it ended up being a whole orchestrator including multi-node operations needing result consolidation, a state machine to track down the... uhmm...state of the system, and things like that.

So my point is that probably there are several axes of "hardness" in a problem that can be mixed together, and that makes it difficult to compare a problem to another (i.e. over which combination of axes are you comparing one to another?). I guess part of the response to such a question in an interview would be then to explain the context so that it can be more easily understood why was that problem perceived as hard and over which axes. Was it because the problem was an optimization one and the previous code was impossible to work with? was it because the business constrains (as I believe was my case) where surreal? was it because the teammates made it really hard to move forward (e.g. bureaucracy, defensive/aggressive coworkers, etc)?

And I know we are talking about "technical problems" but I find it increasingly hard (as my career advances) to make a distinction between what is and what is not a technical problem. If the business constrains dictate certain sub-optimal solution must be developed, and that in turn causes technical issues, was that a technical problem? if a teammate is disruptive and introduces sub-par code that later causes bugs that need to be immediately addressed now, was that a technical problem?

In my mind they probably all are to some degree just by virtue of in the end influencing whatever technical decisions are being made. So maybe that could be part of the answer? asking about what specific sense are you referring to when you (the interviewer) ask me about the hardest technical problem.

I've got two answers that I would probably consider.

#1: debugging what ended up being a hardware problem. I was working on a device with a microcontroller and it had a sleep mode where the micro would program an RTC, shut itself off and the RTC would trigger the board's wakeup circuit when its alarm fired. I'd already told the board designer of two or three hardware bugs that somehow (surprise!) turned out to be my software bugs. So this time I was a little more cautious. There was a more senior software engineer working with me, and he told me to check the schematic. I looked at the processor manual and the board schematic, and followed the traces to make sure I was doing it right. And I just couldn't find out what was wrong. So the senior sw eng said, "well, ok, if you're sure, then just probe the RTC pin with a scope." Wow. A o-scope. WTF is this gloriousness? So I got to learn a bunch about how to go from the board schematic to the board layout, how to probe, what all the stuff on the scope was about. Sure enough, the RTC alarm went off on schedule but the trace showed some funny stuff that indicated that there was a design error in the board somewhere (I didn't understand the details, but IIRC a cut-and-jump of the prototype made the bug go away).

Motto: It's never a hardware design bug. Until it is.

#2: This bug I learned a good amount from. I would see frequent misbehavior in my code where it looked like multiple subsequent sessions were being corrupted somehow, perhaps from a previous session. I was certain that I was releasing resources from the previous session and destroying all of it. I watched my code hit my `boost::shared_ptr<foo_t>::reset()` and so clearly it was now gone. Right? Well, shared_ptr<> not all it's cracked up to be. So I went back to read about conventional advice about shared_ptr<> and people would frequently suggest boost::weak_ptr<> where appropriate. I mistakenly thought about these as a dichotomy for some reason. But that was no good because I couldn't share my weak_ptr<> so it's not really useful. Except -- wait -- the vast majority of the time I'm propagating my shared_ptr to places where they don't need to share it beyond themselves. So my design would actually be better if I shared the shared_ptr as a weak_ptr anywhere other than Right Here. In doing this redesign, I realized that the weak_ptr promotes itself temporarily by effectively asking "hey is this still allocated somewhere?" Turns out that other thread using this resource would occasionally take slightly longer and wouldn't decrement its shared_ptr until after the new session had started, which would mean that the old resource was never destroyed. After the redesign in this case where the background thread loses the race it would just fail the weak_ptr<> promotion and harmlessly skip its activity.

Motto: shared_ptr<> and weak_ptr<> help preserve an ownership metaphor. Which code Owns this memory/resource and which code is just "borrowing" it?

I have solved about ten "hard" problems in my career, most of which has been in R&D. Each one of these had multiple prior failed attempts, and in some cases took me months of thinking before I could find a solution.

1. Qualcomm wanted me to devise a computer vision solution that was more than two orders of magnitude power-efficient than what they had then. There was a clear justification existing as to why such a drastic improvement was needed. Nobody had a solution in spite of trying for a long time. Most laughed it as impossible. I started by looking for a proof as to why it could not be done if it indeed could not be done. After some three months of pulling my hair, I started getting glimpses of how to do it. Some three months later, I could convince myself and a few others that it is doable. Some three months later, the local team was fully convinced. Some three months later, the upper management was convinced. You can read the rest here: https://www.technologyreview.com/s/603964/qualcomm-wants-you...

2. I wanted to solve a specific machine learning and Artificial Intelligence challenge. I would code for a day or so, and then again run into days of thinking how to proceed further. E.g., coded a specific parser algorithm for context-free grammars, including conversion to Chomsky normal forms, in 1.5 days including basic testing. However, what's next. Woke up with new ideas for about ten days in a row. Conceived Neural Turing Machines back in 2013, about a year before Google came up with their paper on the subject. (Unsurprisingly, I did not had that name in mind for it back in 2013.) I also did not get an actual opportunity to work on it, as a result of which I am still not sure if I could have actually done it.

3. Needed to make a very sensitive capacitance measurement circuit, trying to get to atto Farad scale floating capacitance even with pF scale parasitic capacitance to ground. The noise and power requirements were very challenging. After about three months of seeking inputs from the team lead without hearing a solution, I ended up coming up with a solution. I later discovered that the technique was already known in RF circles, though only a few were aware of it. Capacitance measurement circuits with such sensitivity did not show up in the market for several years. (My effort was target at using inside a bigger system.)

4. I was working on measuring bistable MEMS devices. The static response of these was well understood. However, so far, the dynamic response was only measured by the team; there was no theoretical explanation behind it. We invited several professors working in the field to give seminars to us, and asked questions for this, but never heard back a good answer. A physicist colleague found an IEEE paper giving the non-linear differential equations behind it, which worked, but yet provided no insights into the device behavior, and took time to solve numerically. I wanted a good enough analytical solution. I kept on trying whenever I had time-opportunity, while the physicist colleague kept on telling me to give up. Six months later. I woke up with a solution in mind, and rushed to the office at 7 am to discuss with whosoever was there at work at that time. The optics guy I found did not fully understand it, but did not find it crazy either. A few hours later, the physicist friend confirmed my insight by running some more numerical solutions. I could then soon find tight enough upper and lower bounds, and the whole thing fit the measurements so well that most people thought it was just a "curve fit". (It was pure theory vs. measurements plotted together.)

5. I proposed making pixel-level optical measurements on mirasol displays using a high-resolution camera to watch those pixels after subjecting them to complex drive waveforms. Two interns were separately given the task (surprisingly without telling me), and both failed to develop algorithms for pixel-level measurements. Later a junior employee worked on it, was unable to develop pixel-level measurements still, though was able to get it to work at lower resolutions. The system took about 40 minutes of offline processing in Matlab. Later, a high-profile problem came up where pixel-level measurements were a must, and I was directly responsible for solving. Solved in one day. Processed images taken in real-time, not 40 minutes. The system stayed in deployment for years to come.

6. We had bistable MEMS devices, and there was a desire to make tri-stable MEMS devices. Several people at the company attempted it, including a respected Principal Engineer, but no one could figure how to even start. I could not figure either at the outset, but started bottoms up from Physics and using Wolfram Mathematica to create visualizations around the thing. And bingo. In a few days, I had not only figured how to make these tri-stable MEMS devices, but also multiple schemes of driving them. My VP's reaction was "Alok, you should patent that diagram itself", given the clarity it had brought on the table.

7. We were creating grayscale/color images using half-toning. A famous algorithm, Floyd Steinberg, works very well for still images but has lots of artifacts for videos. An PhD student working in the field was brought in as an intern, nevertheless, the results were not great. The team also tried binary search algorithms to find the best outputs iteratively, however, it was not implementable in real-time as needed. I was interested in the problem, but was not getting time to give it a fresh thought that it needed, until one day. A few days later, the problem was solved. I developed some insights into it and just had the solution coded, to the surprise of people who had spent months working on it.

I could go on writing about more cases.

So much work involved. Very complex problems, needs a lot of theory but also practical knowledge. Needs good debugging skills. And endless amounts of time.

It just sounds kinda sad to say that you've never been proud of work you've done. I think if I was never proud of the work I'd done I wouldn't still be doing this job: a feeling of reward is important!

I can imagine you get some pretty warty excel files. I mostly get PDF for my sins. I'm sure, like me, you've spent hours taking bits out of files until they work as expected and then figuring out what the difference is :-)

> It's all relative. Sometimes solutions require insights. Other times solutions involve a ton of grinding.

> Many people have a tendency to overemphasize the first and greatly under emphasize the second, even though the grinding may actually be harder than devising clever solutions.

> No individual problem is hard. It's arguably hard to do tedious work day in and day out for months at a time, but I don't think people would call that "technically hard".

Secondly, programming - to a degree - is "art". My version of a masterwork is different than yours.

But the framework decisions? Lack of documentation? Lack of source control? No dev environment? etc. Decisions and foundational information that is demonstrably wrong and needs fixing? And what we have done/are doing/will do? THOSE are the things to focus on.

Yes they are if you're talking about the types of systems that large companies have. It is in fact so difficult that I believe it's a considerable advantage to have scopes small enough to be manageable by a single engineer, but inherent complexity is often well beyond that, especially for very profitable business engines.

That does explain a few things. I am REALLY good at my job, and after working a job for a couple of years I become very good because of my disability, yet the same disability means I don't know how to interact with people so I interview poorly.

I don't know how to talk to people I haven't met and just doing it for practise isn't the way to learn. I really can't spare the 10 or 15 years it would take.

Ultimately, when someone asks something that's unexpected, makes a claim that's false or incorrect, I just freeze up and can't actually respond. In daily conversation with people I know it's not a problem, but with strangers I have to stand back and wait until I have a grasp their sense of humor, how much they think of themselves, and so forth, before I can speak.

I'd rather have engineers that can actually discuss technical problems rather than deliver smooth talk about how they are incredible.

"Hey I wrote this code"

"Why did you do that?"

"Because the frobnobbitz wasn't accounting for tribbles."

That's a conversation that runs backward. Anytime someone asked you 'why', and you answer, it's backwards. You don't have any control over what direction it is because the person asked you a question.

I'm really not sure but I suspect my own stories are less likely to paint an accurate picture of something than an interrogative conversation is. All humans have cognitive biases, so I wouldn't rule it out by thinking that I'm trying to tell the truth.

I've watched people practice true stories, many times, to a point that they become misleading. Not untrue, just misleading. (NB the author said "not entirely true".) Events are left out, motivations are made to look better intentioned than they were, etc. etc..

To be fair, I've seen plenty of lies made up on the spot too, so I'll refrain from defending the claim that one is more likely to be true than the other.

I still see value in placing conversation over narrative in a job interview.

I'd argue that my response isn't hyperbolic, and is justified considering how ludicrous that statement is.

How do STAR and SOARA not dictate chronology? You specifically discuss the initial situation first, and the results you achieved last (or the analysis of the results).

This ignores the problem that if someone can elaborate, it's not necessarily an indicator. It can just mean that someone can BS well. Relaying too many specific details can actually be an indicator that someone is not telling the truth.

"Getting people talking" is really the only way you can identify these things during an interview. And this is not the same as telling a good story. And now you have to demonstrate that this results in job performance.

On here, tptacek has exhaustively described an alternative approach: https://news.ycombinator.com/item?id=9159557

The vast majority of CS interview questions are really just one or both of two categories:

1. Say something entertaining or that makes me like you.

2. Say something that proves you're competent so if I like you it's not a hard sell to hire you.

When you read this hard into a question that can in this framework be reworded "talk about stuff you programmed that you thought was mentally interesting when you made it", they truly only are thinking about your skills at the most basic surface level, they really just want to let you gush for a minute.

Even in the most embarrassing code you've written there are dumb bugs and little moments of triumph, and they're begging you to share some of the juicy details, of which I'm fairly sure every programmer has a few they can recall.

If you have no example of work you've done you can gush over, then yeah it's a problem, but to me this is a sign that the only truly wrong answer is NO answer or trying to fake a modicum of passion by gushing about something you actually don't care about, and THEN sounding wooden when doing so, because if you didn't come off as wooden, even this would be sufficient.

We have an outside consultant who does one thing: Fix businesses.

When he says this is the worst situation he's ever seen? I take it with a little more weight than I'd take someone else saying it.

While I understand and don't disagree with what you say - a full rewrite is normally not the answer - you haven't seen this codebase. Or the company structure.

We aren't exactly doing a "full rewrite"... it would honestly be easier in many respects - we are keeping the company functional while replacing large chunks.

Aka Keeping the Train Moving while changing the Engine and the Wheels.

This isn't JUST a code base issue. Or JUST a culture. Or JUST management. It's a combination of all those - and many more that can't be covered in 3 paragraphs.

I could talk for 8 hours - and scratch the surface - of where we are and where we need to be.

Some people are humble about their work and abilities. They will never exhibit open pride. Or they don't want to bullshit people and blow smoke up someone's rear. Some may want to switch jobs because they are forced to do poor work and know they shouldn't feel proud about that.

This is probably a 'flaw' I have, but I'm aware of it that I know what I'm expected to sound like when asked this question. Now I'm put in a position where I am being dishonest. You're now testing my ability to BS you to get what I want, and by my nature I am already uncomfortable with BS'ing people.

Now compare this to the person who is too incompetent to not feel pride in the shoddy work they do.

Now the enjoyment part. I can talk about this extensively, but enjoyment is subjective. I could enjoy working on something because of the challenge of the problem, or because I was part of a great team, but it still doesn't speak to my ability to perform the job you have.

A major difference is that some people have deficiencies in autobiographical memory. This can make telling the story of a past project vastly more difficult than talking about a subject of current focus. An extreme case was recently described in Wired [1], but the ability is generally more of a spectrum.

[1] https://www.wired.com/2016/04/susie-mckinnon-autobiographica...

It confuses story-telling performance during an interview, and the stresses of a success/fail situation, with the type required to perform real work.

Mistaking "overlap" for all-encompassing. Sure, there's overlap. There's overlap in being able to type up a coherent response to a post on Hacker News, but it doesn't make me more qualified for your position.

It fails because it assumes that the candidate has a single defined favorite project. If you try to make the question more broad, then it becomes so open-ended that it is hard to equally compare candidates on it. It becomes random chance that a candidate discusses a project in a way conducive to doing the job. Where if more precise or structured, it might weed out people who can talk passionately about a personal project from those who can constructively explain to a manager the challenges of a project.

It also starts stretching it into saying that everything is story-telling so as to make the term meaningless. I agree it is to some degree, but it's important to try to stick with meaningful mental models that can make predictive assessments about candidates.

The huge problem is there are people without the skills who can tell a good story. This process let's these people through, while ignoring good candidates who may not perform well on this interview question.

And big wins like that are usually compacted pretty early on. If you can get orders of magnitude improvements left and right, it means that something about the company's management is off.

It's also not a good question for an interview because it's a hard basis for comparisons. Perhaps another candidate knew how to improve the build/deployment pipeline, but he was blocked by political interference. He wouldn't be able to say "I sped up the pipeline 6x.", but he could talk about his plans to do so with a question more oriented to the task, e.g., "How would you build your dream deployment pipeline?"

That's what I mean when I say they're looking for something spectacular. Some people can say they saved their company or made a change with massive ripple effects, which is not necessarily aligned with the technical difficulty of that change and may cause some candidates to elide mention of it entirely, and some people can't make such big assertions, not because they're not skilled enough, but because the opportunity and/or priority wasn't there.

If you want to know about business gains and side effects of technical work, ask "How did your work help your employer?" If you want to know about technical work itself, ask relevant lines of questioning.

Anyway, I think we're basically splitting hairs here. It's just about what level you're choosing to process the question on, and it seems everyone agrees that it's best to take a very superficial interpretation and allow them to inquire further as necessary. I just don't think it's a good interview question.

Sometimes it's not about confidence. Sometimes it's about complex social dynamics and how people react to suddenly being thrown into the deep end of a domain that's completely new to them.

"Tell me about a technical challenge that won't make me feel inadequate or be difficult to follow. Don't make it too simple though because then I'll think you're an idiot."

Ah yes, just be confident. That's the ticket.