Why Law Firm Rankings Are Useless(blog.litimetrics.com) |
Why Law Firm Rankings Are Useless(blog.litimetrics.com) |
No. That's what TV lawyers are paid for. That's the equivalent of learning about doctors from watching MASH. Most doctors are not surgeons. Most lawyers are not litigators. Many great and valuable lawyers will never win any case. Imho the best lawyer is the one who's client never faces the uncertainty and protracted negativity that is modern litigation, just as a great and valuable doctor is one who's patients avoid the knife. (Apples and oranges but the point stands.)
Lawyering is, in my world, mostly about listening to clients and understanding how their realities fit into external standards such as laws and industry norms. It's about hedging, documentation and interpretations. And sometimes it's about holding the client's feet to a fire. Litigators only appear once all of that has failed. Litigators may have the highest hourly rates, but do not have the highest salaries overall. Those are reserved for corporate counsels and board members.
In some areas, big money comes with big litigation.
According to the article, their analysis does account for that. They allegedly "calculate the probability of a successful outcome for the applicant" using machine learning. But there is no description of how they do that. If I had an algorithm that produced halfway decent results in an automated fashion, I wouldn't use it to get into the legal technology market. I'd set myself up as a litigation funder and make boatloads of money with accurate valuations of potential investments.
Legalist is a recent YC startup aiming to do exactly that:
http://www.newyorker.com/business/currency/what-litigation-f...
My understanding from talking to litigation funders is that lawyers are terrible at valuing cases. There is money to be made for anyone that can use technology to improve those predictions.
Not a problem for the analysis of law firms. Sierra Club is a less effective law firm than the big New York firm.
Why is this not a problem? The parent comment was saying that the Sierra Club takes harder cases and is often the plaintiff, not the defendant, so they may be more effective as a law firm but have worse numbers because the win/loss numbers don't mean the same things in different contexts.
-----------------
how exactly did they model the probability of successful outcomes for given cases?
because those depend on highly detailed fact patterns, many of which are largely qualitative in nature
if they were actually able to put all those in a data base that you could then model
that'd be a more interesting accomplishment to read about
We want to rank lawyers based on effect on outcome, not win-loss record. But it looks like the model is one that derives "counterfactual" outcomes by making comparisons based on win-loss records.
Can someone give a better lay-level discussion of the expected outcome model?
But you can't assume that about lawsuits. If the client should have paid ten million in damages, the better lawyer might limit the award to nominal damages. But the same nominal damages loss might be a disaster in another case the lawyer should have won.
Lawsuits aren't zero-sum, bilateral agreements to compete like most sports. Parties aren't just competing for a 'win,' but looking to maximize gain or minimize loss.
You can apply this to a wide variety of sports where most teams don't get to play most other teams, eg you can guess that Germany are better than India at soccer despite the rarity of the tie.
And of course you can modify the system to take account of various issues such as the retirement of better players, which tends to remove points from the system.
I worked with several "independent" lawyers before, and honestly, they are nowhere near close to the level of skill of my current law firm. In addition, they're quite responsive, much more than the average independent lawyer. And yes, they're more expensive, but not outrageously so. They seem to reach a favourable outcome faster, and seem to be more effective.
Tl;dr: Love the lawyer I work with in this firm, he gets the job done when I need him to + has a great team with broad experience to back him up. He's managed to resolve any tax issue before we even went to court.
Are you speaking hypothetically, i.e., 'if I was a lawyer'? I'm not an attorney but for a variety of reasons I have relatively deep insight into the legal industry, and I think this statement is generally not true.
As in any field, most business comes from referrals and networking, and much of that happens within a firm. If your career is at a top firm then you are going to build a much different network than a similar attorney who opens their own office in a strip mall. When Goldman Sachs needs an outside attorney, where do they go and who do they ask? Even ignoring referrals, just imagine who is there when you are invited to dinner by your colleague: Is it the mayor and local Fortune 500 CEO, one of whom takes a liking to you and asks you to join the board of a local charity, or is it the lovely neighbors who drive Uber or run a small medical practice, and ask for some legal tips?
Personally, I don't enjoy the hunt for status and I think it impedes socio-economic mobility, but I'm afraid that's the way it works.
And, with that sentence, this post becomes bullshit.
The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned.
Even at that, only on the order of a single digit-percentage of disputes ever reaches a verdict.
This is like judging a hospital-system based on its success-rate for transplant surgery. It's just deeply ignorant of what constitutes the vast, vast majority of the practice of law - even in disputes.
This article has no value, whatsoever, aside from a witty aside at a cocktail party about how deeply misunderstood the legal profession is. This is particularly shocking given that litimetrics is a legal service provider.
> Although A has not faced B, if both have faced C, then we have some information about the hypothetical A and B matchup. This is the same reasoning that allows one to compare tennis players across different eras.
The implication is that the lawyer efficacy, based on judgments, is subject to the transitive property? Sweet mercy that is ridiculous.
This post is a catastrophe.
This is a pretty ungenerous reading of the article. Your complaint seems to be mostly about the generalizability of the model (a legitimate complaint, as you've explained) rather than its statistical validity. It still provides some interesting insights about the performance of firms at trial. Notably, there isn't a correlation between favorable judgments in those cases which have gone to trial and traditional law firm rankings. That's surprising, even if it doesn't generalize to overall law firm "success".
We aren't claiming to assess the wide practice of law, just litigation. We do Litigation Analytics.
And do you look at motions other than actual judgments? I.e. if Firm A wins a motion to suppress some piece of evidence and then there's a subsequent settlement, we should infer that Firm A "won" the case, even if there's no judgment to that effect.
If I felt that I owed you nothing, surely this counts as a "loss" for me at trial.
If, however, you had originally sued for $50M and I felt I might owe you some money but no more than $1M so I decided to fight it, the $100k judgment would be considered an enormous "win" for me.
What data does your analysis use to distinguish between the two scenarios given that the verdicts are identical?
That said, it might be interesting to apply machine learning at the appellate level. There, the outcomes are public and more readable by machines. Something as simple as looking for tell-tales such as "reverse", "dismissed" or "we side with the appellant" in holdings could be mapped to firms.
It seems reasonable to me that a firm with a great track record of judgments is more likely to succeed in other significant areas than a firm without a similarly great track record of judgments. That is, I would expect a strong, positive correlation between winning judgments and other desirable outcomes in disputes, like winning settlements.
We could even measure that correlation through a representative sample of firms to prove or disprove that hypothesis (and the OP probably should), but dismissing it outright with hyperbole isn't an effective response.
It seems reasonable to me that a firm with a great track record of judgments is more likely to succeed in other significant areas than a firm without a similarly great track record of judgments.
In science guess who has the burden?
In a legal context the burden of proof is broken down into the burden of production and the burden of persuasion. If you fail to meet the burden of production it's fair to say a claim is bullsh*t. It necessarily follows that if you haven't met the burden of production you cannot have met the burden of persuasion as a logical matter.
Remember, any claim must stand on its own terms. It doesn't matter what is the reality or truth of the matter. We can never know the absolute truth of something. We can only attain a slightly firmer grip on reality when people propose persuasive arguments. Maybe you can mine a failed argument for useful bits, but a good rule of thumb is that it's not worth the time if the argument cannot even meet a minimal burden. If it wasn't worth the claimant's time why would it be worth yours? That's strong circumstantial evidence to move on.
Lawyers can drop clients with cases that they think are losers and refuse to take them to trial. And, very, very often, before or directly after closing arguments, if the writing is on the wall, a case settles.
Additionally, what sorts of cases are we talking about? Insurance defense? Toxic torts? Data breach?
The 'winnability' of these cases varies highly. There are many, many law firms who's entire business model is built entirely only on taking no-brainer winning cases. Think injury lawyers. They may - perversely - have very low winning percentages, because 90-95% of their cases settle, and only the real squeakers get to trial.
Let's look at the inverse. Do you really need Quinn Emmanuel or Gibson Dunn if you have a slam-dunk case? Or do you need the best litigators around when your case is a total coin-toss? And, in that event, is it an example of bad lawyering if your crack-team loses because they are pushing the bounds of appellate advocacy?
The idea of judging 'law firms' without further taxonomic distinction, generally, by trial disposition is just - it is frankly idiotic.
Not so shocking. Law firm ranking services are their competition. They drew a conclusion from the data that supports their business.
Law firm rankings absolutely can provide a value. Both on the firm and client side of the transaction.
Imagine you're on the client side of a qui tam (Whistleblower) or Immigration case. Are you going to bring that case to a law firm that you don't have visibility into a) how the firm is trusted by their clients and b) how the law firm is regarded among other law firms? That would be crazy!
Heck, in a qui tam you can't even get quality analytics into how well a firm litigates a case because the best qui tam cases get picked up and tried by the Federal Government!
(Disclosure: I'm a dev at a law firm.)
Size of marketing team might correlate with success/positive market perception though.
Therefore, I disagree that this post "becomes bullshit" merely because it relies on trial outcomes.
It will take _years_ before your case actually goes to trial and you will meet plenty of lawyers along the way if you aren't careful who will happily fuck off with your money. And you'll have to sue them separately to deal with that, if you even have the energy to spare.
Good luck having a full-time job through that mess.
Read above to my last reply.
This is really a poor way to think about law firms.
Is the firm a toxic tort firm? Is it an appellate advocacy firm? Is it maritime law? Is it an intellectual property dispute? A business dispute?
Win percentages are totally meaningless without more information, including client-positive settlement rates.
This metric is totally useless.
I'm shopping for a personal injury lawyer. How to pick? The two referrals I got didn't pan out. I don't know any lawyers. Aha, search the court documents to see who the winners are!
I (finally) consulted my paralegal friend. He made the same points you have.
Additionally, he said the courts all use different IT systems. One could use an expensive service to do the search, but it's hardly worth it for laypersons.
Too bad.
I did start to wonder if one could data mine the filings, just to see who's getting the most cases, and maybe infer something actionable from that.
All real-world analysis (especially for observational studies) rests on certain assumptions. It is always true that these assumptions might be wrong, but it is important to think about whether or not the assumption is plausible. It seems plausible that on average, a lawyer who is able to get better outcomes when they don't settle is also able to get better outcomes when they do settle.
Furthermore, even if most cases are settled, the rare cases that do go to trial can have an outsized impact. Usually people settle because a bad judgment is devastating (as well as not wanting to pay legal costs).
As a matter develops, parties tend to drop out if they think that their chances have declined beyond some threshold. But if they continue, or more likely, their legal representation suggests they continue, it implies that they think they have a good chance of success.
Of course, you can get irrational litigants but most of the time, if it goes to judgement, both parties think they have a high chance of success. Otherwise they would have bailed.
Furthermore, on average the effects you are mentioning will wash-out, unless there is a systematic bias whereby lower ranking firms and higher ranking firms settle in different manners.
You honestly believe that an opponent won't take a 90%+ trial win rate into consideration when considering a settlement? All you hear from lawyers pretrial is "our chances at trial are X" and then based on whether X is high or low, they'll make a decision to go to trial or settle.
First, I'm a 7th year corporate lawyer.
Second, of course I would look at it. However, it would not be dispositive and I would likely discount it super, super highly. I have vastly superior data about the winnability of my case as a result of being the attorney on the case than I would knowing the 'win' ratio of my opponents law firm - which is almost totally irrelevant to the case at hand. Ceteris paribus, bad lawyering loses more cases than good lawyering wins cases. This in mind, the thing to know is whether or not your opponent is a bozo or competent. Regardless of your opponent's competence, you act as if he is at least as smart as you and I would litigate as hard against Ted Olsen as I would someone no one has ever heard of. The controlling factors come down to your client's budget, your opponents budget, the animus they have toward each other and, lastly, the facts.
Additionally, channel checks that I perform on opposition attorneys are far, far more sophisticated than algorithmically computed win rates. Maybe I'd use it as a data point, but the frank reality is that I have far better means at my disposal for judging the merits of a case and the efficacy of my litigation opponents including google, the phone on my desk and my judgment. Law is a very close-knit and reputationally driven profession. I do channel checks on opposing counsel frequently. It is an extraordinarily rare situation where I cannot get a second opinion, from someone who's judgment I trust, on another law firm or another lawyer when I really need one.
Finally, what is missing from this whole consideration is that law suits are not fair fights and they are highly fact and context dependent. Not every game has the same rules, the rules and the facts may be skewed. Does it matter if a particular NFL team is last in the league if they are going to be playing a HS JV team in freeze-tag? Accordingly, it is not like the win ratio of a poker player or a football team. The case itself very often is going to have a huge skew in favor of one opponent or another. If I had a winning case and you told me that my opponents were Abraham Lincoln's Zombie and Scalia's ghost, I'd still fight it.
Let's make an analogy: Let's look at batting averages as a predictor of general sports ability. But instead of confining the batting averages to major league players and major league pitchers, let's make it the life-time batting average of every American over the age of 18, for every game of stick-ball, alley-ball and beer league softball they have ever played in their lives, referreed or not, as a predictor of their ability to play "sports" generally, regardless of their age.
I think that is a fairer analogy to the 'win' ratio for trial dispositions.