Do you think every business owns the land and building it operates in? Real estate is expensive. Maintaining a building is expensive. There are plenty of businesses that rent to avoid the capital requirement and headache of property ownership.
Someone who is discrediting all of philosophy shouldn't confuse Wittgenstein and Kant.
Further, Wittgenstein disavowed Tractatus as a failed project and completely revised his approach to philosophy. His most important and influential works came afterwards.
He disavowed is as comprehensive account of linguistic meaning, but I don't think he regarded it as false or meaningless, only that the full breadth of ways language conveyed meaning was wider than the account given in Tractatus.
> Someone who is discrediting all of philosophy shouldn't confuse Wittgenstein and Kant.
Getting the names confused is not the same as getting the people confused. My poster child for philosophical nonsense has always been the Tractatus. I just somehow got it into my head that it was written by Kant, not Wittgenstein (I've always been bad at remembering names) and I didn't bother to check because I was writing an HN comment and not a paper for publication.
Okay, but you initially criticized Wittgenstein, the philosopher, not Tractatus, the work. Wittgenstein himself would agree that Tractatus is deeply flawed. He wrote his more influential works later, and they went in a completely different philosophical direction. You're criticizing a philosopher as "pooh-pooh-able" for a work that he personally disavowed and does not represent the positions he is best known for.
I was intending to criticize the field, and in a shot-from-the-hip in a moment of some passion chose Wittgenstein as my example.
> Wittgenstein himself would agree that Tractatus is deeply flawed.
So I am vindicated. I'm not actually criticizing Wittgenstein for writing Tractatus; there's nothing wrong with writing nonsense. Lewis Carroll was a master. The problem is writing nonsense and not recognizing it as nonsense. I'm criticizing the field of philosophy for elevating Wittgenstein to iconic status after having written such manifest nonsense without recognizing that it is manifest nonsense. That is an indictment of the field, not the man.
BTW, the reason that this is a touchy subject with me is that I did my masters thesis (in 1987) on the subject of intentionality [1] in AI. After wading through dozens of inscrutible papers I came to realize that the whole topic was basically bullshit [2], and that the problem had been completely solved by Bertrand Russell in 1905 [3], but no one seemed to have noticed. Even today the vast majority of philosophers (AFAIK) think this is still an open topic.
And BTW, Russell's solution is beautiful and easy to understand. Frankly, I think it has been ignored because it is easy to understand.
I think there's a lot of confidently wrong histories being tossed around this thread, and it's not quite right to say he abandoned his old work as meaningless. He considered it dogmatic, but not nonsensical by any stretch.
My gripe is that the commenter above cites early Wittgenstein as an example of the failure of philosophy as a whole, while ignoring (or perhaps being unaware) that later Wittgenstein is what is philosophical "canon". I'll concede there is some debate about how Wittgenstein's views evolved over his life and the extent to which he repudiated his earlier work. But I think you're going a bit far by characterizing what I said as "confidently wrong history," if that's directed at what I wrote.
I'm actually quite agreeable to idea the that much of philosophy is incoherent nonsense. I would have completely gone to bat for this commenter if they said Heidegger was such an example. Or Searle for that matter. I can even see the case for Kant. And they all have their defenders, just not me.
But even for someone as sympathetic to that argument as I am, I don't see any version of Wittgenstein's reflections on the Tractatus as agreeing it to be nonsense much less a paradigmatic example of it. It's not just a matter of the later Wittgenstein being the "good" stuff. The Tractatus built on the work of Frege and was incredibly dense in its logical expressions, and half the challenge is keeping up with him, because he did philosophy from the perspective of an engineer, knowledgeable in logical and mathematical notation. It's one of the essential works of philosophy from the 20th century.
What do you think is nonsensical about Searle? My sense is that he's very much not obscurantist in the way one might think Heidegger or Kant is (of course, the defense is that they use technical language because they're discussing technical things). But maybe you just mean that his arguments fail.
The deep dive version of this convo might be a topic for another time, but the most concise answer I can give is to grab a copy of I Am A Strange Loop by Douglas Hofstadter, and flip to chapter 2 where he discusses an idea proposed by John Searle about the so-called "terribly thirsty beer can." This is an argument from Searle that he believes is a knockdown argument against the idea of consciousness embodied in something that isn't a biological mind as we know it. It is, and I do not say this lightly, it is just stunningly naive.
Hofstadter's dispensation of it in chapter 2 is to my mind, a completely decisive dressing down of the fundamental naivety of Searle's ideas about minds. I can't find any convenient quotation of the passage on the internet, but in my copy of the book it's page 29 chapter 2.
I think it puts on perfect display how truly ridiculous Searle's ideas are, and I think the Chinese room idea is similarly discreditable, and ultimately I think that Searle was more a fraud who more appropriately belongs in the category anti-science apologists along the lines of intelligent design proponents, rather than a positive contribution to the canon of Western analytic philosophy. And the extent to which he has gained influence in academic philosophy is something I take as discrediting of it as a field, to the extent that Searle is it's standard bearer. So if the commenter above cited him instead of Wittgenstein I would be cheering it on as a legitimate observation.
I would acknowledge that this is a rather original take of my own and you won't find many people who subscribe to it.
But the essence of it is, if you consider optimists about the possibility of computers and AI, whether they be philosophers or programmers for major tech companies, and then you consider an opposing camp, made up of various 20th century philosophers, the most prominent of them being Hubert Dreyfus and John Searle, the second camp attempts to approach the problem by asserting that there's an essence to intelligence or consciousness, and that essence is captured in certain key pieces of vocabulary, such as insight, "thinking" and so on. They declare that these are special things that human minds have, that by definition, in some sense, cannot be modeled by any formal description or scientific investigation, and the essence of their definitions is always a moving target.
Their approach to the topic also parallels that of intellectuals who insisted that Darwin was wrong about evolution, and the essence of their insistence was a failure of imagination for the explanatory power of evolution. Obviously I'm oversimplifying, but in some ways you could consider the crux of the debate to be this posture of incredulity that the spectacular complexity of life could be explained the iteration of essentially simple and blind rules.
Searle and Hubert Dreyfus, but Dreyfus especially, looked at the logic gates of computing, and then looked at the dynamic, associative, poetic, analogy oriented aspects of human thinking and thought that these contain some magical essence that couldn't possibly be modeled by computers, and that the fundamental ideas of computing needed to be replaced by some new set of core ideas. However, our recent breakthroughs, while they are based on special and new principles that relate to vector databases, convolutional networks and so on, perhaps exhibiting the very core ideas that Dreyfus and Searle believed were missing, those breakthroughs have happened on the same old boring foundation of computing, with logic gates and whatnot, and there was a failure of imagination on their part to understand that those foundational principles could give rise to the more dynamic concepts that they believed were necessary, and that these two things were not in fact in conflict at all.
And the preemptive assertion that they belong to two a category inaccessible to computing principles as they understood them, indeed to any sort of in computational principles of any kind whatsoever, is something that I would contend is a fundamentally anti-scientific instinct that comes from a place of lacking imagination.
But just for the record, re-reading it now 37 years after the fact, I would not hold it up as a particularly good piece of work. It's a bit cringey actually.
Critique of Pure Reason is Kant. I thought you were making a joke based on the earlier mixup between Kant and Wittgenstein. Late Wittgenstein is Philosophical Investigations. There are also good texts on philosophy of language that excerpt from the major authors (including Wittgenstein) without requiring you to read the entirety of their books.
Nope, not a joke. Just the same mistake I made originally. I guess I have conflated Kant and Wittgenstein in my mind even more thoroughly than I thought.
My bad. I've gotten pretty overwhelmed with all the activity in this thread, and I'm trying to get some actual work done in between responses so I'm a little distracted.
The third one (CUAD) is a single paper, not blogposts like the others. I think this paper is still the best in terms of being done by NLP experts and understanding the possibilities and not being just some and mirror. But there are so many papers published in this area nowadays that I might not even notice a new one.
The CUAD paper was still based on BERT, so pretraining was needed - that needs a bit more expertise than just prompting GPT-4-32k like in this paper or feeding prompts back to GPT-4 for another round of refining, or doing RAG.
For honest research purposes, "contract review" is not really a good area of approach: the subject field is not standardised, there is no good benchmark yet and your paper can easily get into bad company of snake oil sellers cashing on visceral hatred of average people for all professions.
How do you think organizations can best use the contractual interpretations provided by LLMs? To expand on that, good lawyers don't just provide contractual interpretations, they provide advice on actions to take, putting the legal interpretation into the context of their client's business objectives and risk profile. Do you see LLMs / tools based on LLMs evolving to "contextualize" and "operationalize" legal advice?
Do you have any views on whether context window limits the ability of LLMs to provide sound contractual interpretations of longer contracts that have interdependent sections that are far apart in the document?
Has your level of optimism for the capabilities of LLMs in the legal space changed at all over the past year?
You mentioned that lawyers hoard templates. Most organizations you would have as clients (law firms or businesses) have a ton of contracts that could be used to fine tune LLMs. There are also a ton of freely available contracts on the SEC's website. There are also companies like PLC, Matthew Boender, etc., that create form contracts and license access to them as a business. Presumably some sort of commercial arrangement could be worked out with them. I assume you are aware of all of these potential training sources, and am curious why they were unsatisfactory.
Not op but someone that currently runs an ai contract review tool.
To answer some of your questions:
- contract review works very well for high volume low risk contract types . Think slas, SaaS… these are contracts comercial legal teams need to review for compliance reasons but hate it.
- it’s less good for custom contracts
- what law firms would benefit from is just natural language search on their own contracts.
- it also works well for due diligence. Normally lawyers can’t review all contracts a company has. With a contract review tool they can extract all the key data/risks
- LLM doesn’t need to provide advice. LLM can just identify if x or y is in the contract. This improving the process of
review.
- context windows keep increasing but you don’t need to send the whole contract to the LLM . You can just identify the right paragraphs and send that.
- things changes a lot in the past year. It would cost us $2 to review a contract now it’s $0.2 . Responses are more accurate and faster
- I don’t do contract generation but have explored this. I think the biggest benefit isn’t generating the whole contract but to help the lawyer rewrite a clause for a specific need. The standard CLM already have contract templates that can be easily filled in. However after the template is filled the lawyer needs to add one or two clauses . Having a model trained on the companies documents would be enough.
Do you think LLMs have meaningfully greater capabilities than existing tools (like Kira)?
I take your point on low stakes contracts vs. sophisticated work. There has been automation at the "low end" of the legal totem pole for a while. I recall even ten years ago banks were able to replace attorneys with automations for standard form contracts. Perhaps this is the next step on that front.
I agree that rewriting existing contracts is more useful than generating new ones--that is what most attorneys do. That said, I haven't been very impressed by the drafting capabilities of the LLM legal tools I have seen. They tend to replicate instructions almost word for word (plain English) rather than draw upon precedent to produce quality legal language. That might be enough if the provisions in question are term/termination, governing law, etc. But it's inadequate for more sophisticiated revisions.
Didn't try Kira but tried zuva.ai, which is a spin off from them. We found that the standard LLM performed at the same level for classification for what we needed. We didn't try everything though. They let you train their model on specific contracts and we didn't do that.
For rewriting contracts keep in mind that you don't have to actually use the LLM to generate the text completely. It is helpful if you can feed all the contracts of that law firm into a vector db and help them find the right clause from previous contracts. Then you can add the LLM to rewrite the template based on what was found in the vector db.
Many lawyers still just use folders to organize their files.
Sometimes it feels like people look at GPT and think "This thing does words! Law is words! I should start a company!" but they actually haven't worked in legal tech at all and don't know anything about the vertical.
A friend of mine is a highly ranked lawyer, a past general consul of multiple large enterprises. I sent him this paper, he played with ChatGPT-3.5 (not even GPT-4) and contract creation, he said it was 99% fine and then told me he's glad he is slowly retiring from law and is not envious of any up-and-coming lawyers entering the profession. One voice from the vertical.
People started companies and succeeded for dumber reasons. Generalized "businessing" skills and placing yourself somewhere in the space where money is made counts for much more than actually knowing anything about specific product beforehand.
I would add, that sometimes being a newcomer is a benefit. Many times a particular industry is stuck in groupthink, having shared understanding on what is and what isn't possible. And it sometimes requires a person with a fresh perspective to be able to upend it. See example of Elon upending multiple industries by essentially doing exactly that.
It's a logical reaction, at least superficially, to the touted capabilities of Gen AI and LLMs. But once you start trying to use the tech for actual legal applications, it doesn't do anything useful. It would be great if some mundane legal tasks could be automated away--for example negotiation of confidentiality agreements. One would think that if LLMs are capable of replacing lawyers, they could do something along those lines. But I have not seen any evidence that they can do so effectively, and I have been actively looking into it.
One of the top comments on this thread says that LLMs are going to better at summarizing contracts than generating them. I've heard this in legal tech product demos as well. I can see some utility to that--for example, automatically generating abstracts of key terms (like term, expiration, etc.) for high-level visibility. That said, I've been told by legal tech providers that LLMs don't do a great job with some basic things like total contract value.
I question how the document summarizing capabilities of LLMs will impact the way lawyers serve business organizations. Smart businesspeople already know how to read contracts. They don't need lawyers to identify / highlight basic terms. They come to lawyers for advice on close calls--situations where the contract is unclear or contradictory, or where there is a need for guidance on applying the contract in a real-world scenario and assessing risk.
Overall I'm less enthusiastic about the potential for LLMs in the legal space than I was six months ago. But I continue to keep an eye on developments and experiment with new tools. I'd love to get some feedback from others on this board who are knowledgeable.
As a side note, I'm curious if anyone knows about the impact of context window on contract interpretation a lot of contracts are quite long and have sections that are separated by a lot of text that nonetheless interact with each other for purposes of a correct interpretation.
I think one of the biggest problems with LLMs is the accountibility problem. When a lawyer tells you something, their reputation and career are on the line. There's a large incentive to get things right. LLMs will happily spread believable bullshit.
