Ethical concerns, LLMs, and sensitive data in Legal | Tim Golden | Data Science Hangout

Transcript#

This transcript was generated automatically and may contain errors.

Hey there, welcome to the Paws at Data Science Hangout. I'm Libby Heron, and this is a recording of our weekly community call that happens every Thursday at 12pm US Eastern Time. If you are not joining us live, you miss out on the amazing chat that's going on. So find the link in the description where you can add our call to your calendar and come hang out with the most supportive, friendly, and funny data community you'll ever experience.

I would love to introduce our featured leader today. His name is Tim Golden, and I'm going to make sure that I get Tim's title right. Director of Data Science Analytics at Morrison & Forster LLP. Tim, can you introduce yourself? Tell us a little bit about what you do as Director of Data Science Analytics at Morrison & Forster.

It's Data Science Analytics, basically Data Science and Analytics. I am on the innovation team. I report to the Director of Innovation, and my job is to handle all the engineering and technical bits and gen AI bits for how we innovate in the practice of law. We've got innovation teams in the business of law side for the administrative departments. My focus is primarily on the practice of law, on how to help our attorneys better serve clients using technology.

I've been in legal for 30 years. I've been in three large firms. I've been at MoFo for the last eight and a half years. It's great to be here.

I am at heart someone who doesn't want to do things more than once. So, if I'm working on a project, almost always I'll start a package and, you know, write tests and then build a package from there.

AI and LLMs in the legal profession

Yeah, we have a lot of attorneys who are experts in this area. And so I don't want to talk out of school. I can speak about, you know, what I'm seeing is that it is an arms race in legal to see how, because even before this happened, even before Gen AI sort of took off, there was pressure from our clients to reduce costs, to have more repeatable costs.

This started in the nineties with, and there was a large knowledge management movement in legal, where how do we reuse a work product? There were a lot of clients that were asking for alternative fee arrangements, like fixed fees or not to exceed, that are different from your normal sort of just time and materials kind of billing structure. And so that was driving things.

And that has just skyrocketed now that Gen AI has taken off the way it has. We do a lot of work with internally developed software that uses LLMs, as well as third-party products that use them, specific to legal. All of the firms are trying to figure out how to use these technologies in a way that best serves the clients and is safe. I will say every bit of LLM, Gen AI infused research, we don't use it in legal research, but anything that we apply these to, every single output is reviewed by an actual attorney. None of it goes through without people in the loop. And I think for legal in particular, that is critical.

Building AI tools before enterprise solutions existed

I like to say, don't pre-filter what you think is possible. Just dream big about what you want to do and then let us look at it and then we'll tell you whether or not it's something we can do.

And so it was that sort of attitude that made us build an AI chatbot before there was an enterprise ready AI chatbot. We built it in-house and I was like, oh, great. Now there's Chat2P Enterprise, there's Cloud AI, there's Gemini. We use all of those at the firm. But before we use those, we rolled our own. And my team did that.

My team handles the backend and the APIs and the data science side and wrangling the AI. And then we have another team that's wonderful and they handle the UI side of things. And that's how it's traditionally been. The firm has restructured over the last, let's say nine months. And now we're more into product teams. And so instead of my team just handling the backend AI side, I'm on the innovation pillar. And so we're a full stack group that handles requests from soup to nuts.

Yeah, those happened to show up right at a time where I was building my own versions of that. And so I was very happy. Less happy with ellmer , even though ellmer is amazing, but I had spent so much time making something like ellmer that I was like, oh, and of course, because of who's involved in ellmer, it was way better than anything I did. But I was like, all right, that's cool. And then I was able to just right away use these others because we had written our own, we rolled our own rag packages.

When I first started using them, I had to extend them internally because they weren't at that time built to really work with Azure. And that's where most of our LLM deployments were. Now we've got things all over, but yeah, that suite, all of the Gen AI pieces that Posit and the other developers have put out are, they work really well together.

Evaluating models and keeping skills sharp

Yeah, I mean, certainly there are areas of law that Gen AI lends itself to more readily than others. And I'm not saying that you can only use it in one area, but transactional work tends to be where you get the quickest gains early, as opposed to litigation, right? Analyzing deal books, terms and clauses around mergers or things like that, that's an area where Gen AI found early gains.

If you look at like the big players in legal AI, you've got companies like Harvey and Legora. And, you know, both those companies have great ways of dealing with, let's say you've got 35 contracts. And as I think of those as rows, and then you've got questions that you want to answer about each of different areas of each of those. So those are columns. And so you get this tabular AI going in and analyzing each document for that question and coming up with an answer. Absolutely. Lawyers have to look at that. That is not, you can't take any of that as good out of the box.

Every time there's a new model, you can't just assume, oh, this model is going to be working at the same level as the last one, but better. I mean, I think we can see from when, what was it when GPT 4.1 came out and was so sycophantic that you, you know, it's a step forward in one way, but it could be a step back in another. So there's a lot of work that we have to do on the tech stack side, on the data science side to make sure that when the models change, that the skills and capabilities that you're relying on, that they still meet your tests. And that's where one thing, a package like vitals is really helpful, where it allows you to quickly test across multiple models, across multiple questions against what you believe the answer should be to identify issues.

There is an organization called ILTA, the International Legal Technology Association. And it is a wonderful community of law firms, big and small, includes individual practitioners all the way up through the largest firms in the world. And that is a great organization where you can ask questions.

One of the things I love about the legal industry is that we share everything. We really, that as far as the technology people, we don't see, I mean, obviously we, there are competition, but we're all in it together. And we look at it as if I share something with you, you're going to share something with me and we both get smarter faster, which is only helping us. Where if we hold everything to the chest and say, well, we have special things that we can't tell you about. Well, then you only know what you know, you don't know what everyone else knows.

And we look at it as if I share something with you, you're going to share something with me and we both get smarter faster, which is only helping us.

It's the opposite of what most people think about legal. And it's the reason I've been in this industry for 30 years is because of, I have, I know so many people at other firms and all of us are just perfectly fine talking to each other about what we're doing. Obviously, we're not giving away anything related to specific clients or matters, but we share a lot.

Hiring and what Tim looks for

What I look for in when I'm hiring is not that you necessarily have law firm experience, but that you have professional services experience or experience in education or healthcare, because they have similar types of stakeholders. But even that, it doesn't have to be experience in there. What I'm really looking for is can you tell a story? Can you communicate? Can you actively listen? Can you translate what someone is trying to get over to you as a problem into something that they understand and they agree with, as well as something that can be translated into technical requirements and into something that can be resolved. So I really want everyone on the team to have some level of business analysis capability, because we're just too small to say, well, all I do is code or all I do is work on models.

How Tim got into legal

I was working part-time on a help desk for Circuit City back when Circuit City existed, before Best Buy opened stores across the street. They had what was called Answer City. And if you bought a computer, for the first 90 days you had your computer, you could call this hotline for free as many times as you wanted for as long as you wanted with any question you had about your computer. And so I'd get a 10-year-old trying to make a boot disk for a game, because this is back in, like, 95. And then I'd get a doctor who just bought 30 PCs and needed help networking all of them in his office.

So I did that. And so that was basically, like, boot camp for troubleshooting and customer service. And from there, I took a job part-time on the help desk at a law firm when I was towards the end of my studies at VCU for Jazz. And they said, hey, you should come work here full-time. And I said, well, yeah, that sounds good. So I said, all right, I'll start. But I don't want to be on the help desk anymore. I want to be on your server team. And since I had written their, I'd scripted their Windows 95 upgrade just while I was on the help desk, they said, all right, we'll take a chance on you. And that was the start of my career in legal 30 years ago.

Sensitive data, ethical walls, and championing AI

So while the legal industry isn't a regulated industry, I've always said that we're regulated by proxy, because we have clients that are in the financial services industry, in healthcare and these other highly regulated industries. And so it is those clients' responsibility to audit all of their suppliers and a law firm counts as a supplier. So we're audited by proxy.

So we are ISO certified across our entire data centers. We pass all kinds of certifications because we have a fairly large information security team. And there are people on that team that all they do all day, every day is respond to audits.

When you're a large law firm, you have you have to have these things called ethical walls in place where you might be two attorneys working for the same client, but because of the different matters or projects that you're working on cases for the client, attorney A might have to be completely walled off from everything attorney B is doing because of the nature of those two types of work, even for the same client. And so there are these things, ethical wall systems that are built throughout all of our technology and those sorts of things also have to apply to any of the AI tools.

There are certain practice areas that are just thirsty for it because they, the type of work they do just lends itself to being able to be done better, faster, cheaper through the use of AI. And so those, they tend to come to us as far as how do we champion it across the board. That's where Christine, my boss comes in. So she is the director of innovation and her mission is to help the attorneys understand how to better leverage technology and how to innovate. And so she spends most of her time meeting with the attorneys and trying to understand the struggles they're dealing with.

It is, part of it is you're just, you're listening to what they're trying to accomplish. You're listening to the problems they're trying to address when they bring it to you. And the other bit is you, when you have successes, you celebrate those successes to the firm. And when attorney A sees that attorney B has done something cool and is doing it in a way that is saving time and money and the clients are happy with the result, they're getting a better result sooner than they thought for less than they thought, it spreads.

Package development playbook

So two things. One is Jenny Bryan and Hadley Wickham 's book. I don't know if you're an R or Python developer, but if you're an R developer, their book on packages, on creating packages, it is the guide that I use.

What I used to do is I made some, I had a package generator package that I would use that would make all the scaffolding for what I normally use. And you can use, is it use this? Is it still use this? Use this is a great package for sort of standing up package scaffolding.

But lately I just have made, I created a skill in Claude and I fed it Hadley and Jenny's book, which is freely available on the internet. And along with my own sort of rules around when to use DuckDB and when to use SQL, because I tend to use DuckDB when I'm doing dev and then switch to Microsoft SQL for production because our database, primarily our databases are Microsoft SQL. So I have that skill. And so I will just, I'll invoke that skill in Claude and it will generate a zip file with all of the scaffolding I need that I just unzip into a new directory and Posit Workbench and start from there.

I'm a big proponent of test-driven development. And so everything I do starts with tests. I'll write the tests and then write the code of the tests.

My day is normally 20% meetings and 80% ad hoc calls with folks and coding and a lot of package development. Everything starts with a package.

I highly recommend you do that because, you know, if you assume that, if you assume the work you're going to be doing is reusable, then you're likely going to be doing the right work. Like if you're doing work that's not going to be reusable, I would ask, why are you doing it? You should think hard about it.

AI in music

My feeling is that the state of AI now is that it's very good at doing things that have been done before. And so, it can write music that sounds like music you've already heard, but I haven't heard it do music that sounds like music you haven't heard before. And that's one of the great things about musicians, is that... Innovative.

Yeah. And I think it's also going to drive a lot more in the way of live music, and I'm a huge proponent of live music. As someone who knows quite a few professional musicians, and one of my children is at school to become a professional musician, this hits close to home. I am concerned about it. The amount of copyright infringement and outright disregard for copyright is a huge issue.

But I think that it's helping people who wouldn't normally create music, create music. I'm not going to say it's making them great music, but if it brings them joy, and it moves them towards actually playing an instrument, or writing music that they love, I think it's great. But you can't take people's... You can't take food out of the mouths of artists to create things that just sound like that artist.

Career advice

I have ADHD. I wasn't diagnosed until I was in my 40s when my oldest son was diagnosed. And so my sort of the way it presented itself was in lack of a filter. And so if I thought I was right in a meeting, it didn't matter to me how you felt at the end of the meeting as long as I knew I was right. And I really struggled at a certain point in my career getting things done, like getting results.

And thank goodness, this wonderful developer that I worked with at McGuire Woods named Sherry. She sat me down and said, because I was complaining about she said, it's because you're not listening. Like it doesn't matter if you're right. It doesn't matter if you feel like you've got the best idea if you're not actually listening to what the other people have to say and actually considering it and taking a beat to think they're not going to care what they're not. They're not going to be with you in this.

So I would say that, you know, based on my experience with these hangouts, everybody that comes to these is they're the smartest people on the planet. So I would say, always make room to listen, no matter where you are in your career, take the time to listen to the other people in the room, because it can only help.

Another one is you need to be able to explain something in three sentences, no matter how technical it is. You need to be able to have an elevator pitch for an idea, an elevator pitch where you're in the elevator, you've got 20 seconds, maybe 15 seconds with the CEO who happened to come into the elevator and ask you what you're working on. You need to be able to explain the things that you're trying to do concisely and in a way that other people can understand.

Always make room to listen, no matter where you are in your career, take the time to listen to the other people in the room, because it can only help.

My pleasure. Long time caller. I've been in the chat on many of these and I am just so honored to be in the company of the other folks who have been on this. It's a wonderful community and I look forward to potentially hiring one of you. Take a look at our career page.