Free Form AI

Michael Berk (00:01.17)
Welcome to the first of many episodes of Freeform AI. My name is Michael and I do data engineering and machine learning at Databricks and I'm joined by my wonderful cohost.

Ben (00:10.126)
Hey, I'm Ben. I work as a software engineer at Databricks, working on MLflow and other open source projects. And lately these days, lots of design reviews and product suggestions.

Michael Berk (00:24.368)
and reviewing the performance of literally everyone in the company.

Ben (00:29.294)
Just like 20 or so people today. That's due at the end of today. So yeah, gotta get on that today.

Michael Berk (00:38.292)
Got it. Cool. so we're going to be kicking off this series. both Ben and I are very excited to be owning our own podcast and, we're hopefully going to give a lot of practical and useful tips along the way about the AI industry, career development, how to build shit, how to prioritize stuff, that type of thing. And we're going to kick it off with probably the most important thing you could ever learn in your entire life. And that is how to scope and prioritize.

So I work in professional services where I go and work with Databricks customers. And sometimes they don't know how to build things in the proper way. I'm not saying that I do or Databricks does, but typically there are better and worse ways to approach product development. And the key hallmark of this episode is going to be talking about the ROI or return on investment calculation when developing features and how you know when to cut it and run. How do you affect effectively prototype and gather signal, those types of things.

So starting off with the definition, Ben, how do you think about ROI from a feature development perspective?

Ben (01:46.558)
I think about it as like anything that you do, right? And we've talked about this, and I before about what is that single commodity that's so limited that affects everything that we do, which is time. And when you're talking about building a solution with software, whether it's ML or just straight up software, you're building an application or you're building some sort of framework, you...

have a team of humans that have a fixed amount of time. So how do you most effectively utilize that time and utilize your energy to build things and make sure that you're putting in the amount of time.

just the minimum amount of time needed in order to get a signal about whether you're on the right track. So an ROI estimate for me is for that amount of investment that you're doing, how quickly can you get that return to whether know that you're on the right track? You got something that that's tracking with people. They're like, yeah, this is cool. I want to use this. Or you're getting a signal of, you're

The initial version, it has promise, but there's like some pretty rough edges and that's a good signal. You know, you can go fix those things or is it, Hey, this, you nailed it. This is perfect. This is exactly what we wanted. And then that's going to follow on to other things. talk about this episode, like what the next phases are.

Michael Berk (03:22.964)
Exactly. Yeah. So you've, you've already touched on a bunch of deep topics, but I'll just set it up with a speech that I give all the fricking time when working with customers. gave it last week. and it's very simple as Ben alluded to, we have limited time on this earth. And frankly, I don't want to be spending my whole life working. I would rather be surfing or scuba diving or whatever. And I'm sure you, another employee at my company would also want to be doing something else. So

The key thing here is we want to use the minimum amount of time to develop something that is good enough. We're not looking for perfect. Some people really are passionate about getting everything airtight and that's important, but good enough is by definition, the highest ROI thing that you can do. And so there are two components, right? Return. And that is encompassed generally in value from that feature. And then investment, which is.

How much time does it take to build and maintain? So starting off with, I think the return is probably a more important thing to start with. Ben, let's say we're scoping a predictive model. we're predicting, we're predicting hot dog sales and, we have a cart throughout, or let's say 15 cars spread throughout Manhattan. And we want to be able to properly source our buns, ketchup, dogs, et cetera, and put them in the right place at the right time.

How would you go about scoping whether we should even do something like this? Or what are the different aspects of return?

Ben (04:59.16)
So the big thing with a project like that that could potentially be so insanely open-ended is to make sure that we have a person that's tasked with defining what an MVP is who has knowledge about the problem space. Firstly, secondly has knowledge about the development process. And thirdly has knowledge about the product process, which means when you're talking about product for

some solution you're going to build. That's how are people going to use this thing? Is it going to be an internal tool that people are going to use to place orders on things and make sure that we're doing logistics properly about where our carts are going to be stationed? Or if it's something that's external facing, like, we're using this as a marketing tool to let people know that we have this product at this location or

you know, automating that process or predicting where things are. People will be excited about the fact that there's a hot dog cart at their corner in New York city. You need to know like, what is that product? Because the product definition defines what your MVP is for one. And also who are the players that need to be involved in both the planning phase and in the execution phase.

Michael Berk (06:25.256)
Got it. So you mentioned excitement about like from customers that they have a hot dog cart on their corner. How would you go about quantifying that and assigning sort of a number to it?

Ben (06:36.888)
So before you get into the project green field phase, there's a difference between building something that you know there's a demand for, which you collect that information through data analysis. And you should have some means of getting that. Whether that's at the extreme end, it could be you're walking around outside asking people while holding a clipboard and writing down their answers or recording them with your phone to get data.

about like, people even care about hot dogs around here? And then the other side is you already have a data pipeline where you're collecting data because we run a company that does hot dog sales. have sales data. We know what people are buying at different locations on different days. And we can correlate that with weather data or, you know, times of the year events that happen in the city in order to make sure that our analysis incorporates all of those factors.

So it can range from those two extremes and anywhere in between or a mix of both. But you need that sort of information as your foundation to understand why are we building this? Is this going to potentially work? So it's defining the hypothesis that you're going to be doing in the planning phase of what do we need to actually scientifically test to validate that we're doing anything worthwhile? The worst thing that you can do is just

have a bunch of smart people sit in a room and think about like doing just raw brainstorming. Like, what could we do? We have all this time that we could work on a project. What project should we do? And people just theorize and they project their own bias into the project planning phase, which is super dangerous. That's how you build products that nobody cares about or things that just completely flop. Having data is absolutely critical to inform like

Michael Berk (08:30.91)
Yeah.

Michael Berk (08:35.081)
Yeah.

Ben (08:36.224)
Why are we doing this?

Michael Berk (08:38.622)
Yeah. And a really good way to start, if you are not sure how to start is just go embed yourself with a team. Typically, if you try to do a process, especially with fresh eyes, you see inefficiencies that people who have been very focused on a specific problem might not be able to see. I remember in the Adventures of Machine Learning podcast, we interviewed the Lyft development team and they have sort of a rotating specialist that goes and actually shadows teams and optimizes.

at least looks to optimize their processes with fancy ML. And they were using reinforcement learning implementation, I remember. But it's a really great way to go and actually see what the inefficiencies are because then you have a baseline. So to put it tangibly, I was working with a customer recently and they wanted to basically search over a bunch of PDFs and they were using OneDrive and local file system and a single exact match search for a substring took up to an hour.

Inefficient, right? And if you, yeah, just, just a tad. and if you know a more modern tech stack, you can get that down to milliseconds if you want. so just simply by embedding in their process, the ML product development team was able to see inefficiencies and find a need. So for our hot dog use case, talk to people, but if we already have analytics about demand data, just analyze that data, see if there's trends or inefficiencies.

Ben (09:38.576)
a little bit.

Michael Berk (10:06.204)
So cool, we've determined there's demand. What's next in the return calculation of our RO?

Ben (10:13.886)
Next is what, so we've defined why the next big thing is what, like what are we going to build? And that can be crowdsourced, you know, among the product and engineering cross-functional team, get some, some people from sales, interview them, get their feedback on what their thoughts are on.

Where can we drive business growth? What would we need to do in order to do that? And you'll amass a massive collection of documents from doing all of this stuff, from handling internal brainstorming, external brainstorming, interviews with stakeholders. And what you want to do is have somebody who's experienced, who's maybe been part of teams that have built products before, who has been able to see that process and understand.

So that you're going to get that lessons learned wisdom involved in this process. Theoretically, you should have the whole team in that case and maybe a couple of junior people who have never done it before so they can get that experience. But you need to now distill all of that chaos into a minimum viable product that actually meets the demands of the why are we doing this? And that should be a list of

things for MVP that are the least amount of things that are required to build in order to meet the, that first stage goal. And that first stage goal is we need to know whether we should pop smoke and bail from this. Like is it, was this a good idea? Was this something that was, you know, directionally correct or were we way off base here and we should.

minimize the amount of time spent on this so we could go work on something that could potentially work.

Michael Berk (12:16.18)
it. So I'm still not super crystal clear. So let's say we want to create a, all right, the demand problem is that our hot dog stands are throwing out a bunch of buns at the end of the day. And that is equating to some amount X number of dollars every week that we're losing that if we optimized our allocation of buns and theoretically do reduce our total spend, we'd be saving money.

What's the next step from what you just described?

Ben (12:48.366)
The first hypothesis is, the people who are setting up the carts every day, are they just taking too many buns from the warehouse? Do we put quotas on that? And that can be an analytics question that we can answer fairly quickly with just looking at the data. Like, oh, this cart manned by Joe, he always pulls like twice as many buns as he needs. Let's go talk to Joe. Like, dude, why are you doing this? And it's like, oh, I just don't want to run out.

Well, let's adjust your quota capacity here and make that just a system rule. So it's a heuristic that you put in. And if you can identify all of those people and all of those cards that are in these locations that are consistently doing this time and again, that's your simple solution right there. Or at least the first phase of something. If it becomes, well, there's no prediction that's accurate here.

or there's no consistency of a particular person or a location or a cart or any of those factors that are covariant with one another. Then you need to think about, okay, is this problem way more complex than we thought? Let's analyze our hypothesis during the analysis phase to see is there a signal here that we're missing? Something that post-hoc we can correlate back to a

a variable that existed that morning. That if that had been part of that person's selection process, I was like, oh, is it something like the weather forecast? So every time it's going to be like a super rainy day in New York, everybody pulls way too many buttons. Well, then now we're talking ML could help with that. If we analyze all the available data points that we can think of and we still find no correlation.

Maybe that's a very complex problem that has a lot of interactions within the data that we don't quite know. And I think from a profitability perspective, it's worse to run out of bonds than it is to have too many. So we would want to definitely pay attention to that negative signal. Like, Frank on cart number 37, you know, he's consistently running out of bonds. Do we need to set up a logistics network where

Ben (15:17.078)
Midway through the day, we restock these people.

Is there a way for them to like use an app because everybody's got a phone where they can just request this when they're at 20 % capacity and then we could just buy a van and ship buns out to the carts that need it.

Michael Berk (15:37.31)
Got it. So this is more in sort of this product research phase where we're trying to find root causes of inefficiencies. Cool. So it.

Ben (15:41.73)
Mm-hmm. Yes.

But theoretically, what we're talking about is we do have a signal through analytics that makes it so that we could optimize this at the beginning of the day when people are getting their raw materials for the carts, assuming we have sufficient storage to handle the highest demand ones that are out there.

Michael Berk (16:10.778)
So sounds like the next step would be sort of get deep into the problem, figure out potential root causes. And then often with an ML team, you'd be given requirements from a product, a specialist or something like that, just a stakeholder that's looking for this specific solution to be solved. And they're often bringing a lot of the return aspect of the calculation. So if it's really greenfield like this, it's a bit more challenging. You have to be more subject matter expertized, but often what we see is with larger companies.

Ben (16:27.224)
Mm-hmm.

Michael Berk (16:41.172)
They're just given a thing to do.

Ben (16:44.334)
Yeah, if they build that in a vacuum without going through this process, I can't even tell you how many times I've seen this blow up in their face. Like, it's happened to me before when I was that person who was just off in the corner. My director gave me a task to complete and say, hey, just figure this out. Then go and build some abomination over the course of like

you know, four to six weeks and then present it to that team and they look at it and they're like, yeah, this is cool and all. We already knew this, but the real problem is X and you solved for Z. Like that's, this is not helpful to us. You do that a couple of times in your career and you're just like, okay, I got to pivot and work some weekends right now in order to fix this and.

Michael Berk (17:28.83)
Yeah. And thinking.

Ben (17:39.732)
I should go and talk to people before I write a single line of code.

Michael Berk (17:43.57)
Yeah, no, that makes a ton of sense. Lack of communication between the stakeholder and the developer is an issue, to say the least.

Ben (17:54.683)
And lack of a project plan that's had other human eyes on it is super dangerous,

Michael Berk (18:01.128)
Right. Okay, cool. So let's say we have open communication with our hot dog stands and we're looking to figure out a array of solutions effectively.

question typically at Databricks, we leverage the Moscow method of must have, should have, could have, won't have, basically just create a bullet list of under each of those describing high level feature requirements. And as importantly, what is out of scope for this given solution? Where does that come in? Does that come in prior to this prototyping MVP phase or is it after?

Ben (18:37.742)
depends on what you're doing. So for what I work on these days, it's not ML. It's the tooling that allows you to do ML a little bit easier. we have a project plan that we've developed like an MVP, and we're talking more about individual features to support. if you, the way I do it at least,

I start off with a document that only my eyes will see, where I start thinking through what is the ideal end state for me if I put myself in the shoes of a user, what's all the cool things that a user would probably want to be able to do. And I've list all of those out and I put, I put that in that document and then I closed that document. I come back to it the next day and I read through that list and then

I start doing something that we just call sort order rank, is what are the, the, what are the features that I've listed out? And I'm not, when I'm doing this, I'm not listing out.

like anything about implementation details. It's not like, what's the complexity of implementing this? It's more just the features. What do I think are going to be the most used features and the most useful things to have in there? And you start putting that in that order of importance. So maybe you're talking about a CRUD API in order to interface with some data that you want to write. Well, what's the most critical thing in there? Well, probably the database.

You need like a table to be created or a series of tables. So we can't do anything without that. So that's pretty high up there. You also need the CRUD APIs. So we need the ability to just create those records, load those records. We might not need to update them, depends on what it is. Maybe that's not critical for the first part. And you might not need to delete them at first. So that aspect of that CRUD API,

Ben (20:49.742)
can inform what is your minimum viable product. Typically with most APIs, you need all of those things. That's just the bare minimum. But there could be other things like, well, I need to search over the data. What is that pattern? Can I get away with just listing everything? How much data is there going to be? Do I need to introduce pagination to that? Because if not, users might be loading in like a billion rows of data to this over a

a short period of time. And when you hit that, that list API, like, no, the backend is going to freak out. The user is going to be sitting there waiting forever. You could get timeouts and all sorts of nasty stuff. So you have to understand that that data model or what is the nature of this product that we're building? How frequently is it going to be accessed? What does that aspect of it look like? So that'll start saying, okay, we need these features that are just core.

But then you start thinking about, or looking through that list and saying, well, it would be really nice if I had the ability to do this like advanced search or join this data to this other thing in order to, you know, give more information to the user. And you have to start making those hard decisions about is this still usable without that feature? Or does this become just dead on arrival if I don't have this? And that's where you start getting to that Moscow.

you know, approach for the MVP. Basically your MVP is only the must haves. You don't start looking at the should haves or could haves. And you definitely want to list out all of the won't haves. Cause it's stuff that, we don't, we don't have infinite time or infinite resources here. And if we start putting a bunch of time into those should haves or those could haves, like the cool ideas that we just don't know if they resonate.

There's this philosophy in like product engineering about when you release something, it's out there forever. it's really, it's really challenging to like, like you can build an API that does something and it provides some sort of value. And for your customer base, you can change that API.

Ben (23:19.95)
or rename it or modify its behavior in order to make that process a lot better. And we do that all the time. Hopefully not with too many breaking changes. We try not to rename stuff unless it really needs to be renamed. But you can change the functionality of that and nobody's going to have a problem with it. But if you just delete that API and people are using it, you just pissed off your customer base. Even if it's only like 15 users out of 10,000 that were actually using that.

The 15 people are going to be like, why did you delete this? This was super critical to like how I do what I do. And I don't want to use the work around. So it's very, very important that you don't overbuild. Cause you don't know what people are going to actually attach themselves to. So you want to limit the initial exposure to it and wait to have those 15 people that maybe are a hundred people contact you and request that feature. If nobody requests that.

that functionality, then don't build it. But if everybody requests that, then you know, yeah, that was in my could haves. We should go build that right now.

Michael Berk (24:29.82)
Yeah, that's super interesting. You say that, two things stick out. One is the data, the Databricks implementations of Lang chain integrations. I don't know how many there are, but I think there's Lang chain underscore community Databricks, Lang chain, Lang chain Databricks. And, it's just really annoying. Like put simply, like there should be one PIP install since origin. should ideally design it perfectly from the outset. I know that's very hard. but now there are.

tutorials spread out through the internet with different installs and imports. And so it's just frustrating.

Ben (25:08.76)
That really hits close to home because I'm the reason that two of those exist. So that comes to the evolution of a product over time that is shipping an actual organization structure. So that first version that was made as like a community thing, that was the only means of getting code out at that time. And that was like how that

Michael Berk (25:12.622)
Ben (25:38.35)
that organization did it. So that's what we built in there. And then they opened up a new flow about, I think it was like nine months later. And they're like, actually, we recommend that you do like your own hosted version of this that's in our repo, but you control the code. And we're like, yes, sign us up for that for stability reasons. And all of sudden our integration became way more stable because we were the ones controlling that. And we had CI set up on our own and to check for breaking changes.

And then there was an initiative that we had for, well, we're gonna take full ownership of this because we're gonna be doing a unified access point to all things Gen.ai at Databricks, which is.

That initiative, what you mentioned is the first package that's in there. There's now like Unity catalog AI is coming out with that. But that's to simplify the process for our customers to provide a single access point. But yeah, those old code bases, they still live around and they have to because people are still using them.

Michael Berk (26:42.046)
And it's really per like pervasive and problematic. Like GPT was trained on those old repos. So basically all of GPT's Databricks generated code is now wrong and will be for the foreseeable future. So just targeted question to you, how would you go about resolving all of those legacy decisions that are no longer valid?

Ben (27:03.874)
You need to do community outreach. need to do marketing and you need to do code deprecation, which we do. Like we, if you use those packages, you get a warning. that's really all you can do until the true maintainer of the legacy code, which is not us. If it were us, we just delete it. and feel like, sorry, this doesn't exist anymore. you'd warn people for like six months.

and make it kind of annoying to use that package until people are like, yeah, this is being replaced by this other thing. Let me just update my code. But if you just delete it without warning, that's not good. You're going to get a bunch of support tickets, a bunch of people complaining, and you're going to lose a bunch of your audience. So you have to, it's a balancing act. But eventually you want to get to like the North Star, which is, okay, we have this consistent package that

we can modify freely as we need to in order to provide additional feature support and functionality for users where they don't have to think about all of the old ancient stuff. So it happens. And it's more pervasive now. I think it's more jarring now because everybody's been so used to over the last probably 10 years in the Python ecosystem of these really stable, consistent packages that aren't

Michael Berk (28:10.729)
Yeah.

Ben (28:27.65)
doing weird stuff like this, like breaking APIs and changing namespaces. But this is the next gold rush in like software, which is Gen.ai. And whenever you move really, really fast and have a lot of people playing in the playground, things like this happen. And it happened during the dot-com bubble era in the late 90s, early 2000s.

Michael Berk (28:30.654)
Yeah.

Ben (28:53.538)
Like you had so much inconsistency standards being violated, like weird things happening. Things were unstable. Everybody was using like a new, new means of building websites and new frameworks that they were using. There's this huge explosion of tech and the strong survives. The rest got left behind in the dust. So right now every major tech player is trying to put their cards on the table and try to come out to be one of the ones that comes out on top.

Michael Berk (29:08.052)
Mm-hmm.

Ben (29:23.534)
which means you move fast, you break stuff, and sometimes there's a lot of chaos.

Michael Berk (29:29.438)
Yeah, it's just like how much abuse will a customer take until they turn? I love toeing the line. But yeah, it makes a ton of sense. And it's really hard. Like fundamentally, it's iterative development. And getting it right the first time is super challenging. So.

Ben (29:45.794)
Yeah. And those packages that you mentioned, we did exactly that MVP process. Like the first couple of versions that were released didn't do everything that people wanted. But if we had set out from the get-go to be like, Hey, let's build the gold standard for this integration. Not only would it have taken four times longer to, and all of those engineering resources to build that, but I can promise you that if we had built that original design doc as it was written.

and doing all the should haves and could haves. More than half those APIs would not be used. They'd still be there right now. And people would be like, why did we build all these? Like there would have been custom things specifically for just rag use cases and just focused on that. Cause that was what was hot at the time. And now people have moved it on to agents. So by limiting the design and limiting that implementation to make it more

abstract, like, you can go and build all this stuff with the framework that this is integrated with. We're just providing the plugins that you need in order to do that. And we're not going to build our own abstraction layer over all of this stuff to, because that introduces complexity and confusion to users. Even if for one particular use case, it becomes super simple to do. That's why we do it the way we do.

Michael Berk (31:14.974)
So going back to our use case of predicting hot dog stuff, we have, let's say we've built this MVP, but actually, no, I would like to go a little bit deeper into a very common problem that I see when working with customers, which is they don't have a tech lead such as yourself. And they're a small scrappy analytics background, data engineering background, maybe some traditional ML. And they haven't built the gambit of like the, that

Ben (31:19.116)
Mm-hmm.

Michael Berk (31:44.414)
would allow you to have the answers to all of these questions or at least a better approach and they need to iterate really efficiently. So under that assumption that our team is not tremendously technical, they have some software background, some ML background, they can generally code. How do you advise iterating towards predicting those hot dog ingredients?

Ben (32:08.046)
Start small. So an MVP should be not just don't build all the halo features that should be part of it, but it's what's the simplest thing that you can get out there that is not shipping a tech stack. So is that first iteration just a basis on analytics? Like, Hey, every night we're going to pull the data from the last two weeks. We're going to analyze that statistically.

And we're going to provide a projection of what we think based on the data that we have. That could be your first thing that you ship. There's no model there. It's heuristics based. It's simple. It's easy to understand. But what you're trying to ship is not tech. Cause your end users don't care how you solve the problem. They really don't. They can't see your code. They don't care about your code. They can't read your code even if they had access to it.

What they care is, this solving my problem of not pulling too many hot dogs off the shelf and putting it on my truck? So start with seeing, does this system that we're going to build this product, is there a way to get signal about, do people even use it? Is it even directionally useful to them? Sit down with them as they use it and see like, what do you like about this? What don't you like? And you could

Like sometimes that MVP that you build when people sit down and use it, they're just like, yeah, this is exactly what I need. And you might feel like crap when you're like, this was, it was like four SQL queries. This was super simple to build. took us like an afternoon to build this. And I, I think people in the data science and ML world that I've talked to, and I used to be one of these people that thought this way.

you would sort of devalue the product that you were building if the implementation was super simple. You're like, this isn't that important. Or I didn't get to do something cool. All I did was data engineering stuff. Now, when I have these interactions with a customer who's evaluating something that was very trivial to implement, and they're like, this is amazing. This is exactly what we want. I'm like, mission success. This is awesome.

Ben (34:33.742)
because it's super easy to maintain, super easy to extend, and it solves the problem. That solving the problem is the number one most important factor.

Michael Berk (34:46.162)
Yeah, this, don't know if it could be a whole episode, but it probably could be a whole episode. two thoughts. The first is this is sort of the second part of my speech where I describe ROI. And then I'm like, when we think about the investment piece, if me clapping five times once a day will solve your problem, you would pay me the exact same amount of money as it would take to build, maintain, and run the solution. It doesn't matter what it is and insert like any like funny.

extreme example of like me clapping five times. It could be saying 10 hell marries or like running around the block. Like it doesn't matter what it is as long as the problem is solved. So as you alluded to for SQL queries that if it works, stakeholders really don't give a shit. and the annoying piece for people like us typically is typically we're nerds. Typically we like building cool stuff. want to use the latest and greatest

And sort of test our skills a lot. Like often I want to see if I can build a sufficiently complex pipeline that I don't know, automates the solution. so it's really important to be aware of that as a force when you're making these decisions, because all the stakeholder cares about is you solving the problem. And if you can do it in the easiest way possible, it might be a little bit less fun, but it's actually really high ROI because the business gets the simplest solution that meets the need. And so just be aware of that. sometimes.

say, fuck it and build the right, like the really cool thing. I've done that a few times, but it will bite you in the ass. I promise.

Ben (36:20.238)
I haven't done that in years now. The whole fuck it, this is gonna be something cool that I'm gonna build. I've done, well I can't say that specifically, I've done halo features on things that weren't in the MVP scope because I've done my own bug bashing. Like I've tried to use something that I built. Most recent one was with like Unity catalog AI where...

our initial MVP was basically you write SQL and embed Python like a Python body, everything but the signature that had like the deaf name, you know, arguments. You would just write the Python body within that. And while I was testing it, me and Serena, who both co- you know, built this solution and we were both testing this and I was like, hey, Serena, I

I tried this with like this massive Python function and I don't know, maybe I'm dumb, writing Python code within like a wrap string is super painful. for a really stupid reason, indentation, I kept on messing up cause the dev loop for that. And that's what I was estimating is like,

Michael Berk (37:34.494)
No kidding.

Ben (37:46.582)
If I want to write like a 500 line Python function and it's wrapped in triple quotes as an, as part of a SQL body, have to be really careful about how many times I hit the space bar and I was doing like nested functions and stuff. So it was pretty complex. Cause I was trying to emulate like, what would a user actually do with a sufficiently complicated function here? And the dev loop for that was write it and then.

take that SQL body and attempt to create a function in Unity catalog with it, and then deal with the Unity catalog validator about the basically Python syntax. But you'd only get that after trying to execute it. So it would, it would write the function, whatever that body is, it's just a string. It doesn't care. But when you execute it, that's when it's like, oh, Python syntax error on line 731. I'm like, what line is that?

Cause now I'm, I'm in a wrap string. So I don't have anything that I would have in like a notebook or within an IDE. So like this, this sucks so bad. Like I hate this. and after wasting an entire six hours, I was like trying to get one thing working. I was like, you know what? Screw it. I'm going to the should have, I'm building like the callable extractor. So I built that and then.

Michael Berk (38:55.54)
You

Michael Berk (39:09.661)
interesting.

Ben (39:13.23)
You know, we started testing it out. We're like, okay, dev loop is now seconds because either my Python function executes locally and I can just tell if it works or, you know, I just got to make, you know, do some fixes and change it. And I know that it's going to execute. Theoretically it should execute. Uh, it does now, but I can write that and it'll create that SQL body for me. So it's.

Sometimes you have to build that stuff that even in pre-planning, you're like, it'll be fine. Users can just do this because all of our examples were hello world examples. Like, Hey, my function is, has two arguments X and Y. They're both integers return X plus Y. Anybody can write that super quick, but it's more like, what is a user actually going to do here? Well, I want to search Google.

And I want to get the search results and I want to parse them and I want to do all these things within this function. I want to do the keyword extraction or I want to pull metadata out of the request or I want to hit like the maps API and get coordinate information. You got to write a shitload of Python to do that. And embedding that in a triple wrap string. Good luck, man. So sometimes you got to do that. And I was, I happened to be right that case, but that implementation.

uh, within that core client, that was a geek out session for me of like figuring out like, how do I extract all of this information from a Python definition? How do I get this from a doc string? I had no idea. I've never done any of that stuff before. It's going into like Python source code and seeing like, how does AST do this? Like how does that whole system work? How does Python actually understand scripted text in order to execute it within its engine?

Michael Berk (40:56.873)
Yeah.

Ben (41:04.376)
And you end up learning a lot of stuff and then, yeah, did some geek out moments there.

Michael Berk (41:09.556)
Yeah, that's super cool. I have a similar user journey, but I'll skip it for a more important question. So this speech that I give routinely is, what the hell is ROI? How to think about it.

Michael Berk (41:25.488)
And I think it's missing one really key component, which is a really scary anecdote that gets everybody on board because sometimes I come into the room and I'm like, all right, this is ROI. This is how Databricks does it. this is how you sort of iterate and it's like a fun way to do it. And then you'll ship good features. It'll be really awesome and amazing. But the flip side of that is if you don't do it, you'll ship horrible code that will sink thousands, not millions of dollars.

And I was wondering if you had a horror story or two that I can steal and then tell to all my customers.

Ben (42:02.564)
man, I've, it's been years, but there were some stuff I did early on in my, my career that I'm in now. cause before this, used to do other stuff that had nothing to do with software, but yeah, there, some of my early stage projects were so convoluted and like overly complex that

I learned a lot from them because sometimes I built some stuff that people depended on. And I think the worst place that you can put yourself in as somebody who builds software is building something that everybody loves that you can't maintain. That's the worst. And what I mean by that is I had no idea how to write software. So I had this

I remember it was one monolithic file that I wrote in SAS and one of their desktop tools that they had called JMP. It's like a SAS Institute software. And I didn't know anything about abstraction or I had functions and stuff written in that. But.

Michael Berk (43:13.076)
Jump baby, yeah.

Ben (43:25.132)
I didn't know that it was bad to have all your code in like one single notebook effectively. So this, this single notebook was like, it was over 80,000 lines of code in this one notebook and zero tests. like nothing was validated, but everything depended on previous iterations of this execution flow. And it was all based on data and there were

I there's like over 30 models that were constructed in that entire chain. Most of them like tree based models. And if anybody has ever tried to debug something like that, where effectively you have your input data, you know what it is, and then you get your output and the internals of that is just a wall of code that has internal state that you have no idea.

what's going on. So to debug something like that, you go in there and you put a bunch of print statements. I'm like print state of this and then print state of this. And you're just guessing where this thing would be. So I would go in and start doing the, you know, the young dumb engineer process of just putting code comments everywhere in the script. like, if I see this issue, then

I should put a print statement in here and I have the print statement and like the evaluation that I would need to do comment it out. So I, I knew at a point of this project going on that I kind of knew like where in the code I needed to go to validate if something went wrong. That's all in my head and in my, my stupid notes that I had in there. And when we talked about it internally within, when the team grew a little bit bigger,

people are like, this is a super cool project and super cool data. It's so valuable.

Ben (45:28.398)
How do we like understand what's going on here? I'm like, oh, I have comments in the code and like, yeah, this is going to take me like a month to understand because there's a lot of stuff in here. I'm like, oh, just ask me questions or I'll write a doc that explains it, which is another like stupid answer. And then we hired somebody who had previous software engineering experience onto the team. Uh, and I really wanted this person so that I could learn from them. And he looked at it he's like, I remember him just staring at me for

almost a whole minute without saying anything. And then just like look back at his computer screen, scroll down a couple more pages. And he's like, did you really think that this is the way to do this? I was like, well, how should we do it? He's like, I'll bring in a book for you tomorrow. And he didn't bring one, he brought like four. They were like fundamentals of software engineering and like abstraction patterns and like how to do object oriented coding.

And six months later, he and I worked together on building something that was actually maintainable written in a proper language. And, know, I learned a lot from that, but it's so scary when you have this pipeline that there are hundreds of people depending on this every single day to keep like the business going. Not that it was like, if it, if it shut down the factory, it wasn't going to shut down. was more like this would inconvenience so many people.

they would have to go and manually do this stuff that they become dependent on that people would end up working like 18 hour days because they'd have to go and fetch all this data and make their own logical conclusions based on what the data is saying. So yeah, that taught me an important lesson about how do you go through that process of MV2P development while hand in hand making sure that you're writing code that's maintainable and testable.

Michael Berk (47:23.292)
If you did it the proper way, what would have been the benefits versus the way you did it?

Ben (47:28.782)
I wouldn't have had to spend probably half of my working time. Like six days a week. I work six days a week at that place. That's sucked. But half of my time was spent just debugging and keeping this thing running. Some data source changed on one of the 800 data sets that this thing ingested. And they changed the name of a column or they migrated the data.

to like a new table and you have to go and like figure out why this thing broke and then go and look in the database and check a bunch of stuff. Or there's some weird edge case where one of the models that I was building, if the, if the state of the data changed in such a way because the product line changed at the time, then it would render this model to just generate the same value for everything, which would break downstream stuff. And it's really hard to

to diagnose when something like that would happen.

Michael Berk (48:31.886)
question if you had followed the Moscow method with ROI calculations for each line item what would have not been built like what percent of that code would have not been built

Ben (48:43.758)
So that project, I accidentally used that method without knowing that method. Cause it started super simple. It was one model. It was like three datasets. And it was solving like one particular thing with a yield analysis problem. But it was so successful that everybody was like, Hey, could you build this in there? This is super important. Or, Hey, we have this new product line that needs to evaluate this. So it started to just get more and more stuff packed onto it, which is fine.

Michael Berk (48:48.796)
Nice.

Ben (49:13.72)
provided that you have a maintainable foundation to work off of. what you don't want to do is ship your hackathon project to prod effectively, which is what I was doing, and then not refactor it at all and just keep on patching more and more junk onto it.

Michael Berk (49:34.428)
Okay, cool. I'll say that I did that as well. Thank you.

Ben (49:38.988)
I think everybody's done stuff like that at some point, unless you're coming straight out of an undergrad and working directly embedded in a professional software engineering organization, then you're going to learn from the start, this is the way that we do things. But if you have no supervision, yeah, human creativity and innovation will build things that you regret.

Michael Berk (49:41.618)
Yeah.

Michael Berk (49:55.828)
Yeah, what if you've, yeah.

Michael Berk (50:05.788)
Yeah. Cool. I think that's a nice ending point. Let me summarize. so we sort of talked about how to build features specifically, how to think about a return on investment or ROI calculation. The return is the value return from the feature that you've built. The investment is the time, money, et cetera, that it takes to build and maintain a feature. So the steps that we outlined is first determined demand.

If it's Greenfield, go ask people, do research, see if people actually want it. And if there's existing adjacent use cases, just query that data and do some analysis. And if it is Greenfield, a really valuable way to approach this. And even if it's not exactly Greenfield, go shadow a team, embed yourself, try to do what they're doing day to day with fresh eyes and scratch your head when stuff is annoying or doesn't make sense or is just suboptimal.

Step two, once you see these inefficiencies and the demand, try to look for root causes. Once you have some root causes, then you can go about resolving those root causes. So step three from those root cause sort of explorations is create critical user journeys or CUJs from a stakeholder perspective. And you can sort of see what the different ways that stakeholders would be leveraging a theoretical solution.

And from here, you can develop a Moscow method of must have, should have, could have, and won't have. Once you have this research done, once you have a product spec done, then you can go out and build an MVP. And that's just implementing the simplest solutions for the must haves starting small. And then finally go and get feedback and just iterate from there.

So some general tips additionally beyond this is time is the commodity. Look to optimize that. ensure that there's communication between developer and stakeholder. If there isn't, you'll build in a silo and that's dangerous. once you release something, it's out there. So air on the side of underbuilding and don't expose a bunch of high level APIs and lifts you. You really want to be doing that and you know, it will continue to be supported. And then finally be aware of the urge of building something cool, despite a set of four SQL queries, perfectly solving the solution.

Ben (52:22.36)
Yes. You can do cool internally. So you can get clever no matter what the problem is, even if it's like SQL, right? Maybe you do like an elegant CTE implementation of the SQL where it's like, this is just like sexy SQL code. It's like really easy to read. It's all modularized and yeah, like.

Michael Berk (52:39.028)
You

Michael Berk (52:43.988)
Sexy sequel, that's an oxymoron and a half.

Ben (52:48.224)
I've created some, I've seen some, I've learned from some really good people about like the best way to write maintainable and readable SQL. Like that's a single statement is like 20,000 lines of SQL. If you write that as like an abomination, yeah, nobody can read it and understand it. Even Gen.ai would be like, I don't understand. But you can write that same query in such a way that's like,

Michael Berk (52:51.29)
man. Alright.

Ben (53:17.208)
hang on a second, I can follow exactly what's going on here. So you can apply cool things and like internal challenges to push yourself to excellence internally, even in something that's not like, I'm going to build a super cool torch solution that's going to solve this problem.

You can find complexity where you want to and make sure that you're challenging yourself in whatever you do.

Michael Berk (53:47.208)
Heard? Yeah, and after you've built enough shit, the simple stuff is actually more rewarding and satisfying. But there's always a frontier.

Ben (53:55.83)
Yeah, definitely because that means you're you're on to the next problem that you can solve faster

Michael Berk (54:01.2)
Exactly. Cool. So anything else before we wrap?

Ben (54:06.208)
No, it's a good summary,

Michael Berk (54:07.892)
Cool. All right, well, until next time, it's been Michael Burke and my co-host and have a good day, everyone.

Ben (54:11.874)
Ben Wilson, we'll catch you next time.