I am so excited to share that I’ve been elected to the Kubeflow Steering Committee! 🎉

Kubeflow

Kubeflow is an open-source platform that makes artificial intelligence and machine learning simple, portable, and scalable that was originally introduced by Google. Kubeflow is an ecosystem of Kubernetes based components for each stage in the AI/ML Lifecycle with support for best-in-class open source tools and frameworks.

Thank you

I am grateful to the friends, colleagues, mentors, and the Kubeflow community for placing their trust in me. I want to especially thank those that shared such kind words: Yuan Tang, Willem Pienaar, Max Mitchell, Matt Green, Matteo Mortari, Jesse Collins, Eder Ignatowicz, Hao Xu, Christian Zaccaria, Anish Asthana, Edson Tirelli, Shuchu Han, Ross Briden, Antonin Stefanutti, Mark Campbell, Varsha Prasad, and Josh Bottum.

Kubeflow is a vital part of the AI/ML ecosystem, and I’m honored to contribute to its growth and success.

Goals

We are at a critical moment in time for AI/ML. Open-source models have taken center stage, and resilient, scalable infrastructure has become critical to the success of AI. However, infrastructure alone is not enough—tools must inspire excitement and engagement among users.

As a member of the Steering Committee, I will focus on several key items:

1. Expanding Kubeflow Adoption: increasing awareness, adoption, and engagement through collaborations with enterprises and startups.

2. Making RAG a first class priority: the AI community has settled on Retrieval Augmented Generation (RAG) as a critical component for production AI. I will work to make this a new focus area for Kubeflow.

3. Engaging and Empowering AI Engineers: supporting software engineers integrating AI/ML into their products by providing intuitive workflows and intelligent defaults.

4. Improving the User/Developer Experience for Model Developers/Users: simplifying workflows and enhancing usability to reduce barriers to entry.

5. Building a more Feature-Complete AI/ML Platform: driving development of critical features to support end-to-end AI/ML workflows.

6. Improving Documentation: ensuring comprehensive, user-friendly docs and resources for the community.

7. Strengthening the Community: fostering a vibrant, inclusive contributor and user ecosystem to support Kubeflow’s long-term growth.

Kubeflow is already a robust and scalable platform, but expanding its relevance and audience is essential. By focusing on AI Engineers and Data Scientists, we can ensure Kubeflow not only remains at the forefront of AI/ML innovation but also inspires excitement and engagement among the broader community.

Why this Matters

The AI/ML landscape is evolving rapidly, with open-source tools and scalable infrastructure taking center stage. I believe Kubeflow is at the forefront of this transformation, and together, we can help build the future of production AI using open-source, resilient, and enterprise-grade software.

Join the Fun

If you're building in AI and want to learn about Kubeflow, let’s connect! If you’re not familiar with Kubeflow, now is the perfect time to see how we can help you scale your AI/ML products. Come join our Slack community, a community call, or feel free to reach out to me!

Thank you again to the Kubeflow community for this incredible opportunity. It’s time to build! 🚀

With gratitude,

Francisco 🤠

Since the outbreak of the industrial revolution, human society has undergone four rounds of technological revolution, and each technological change can be regarded as the deepening of automation technology. The conflict and subsequent rebalancing of efficiency and employment are constantly being repeated in the process of replacing people with machines. When people realize the new wave of human economic and social development that is created by advanced technological innovation, they must also accept the “creative destruction” brought by the iterative renewal of new technologies.

-Shen and Zhang, The Impact of Artificial Intelligence on Employment: the Role of Virtual Agglomeration.

The Emergence

The end of software is upon us.

The rise of the AI Agent—artificially intelligent autonomous software that acts on your behalf with limited intervention—has dawned a new era of “Software 2.0” (coined by the legendary Andrey Karpathy).

Some believe that this is the end of software. That belief is foolish.

Software 1.0—code forged from the tedious review of endless documentation—is dead, indeed.

But we are not at Software 2.0 according to Karpathy’s definition. Instead, we’re at something like Software 1.9—AI boosted software with humans in the loop1.

And my view is that Software 2.0 may not come for a long time.

Not because we won’t be able to do it, we are closer everyday but because how people think about software agents today is wrong.

AI Agents

AI Agents emerged from people tinkering with LLMs to automate tasks. Simon Wilson was one of the more notable software engineers publicly building in this space. Eventually startups like Devin from Cognition came into the picture. AI Agents became appealing to investors and it has become one of the most exciting investment areas.

A major appeal of AI Agents is the expectation that it will significantly reduce the cost of building software. I think they will reduce the cost of software but, counterintuitively, I believe they will also make software much more complex.

That’s a good thing because we need more software in the world and we also need to migrate a lot of old software (e.g., payments). As it turns out, much of what we take for granted and assume is simple is actually quite complex (again, payments).

Even if AI Agents get better there’s still an accountability problem. It turns out humans like having a person or legal entity they can hold accountable when things don’t work. More practically, using generative models to write all of your code without any supervision could result in frequent breaking changes of APIs (i.e., instability of your product).

People want reliable software.

Said another way, we want deterministic behavior in our software and generative models are inherently probabilistic.

I’m sure there are ways we can modify the context or prompt to say “don’t refactor the code like crazy all of the time and introduce breaking changes”, but that won’t address the accountability problem and I’m skeptical that the reliability piece can be completely solved without a human in the loop.

Software engineers will become more impactful but they will have to expand their knowledge and skillset quite a bit more…the good news is that LLMs are great tools for this. So Software 1.9 has already supercharged engineers that use AI.

“LLMs as a Copilot” seems to be the consensus opinion among most LLM users anyway, so I’m not saying anything terribly controversial or novel here.

The end state is that software engineers will become gatekeepers of the code while agents will become the actors.

How?

The Agent of Orchestration

People love chat bots.

Maybe it’s the lazy nature of “Tell me about X” and suddenly you’re given a nicely summarized piece of content about some arbitrary topic, maybe it’s something else—probably it’s the preference for laziness.

The laziness implies that humans really like flexibility in the input—i.e., I give the computer a command (a prompt or request) and the computer does the thing.

So the thing is really important. And what we actually want is software that dynamically orchestrates operations probabilistically and executes operations deterministically. Simply, if a customer changes what they want, the computer should do something different but always do the same thing for the same input so long as you believe the customer wants that thing.

This means agents should be using existing software and that humans should be spending time making resilient and high-performing software. Probably humans shouldn’t use agents to write entire enterprise-grade applications (i.e., high stakes production software where money is on the line—hello Fintech!) without humans in the loop.

I should note that “high stakes production software” is a really important phrase. AI Agents can be autonomous proportional to the tolerance for errors. In less regulated or lower-stakes areas (i.e., not Fintech, Healthcare, Education, etc.), AI Agents will likely do a lot more.

Evals Are All you Need

AI is powered by data. To build powerful AI you need the ability to measure the performance of your model and actions on some set of data. This is called model evaluation, or “evals”.

Evals require data, obviously. Data is generated by a process that you are quantifying.

To quantify something correctly, you have to measure it in code and this is the most underappreciated thing in AI/ML—probably because it’s boring.

For Agents and AI to work more broadly, you need high quality data and that means you need high quality tracking systems2.

So, tracking systems and the data they produce are the foundational components of AI, without them no amount of math, LLMs, or other AI will make you successful.

To make this more concrete, let’s consider a chatbot (or code generator): the data is the actual words/tokens authored. For an AI Agent that orchestrates operations, this would be the sequence of events and the actions it took encoded systematically.

Let’s say you have a bunch of financial documents and you want to summarize them for a customer and let the customer ask questions about the documents.

There are approximately3 four things that you need to do:

1. Parse a bunch of data.

2. Extract the most relevant sections of the documents to use in Retrieval Augmented Generation (RAG).

   1. Note, it is terribly underappreciated that retrieving a set of documents for RAG reduces to a classification problem at the sentence/chunk and document level, but the community settled on cheap heuristics because it’s easy to compute cosine similarity and it works surprisingly well in practice. Still surprises me. 🤷‍♂️

3. Generate an initial summary of the documents.

4. Answer the questions with the best context (using the extracted documents).

We want (1) and (2) to be deterministic and (3) and (4) to be probabilistic. Since (3) and (4) are generative, again, they are probabilistic by definition.

An agent executing these four steps while interfacing with a human in a production application means that the process won’t actually occur sequentially. You probably will have to parse new documents, answer other questions, or have a human intervene (i.e., in the loop).

The key detail is that agents need to be able to deterministically parse old documents the same way while also handling new/unseen formats.

The best way to handle deterministic parsing is not to just throw the document at ChatGPT (that’s a good idea when you’re dealing with a cold start4 in general).

The actual solution is two-fold: (1) throw it into ChatGPT and cross your fingers and (2) use a pseudo-deterministic approach.

A pseudo-deterministic involves the agent first attempting to execute existing code. If the code fails, the agent then parses the data and generates new code to handle the new format. Next the agent would submit this updated code as a Pull Request to be reviewed or modified by a human—i.e., you need a human-in-the-loop. Once approved, the new code would be incorporated into the application.

You can think of this approach as self-evolving software with a human-in-the-loop and I believe this is the way that Agents will prove to be the most impactful and build resilient software.

Data is the Foundation

Outside of infrastructure and compute, data is the competitive advantage for every major company in the AI race and that’s because ultimately those companies hold the proprietary information of an explicit goal and the operations needed to accomplish it—i.e., training data.

So, the biggest obstacle in everyone’s way of successfully launching production grade AI Agents is the ability to correctly measure the thing they’re trying to automate.

The Future of Agents

The benefit of starting narrow is that you’ll be able to build a training data moat that will create an actual competitive advantage.

So, builders should start small, invest in measuring every single step of the process that you’re automating, ensure you’re emitting high quality data, measure the accuracy of your agent, and then fine tune your model based on your proprietary data (if necessary).

Friday Harbor, Benchmark, and Cleric are great companies that are already building products this way.

Closing Thoughts

I’ve said before, the future will use more AI and software, not less. I am excited about a future that runs efficiently, solves novel problems, and that forges a brighter world for humanity.

Technology, with its many flaws, has helped reduce the suffering for billions of people and AI Agents will accelerate this by orders of magnitude.

The most exciting thing is that we are still so early and there’s a whole world to build.

Happy building!

-Francisco 🤠

Some Content Recommendations

interviewed the one and only Max Levchin, CEO of Affirm.

Alex Johnson wrote about the Future of the Financial Data Economy.

Simon Taylor shared the Top 10 Fintech Brainfood Rants of 2024.

Nik Milanović asked an important question: “Would you let an AI agent make payments for you?” The answer will be yes, with a human (yourself) in the loop.

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

I generated the podcast above using Google’s NotebookLM. Give it a listen.

Here’s the summary provided by NotebookLM:

This podcast explores the use of machine learning in finance, particularly within the domain of Fintech and lending. It examines the challenges and opportunities associated with applying machine learning to complex financial problems, including the need for robust data infrastructure and careful consideration of regulatory constraints. It also emphasizes the importance of open source software in advancing the field of AI and lending.

You can also access it directly from Google.

I fed NotebookLM seven articles I wrote about Fintech, AI, and data to generate it.

I did not do anything else to generate it.

I am quite blown away. The 10 minutes are entertaining, pointed, and outline much of what I was trying to say in my content (but it is much more engaging). There was some hallucination but, overall, it is just another example of how incredible AI applications are becoming.

Hope you enjoy it as much as I did!

Happy Generating!

-Francisco

Post Script

The views expressed here are those of the author and the author alone, they do not reflect the views of his employer or previous employers.

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

Editors Note— Today’s essay was written in collaboration with . Alex is a Fintech legend and the founder of Fintech Takes, a media brand focused on the intersection of financial services, technology, and public policy. Prior to founding Fintech Takes, Alex spent 20+ years in various marketing, strategy, and market research roles at companies, including Cornerstone Advisors, FICO, Mercator Advisory Group, and Zoot Enterprises.

The History of FICO and Credit Risk

Credit risk decisioning, as we think of it today, got its start in the 1950s, when mathematicians and engineers started looking for commercial applications for the theories and technologies that had, for the prior two decades, been primarily employed by the military.

Early pioneers like William Fair and Earl Isaac (the co-founders of FICO) saw an opportunity to apply information theory and statistical analysis to the process of deciding who to give loans to, a process that had been entirely within the realm of human judgment and intuition up until that point.

It’s lucky that they did.

Lending is a fantastic field to apply statistical analysis techniques to because the process of determining whether someone should be approved for a loan or not is a process bound by constraints.

The primary output of any credit risk model is always binary — predicting if a borrower will default on their loan obligation or not.

And many of the inputs into credit risk models have become bound by constraints as well. While Bill Fair and Earl Isaac didn’t have to deal with regulations like the Fair Credit Reporting Act (FCRA) and the Equal Credit Opportunity Act (ECOA) when they first founded FICO in 1956, by the time they (and the three national credit bureaus — Equifax, Experian, and TransUnion) were ready to commercialize the first general-purpose credit score (the FICO Score) in 1989, the restrictions imposed by the FCRA and ECOA were well-established.

These constraints have created a narrow search space for lenders to innovate within over the last 60 years. Over that time, clever statisticians, data scientists, and machine learning engineers have discovered a lot of what does and doesn’t work in credit risk decisioning, which has left the lending industry with a very mature and well-tested set of tools and best practices for determining who should and shouldn’t get a loan.

This is generally a very good thing — we know what works! — but it also tends to lead to rigid thinking.

When you talk to experienced credit risk professionals, you often hear some version of “that’ll never work” or “that’s not how we do things.”

This rigidity is understandable, but it misses a key point — constraints change.

Laws and regulations (and how they’re interpreted and enforced) change. Technology continues to advance up and to the right, unlocking capabilities that would have astonished messieurs Fair and Isaac.

It’s not right to say that there’s no room in the field of credit risk decisioning for innovation.

The right way to think about it is that there’s always room for innovation, provided that it’s done with sufficient knowledge of and respect for the lessons we’ve learned over the last 60 years.

So, our goal in this essay is to describe the process of developing and operationalizing credit risk models, explain how that process has been shaped (and constrained) by different regulations and technologies over the last six decades, and discuss where new technologies and regulatory developments are unlocking opportunities to improve the quality of our decisions.    

Developing a Credit Risk Model

AI is all the rage these days, in financial services and outside of it. However, it’s important to remember that lenders have been applying specific techniques and technologies within the field of artificial intelligence (machine learning, most prominently) to the question of who to lend money to for decades. 

AI/ML can help lenders save themselves money (through increased efficiency) and save their customers money (through better pricing of risk). The big question is, “how?”

Lenders usually start with credit bureau data. The virtue of this data is that it is A.) highly predictive, B.) FCRA-compliant, and C.) old enough and deep enough that lenders can backtest their models against it. 

You start by purchasing a dataset from one of the bureaus and turning that into something that can be used for machine learning.

A critical step in the model development process is precisely defining your outcome (in the context of lending, this is usually delinquency or default) because it will ultimately dictate the performance of your portfolio.

You define it by making an explicit value statement about time. Concretely, this looks something like, “I want to predict the likelihood of default within the first 3 months.” 

Whether or not you want it to be, the choice has important consequences to your portfolio, so choose it carefully.

For simplicity’s sake, let's say you wanted to launch a 12-month installment loan to an underserved consumer market segment. You would start by getting historical data of other lenders’ loans from a credit bureau to build your model.

You will need to make sure you pull the data at two different time periods: (1) when the original application was made so you can use data that is relevant for underwriting (and so you don't have forward-looking data resulting in data leakage) and (2) 12 months later (or whatever time period is appropriate for you) to check if the consumer defaulted on their loan.

Then, you'd clean the dataset into something that can be used for machine learning.

Once you have your dataset, you can start to run different Logistic Regressions or other classification-based machine learning algorithms to find hidden patterns and relationships (i.e., non-linear functions and interaction terms).

There's a lot more to it, and you can expand on things in much more elegant ways to handle different phenomena, but for the sake of simplicity, this is essentially how it's done.

Evaluating your Model

Once you have a model for predicting default, you need to figure out if it is any good (or, more accurately, if it will be an improvement over the existing model).

This is important because we already have an extraordinarily good model for predicting default in consumer lending — the FICO Score. Building a better-performing general industry credit scoring model than FICO is essentially impossible (FICO has a 60+ year headstart on you, and they employ some very smart people), but you don’t need to build a model that outperforms FICO for the entire industry. You just need to build a model that outperforms FICO for your specific product and target customer segment. This is difficult but doable. Indeed, most large consumer lenders have built custom in-house default models superior to FICO for their specific use case and they continue to be a competitive advantage for them.

(Read Alex’s essay on the eventual end of the FICO Score for more details on this front.)      

So, how do you know if your model is an improvement (for your business) over the status quo?

It turns out that accuracy is an imperfect metric when you have a low default rate (or, more generally, when you have severe class imbalance). As an example, suppose you have a 5% default rate. That means 95% of your data did not default, so if your model predicted that no one defaulted, you'd still have 95% accuracy. 

Without proper adjustment, this behavior is actually very likely to occur in your model, so we tend to ignore the accuracy metric, and instead, we focus on the rank order separation of the model.

To measure that rank order separation, there are four metrics industry professionals typically look at: Precision, Recall, the Kolomogorov-Smirnov (KS) Test, and Gini/AUC. 

Gini/AUC has increasingly become the most popular. A good credit risk model is usually around 70% AUC / 40% Gini. You don’t need to understand what that means; just trust Francisco — the higher, the better.

An important point here is that any single metric is very crude, and a dataset can be pathologically constructed to break it, so while these metrics are helpful, there are cases where things can still misbehave even though they seem normal. So, make sure to validate your model with sound business judgment.

But, for argument’s sake, let’s say you’ve done that. You’ve built a model that is better at rank ordering risk of default for your 12-month installment loan product than any of your prior models or the general industry models available from FICO or the bureaus.

Yay!

Now, what do you do?  

Building a Compounding Analytic Advantage

Turns out, everything we’ve been discussing so far is the easy part.

The hard part is operationalizing our new model in a way that creates a compounding analytic advantage over time.

This is not a concept that William Fair and Earl Isaac would have grokked.

Back in their day, model development and model execution where two completely different processes, playing out in parallel.

Statisticians would pull data sets from the bureaus, look for hidden patterns and relationships within the data, develop features, and build a model. That model would then be deployed into production, where it would be called on to assist with a specific step (credit decisioning) within a much larger, human-powered workflow (relationship lending). That workflow would then produce outcome data (how did the loans perform?), which would be fed into the credit bureaus and aggregated into credit reports, which the statisticians would then draw on to start work on the next iteration of their model.

The process was slow — any use of computer systems for pulling applicant data or making credit decisions using statistical models happened in batch; there was no concept of real-time transactions — and reliant on industry-wide data aggregation (through the credit bureaus). There was little ability for individual firms to build what Francisco calls a “machine learning flywheel.”  

Fast forward to today, and the big difference that we see is that the lending products that customers acquire and the experiences that they have using and managing them are all entirely orchestrated by software. 

This allows for a much more tightly integrated and proprietary approach to model development, in which every interaction that the customer has with the product generates data, which can then be fed back in to improve the model, which can then be deployed to generate better, more personalized product experiences for the customer. And on and on and on.

(Read this article from Francisco for a deeper dive into the challenges and rewards of building a machine learning flywheel.) 

Now, to be clear — building this flywheel is not easy.

The foundation is a great product. None of this works if you don’t build a great digital lending product, and building a great digital lending product requires meticulous and thoughtful engineering, complemented by a deep understanding of the business.

But again, for argument’s sake, let’s say that you’ve succeeded in doing this. You built a great 12-month installment loan product for an underserved customer segment in the market.

How do you architect the systems around that product to create this compounding analytic advantage we’ve been talking about?       

It starts with the data.

Data

Machine learning relies on data. When making credit risk decisions, bad data will ruin your business, while good data (especially good data that only you have or understand) can help you outperform the market.

Historically, credit risk models relied on loan repayment data from the bureaus (which is just a customer’s repayment history on another lender’s portfolio). Some lenders would complement this data with proprietary internal data, which offered more valuable and granular insights, but in general, bureau data was the center of gravity for all credit risk models.

However, in today’s world, in which the lending products themselves can generate proprietary data continuously in real time, this isn’t necessarily true.

There are a few different examples of this in fintech, which are worth paying attention to.

The first is product design. By iterating on the parameters of the lending product (loan amount, repayment terms, pricing, etc.), lenders can create low-risk product structures that can quickly produce useful proprietary signals for determining the risk of default. This is exactly what BNPL providers like Affirm, Afterpay, and Klarna did with the Pay-in-Four BNPL product, which is why these providers have traditionally not relied on or contributed to the traditional credit bureaus.  

The second example is consumer-permissioned data. Fintech companies like Plaid, Finicity, and MX have helped to popularize and standardize the ability for consumers to share their financial data with authorized third parties. This ability (frequently referred to as open banking) is about to be codified by the CFPB’s Personal Financial Data Rights Rule, which is intended (among many other goals) to help improve consumer outcomes in lending via cash flow underwriting. It turns out that cash flow data (the patterns of in-flows and out-flows of money in consumers’ bank accounts) can result in a powerful set of features for predicting risk, which is why forward-thinking lenders are thinking very hard about ways to rearchitect their lending products in order to incentivize and easily facilitate consumer-permissioned data sharing.

(Read Alex’s essay on everything you ever wanted to know about cash flow underwriting for, well, everything you ever wanted to know.)  

The third and final example is embedded lending. By distributing lending products within the context and workflows of non-finance activities, lenders can significantly reduce adverse selection and can take advantage of proprietary data that the embedded channel provider has about its customers.

A word of warning — utilizing novel forms of data, even when used as a complement to traditional bureau data, can be hazardous if it’s not done with extreme caution. The reach of fair lending laws is longer than many in fintech might expect and can be applied in ways that might not seem logical or intuitive (e.g., disparate impact).

That said, many of these new types of data are being operationalized in ways that provide a compliant path forward (FCRA-compliant data and attributes are very common in the world of cash flow underwriting these days). And many others can still be utilized in non-FCRA contexts (fraud, marketing, etc.) that provide tremendous franchise value to lenders.    

Designing Credit Risk Machine Learning Systems

We’ve talked about how to develop a model and how to generate or acquire differentiated data that can strengthen that model, but we haven’t talked about how to integrate it into the systems used by the lender to make credit decisions.

So now we should talk about decision engines.

Just like you, risk models didn’t fall out of a coconut tree. They live in the context, terribly burdened by what has been. That context is public policy, regulatory requirements, and the business requirements of the line of business leaders that own the lending products that these risk models were built to support.

Lenders encode that context in a decision engine, which typically offers the ability to call third-party data sources and other internal services, run feature extraction, and execute model inference. 

From an engineering perspective, these decision engines play a central role in orchestrating the larger loan underwriting workflow, which is (as this graphic from Francisco aptly illustrates) often quite chaotic:

This chaos is, to a degree, expected (lending money is a complex business!), but it’s enormously beneficial to lenders if they can apply good systems engineering to rein in this chaos as much as possible and prevent their credit risk modeling and decisioning infrastructures from becoming brittle or difficult to change.

Here’s one view of what a well-organized architecture for a credit risk decisioning environment could look like (again, courtesy of Francisco):

(If you are wondering what Feast is, it’s an open-source feature store that Francisco has been working on that is designed to help address some of the challenges that arise when deploying machine learning models into production.)

It also may be enormously beneficial to lenders if they can figure out how to apply generative AI to Francisco’s graphic above.  

The Generative Emergence

Generative artificial intelligence has emerged as one of the most extraordinary innovations of the 21st century. But not every problem is a nail and using LLMs as a blunt hammer for everything could lead to consequential mistakes.

For example, using Gen AI for synthetic data generation is great for many applications (e.g., question answering based on existing documentation). Credit is not one of them.

On the other hand, using Agents (i.e., autonomous software to execute some set of configured operations) to 10x your data scientists or machine learning engineers building your models can be extremely powerful. This is basically a copilot, and tools like Julius.ai are doing this today.

However, the highest impact use case for Gen AI-powered agents is document verification and parsing. 

Document verification is well positioned for automation as it is often a high-friction, manual task that results in a meaningful drop-off in the customer funnel. Small percentage changes could add up to meaningful conversion amounts for lenders and, while this space is still early, could be one of the biggest impact areas over the next five years.

Closing Thoughts

Nothing in the world of credit risk decisioning is easy. However, if we were to rank order the jobs to be done from least challenging to most challenging, that list would be:

1. Developing a proprietary credit risk model that outperforms off-the-shelf models for your specific product and target customer segment (difficult, but also relatively straightforward given the small search space that you are building within).

2. Deploying that credit risk model into production in a way that doesn’t infuse more chaos into an already chaotic system (confoundingly difficult, especially at a large scale … but still achievable; check out this Feast demo from Francisco to see a glimpse of what a production ML system wielding chaos through great engineering and open source software can do).

3. Architecting lending products and credit risk decisioning environments into a flywheel that produces a compounding analytic advantage (very few banks or fintech lenders have really figured out how to do this, and the ones that have are intentionally very quiet about it).     

The focus that many folks who are new to lending or financial services have on #1, relative to #2, reminds us of that Omar Bradley quote, "amateurs talk strategy; professionals talk logistics."

However, it’s only by mastering #1 and #2 that lenders can begin to reach for #3 and start down the path of building a great lending business.

-Happy Risk Taking!

Alex and Francisco

Post Script

The views expressed here are those of the author and the author alone, they do not reflect the views of his employer or previous employers.

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

Auto-Regressive LLMs are not controllable and are not sufficient to reach human-style intelligence. They will disappear in a few years. Future AI systems will use a different blueprint.

-Yann LeCun

Large Language Models

LLMs are incredible and they have already changed the future of humanity.

They are an important step along the path to Artificial General Intelligence…but they are not the end game.

As Yann LeCun stated, this is largely because of their auto-regressive nature; i.e., they are built to maximize the probability of the next token. This works surprisingly well in practice…but it has its limitations.

Agents

Some have suggested that LLMs could be a new Operating System. I think it’s a reasonable conclusion to see something as amazing as GPT Agents and conclude it’s going to be the next wave of an OS but I disagree.

LLMs are inherently probabilistic.

That’s bad for programming. In software, we want pure determinism (maybe stable APIs) and for some reason people seem to ignore this.

Regardless, using LLMs for product experiences (e.g., ChatBots or a narrow task like writing specific types of writing) seems to be an increasingly common use case.

When the cost of being wrong is low, it works great but when the cost of being wrong is high, that can be massively consequential.

Hallucinations

When the probabilistically generated sequence of tokens goes in a direction users don’t like, we call this hallucination. Hallucinations are reasonably well understood but current approaches to prevent them are still not effective for the the long tail of problems…which is because the long tail is something much more challenging—general understanding, planning, and decision making in a world with a lot of uncertainty.

Beyond auto-regressive methods being insufficient to model the world, I would add that latent embedding representations are also insufficient.

Let me elaborate on what that means.

Representations, Vector Embeddings, and Feature Engineering

In the world of LLMs you hear a lot about tokens. In short, tokens are numeric representations of words or special characters. Machine learning algorithms do their mathematics using numbers and these tokens are pointers to a location in a matrix. When the token is fed through a neural network it is usually represented through a simple binary-encoding in a matrix—i.e., 0 or 1 indicating whether or not the token was present.

Vector Embeddings

Here’s a simple example of three documents with five unique words (tokens).

Binary encoding, while simple, turns out to be surprisingly effective and it is a basic example of featurization.

Featurization is the process by which some phenomenon is quantified or represented to enable a computer to do mathematics. In the simple regression setting you can think of age, education, or employment status as examples of other features. More fundamentally, these features exploit the structured nature of data and represents it explicitly.

For LLMs, we extend this encoding from a single binary representation to a vector embedding (a list of numbers).

We learn the vector embedding representations by initializing them to some random numbers and running gradient descent during unsupervised pretraining and, through what can be only described as mathematical alchemy1, those random numbers begin to mean something.

This representation approach was made famous back in 2013 by folks at Google and the diagram below provides some simple intuition. You can take these representations and can apply things like King - Queen ≈ Woman2.

It turns out these representations, along with new auto-regressive architectures, started achieving the state of the art performance for various NLP benchmarks.

The Old Ways

In the old days, before the dominance of Deep Learning, researchers used to do explicit feature engineering as a part of their modeling pipeline.

Explicit feature engineering is a generalization of the binary encoding example and it has a rich history in natural language processing, computer vision, and tabular data mining. In fact, explicit feature engineering and XGBoost still represent a meaningful share of tabular machine learning in production today.

Feature Engineering

Explicit feature engineering is really just writing code to make quantitative representations of things, often with the goal of predicting something (supervised learning). For example, if you want to build a machine learning model to generate product recommendations, you would engineer features that quantified what a user likes, what they interact with, what they are interested in, what’s novel to them, and really whatever else you can think of and measure (even crudely).

As it turns out, feature engineering can get quite involved and you can end up writing very complex feature transformations, which means you end up with very complex code, which means the engineering systems to deploy them can be quite complex, too. This is because all of those complex transformations depend on different data sources with different shapes, integration patterns, and tradeoffs that really make the engineering challenging at scale.

Human Constructs

I want to pause for a moment here.

These features are usually representations of human constructs or crude approximations for structural constructs. Human constructs are exactly what they sounds like, things in the world that we humans define, categorize, and name.

They are very different than the laws of physics or mathematics that can be rigorously derived and proved—somehow we encoded the language of the gods and in doing so we discovered that some representations are explicit; indeed, one plus one provably equals two. Human constructs are also very different than representing a raw image in machine learning where models learn the latent representation of objects we classify with our set of arbitrary categories.

In this case, “latent representation” is what I would call structural (i.e., a representation of true nature) compared to the category, which is a human construct (i.e., today we call this thing a Zebra but tomorrow we could call it something else just because our view of the world changed).

To summarize, when we define, categorize, or name something usually there is additional meaning behind it and it turns out embeddings approximate that meaning very well using a latent high dimensional representation.

Unfortunately, that representation is hard to make precise statements about…and it also turns out there are other explicit things we are representing and machines should use that information because we humans do.

An Example

Suppose you wanted a multi-modal generative language model to take an image as input and tell you what’s in it. Let’s say you want the LLM to do this. To make it concrete, let’s suppose you uploaded the Zebra below.

The inputs would probably be (1) a query such as “what is this thing?” and (2) the image of the Zebra. The ideal output would be “This is a Zebra” or simply “Zebra”.

In the silly diagram above I’ve outlined how the word token can relate to a latent embedding vector versus explicit feature engineering. For convenience I made the embedding vector the same length as the number of features but they’re entirely unrelated3. I also gave examples of features you might see in practice but they all would rely on another vision model to construct them. Ignoring that massive detail, you can see the obvious difference being that features represent specific constructs we define and embeddings learn a construct that is difficult for us to derive clarity from—what does the “0.2” in the second element of the word embedding precisely mean? We don’t really know, that’s why it’s latent! For explicitly labeled features, we do! But the cost of creating these features is high because it is entirely manual and often requires domain expertise. That doesn’t scale very well at all but it is terribly effective.

We shouldn’t have to choose one or the other, we can and should use both.

Prompts, RAG, and Zero-shot Learning

Prompting is the starting point for people using LLMs in their products and this usually works quite well. The next step is to add Retrieval Augmented Generation (RAG) and then the next step after that is to use Fine Tuning. It’s worth mentioning that some people have moved to RAFTA (Retrieval Augmented Fine Tuning) which I would include in Fine Tuning.

These tools are all mechanisms aimed at reducing hallucinations and they work well in some cases…but (1) any autoregressive generative model will expose some non-zero risk of hallucination and (2) the RAG approach heavily relies on these embeddings and a single convenient distance metric.

The only way to reduce hallucinations to something more palatable (maybe for higher stakes use cases) is to confine the output space and I don’t know if current approaches will ever be able to do that without more fundamental changes.

Facts, Knowledge Graphs, Search, and Retrieval

Along with what Yann called out, I believe five additional things will happen as researchers and practitioners continue to make progress on more general understanding.

1. We will need a mechanism to explicitly encode facts4 and relationships. I believe Knowledge Graphs will be a useful tool to enhance RAG.

2. Search and Retrieval approaches will need to explicitly incorporate these facts into the documents used for generation.

3. We will have to develop new model architectures to exploit these facts and structure of the data.

4. We will need some generic framework to learn or generate explicit feature representations that are interpretable to humans.

5. We will use more fine tuning on smaller, specialized models.

At the core of (1)-(3) and maybe even (4) is feature engineering.

Fine Tuning is The Way

Fine tuning is how all of the successful machine learning applications have quietly improved the world around us for decades, so it is obvious that this is the way. Of course, RAG, prompting, feature engineering, and other levers are useful to improve model behavior but fine tuning lets you explicitly optimize an objective and that’s usually the goal of some software system, so it feels very natural to me…but most users of models are preferring to use one giant model and wait for it to continue to get better. That approach certainly isn’t wrong but I do think there is a lot of utility in narrower models…and the open source community seems to be trending that way.

So, what does all of this mean for Production Artificial Intelligence?

Production Artificial Intelligence

As Knowledge Graphs, features, and specialized models become popular, the uninitiated will learn that running production applications using artificial intelligence requires highly optimized data systems.

From an engineering perspective, LLMs and diffusion models are convenient5 as they pose minimal data constraints6 for online inference—it is sufficient to input text to get relatively high quality output.

This convenience exploded the scale of possibilities. Historically, deploying machine learning models to production has been challenging for many companies, leading to a high failure rate of machine learning projects.

In my experience, these failures are rooted in data challenges. As it turns out, for many business applications some of the biggest obstacles are not building machine learning models but building pipelines to send data from one place to another and exposing that data to customer facing product experiences.

Feature Stores

Feature Stores emerged in 2017 as the potential solution to help solve this plumbing problem and they have been very successful for many companies7. In short, these systems focus entirely on enabling machine learning engineers to get features in online production environments by plumbing the data into a centralized database that can be optimized for online, real-time retrieval.

Vector Databases

Vector databases recently became the preferred tool for Retrieval Augmented Generation (RAG) applications. A vector database is merely a narrow application of a feature store and leveraging a feature store can allow for richer context injection (e.g., relevant user data) along with the retrieved documents.

This is important to state because the current popularity of vector databases omits how LLMs can benefit from rich, structured data both in model training and at inference time.

The Future

The future of production artificial intelligence will rely more explicitly on unstructured tokens and structured data for model inference, so I believe feature stores will become the foundation of the artificial intelligence stack.

I briefly hinted at this point in my last post and I wanted to finally expand on what I meant.

I believe Feast is a solution that enables companies to deliver more products powered by machine learning to their customers easier and faster.

The Plan

My plan now is to pontificate less and code more.

I have two goals: (1) improve the Feast infrastructure and (2) build an open source consumer Fintech chatbot (reach out if you’re interested in collaborating on this).

For this newsletter, I plan to write shorter articles more frequently and focus on technical content to give updates on my progress on my two goals. If you’re interested in Feast specifics, feel free to follow Feast on LinkedIn or Twitter/X.

Closing Thoughts

I am excited by this moment in time where brilliance across the planet is united in driving AI progress forward. It is awe inspiring and I feel privileged to be along for the ride.

The path to artificial general intelligence will be paved in sophisticated software, novel learning paradigms, community collaboration, and hard work.

Happy building.

-Francisco 🤠

Some Content Recommendations

Lex Friedman interviewed the extraordinary Yann LeCun on the limits of LLMs.

covered Synapse’s Bankruptcy filings in elaborate detail.

wrote about Fintech’s comeback.

Alex Johnson wrote about The Servicing Imperative.

Post Script

The views expressed here are those of the author and the author alone, they do not reflect the views of his employer or previous employers.

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

Perhaps I'm old and tired, but I always think that the chances of finding out what really is going on are so absurdly remote that the only thing to do is to say, “Hang the sense of it” it and just keep yourself occupied.

— Douglas Adams, The Hitchhiker's Guide to the Galaxy

I have spent over a decade working in finance and technology.

Some History

I began my career as a statistician building models for marketing and insurance back in 2012. My foray into consumer and commercial banking began in 2014 where my focus was still on building risk and marketing models. In 2017, I left to work at Goldman as an early employee at Marcus. I was a part of the Credit team but most of my work was all in risk engineering, data engineering, and deploying statistical machine learning models1. Then I went on to build my own consumer fintech startup where I was building a personal financial management tool.

After my startup, I joined Fast where I was hired to build risk machine learning—I ended up building out their data and machine learning infrastructure, building their recommender system, and leading two teams.

After Fast imploded, I joined Affirm as part of an acquihire. At Affirm, my team and I also built data and machine learning infrastructure—this time at a scale that was the highest I’d ever supported.

And so my career went from building models, to deploying models, to building infrastructure to deploy models…all for the glory of Fintech.

One might misinterpret that my career choices were more deliberate than I intended. At each step I tried to make choices that were rational but I mostly stumbled into machine learning infrastructure because I thought it was fun.

And now my time has come to join the artificial intelligence race.

I spent the past two years at Affirm managing a team that built out their feature store infrastructure (among other things). A feature store plays a rather critical role in the machine learning workflow and over the past year I became a maintainer for one of the most popular open source frameworks: Feast.

Open Source

I love open source software—it is the accumulation of knowledge, pain and suffering, and thoughtfulness forged by the hands of a few who were willing to build something bigger than themselves. That may sound like hyperbole but it’s not.

To quote Soumith Chintala, the lead author of PyTorch and AI Infrastructure leader at Meta:

PyTorch currently powers most of the world’s AI research and product – with hundreds of companies, research labs and individuals using and maintaining it. It has significant and tangible real-world impact from self-driving cars to drug discovery to cancer research to NASA’s Mars Rover to several consumer products, and that amount of real-world usage often intimidates me.

PyTorch began as a niche project using the programming language Lua and over the last ten years it exploded into one of the most powerful open source projects powering AI. Getting to see its glorious rise has been inspiring.

At a moment when LLMs, foundation models, and vector databases are dominating the attention of most of the media, I am choosing to work on the overlooked (perhaps underinvested) portion of AI: data2.

In most business applications outside of Computer Vision and Natural Language Processing, plumbing structured data is one of the biggest challenges in getting models into production (i.e., providing value to customers in live product experiences). In fact, I wrote an entire article about the challenges of production machine learning.

I believe Feast is a solution that enables companies to deliver more products powered by machine learning to their customers easier and faster. I also believe it can have the same community and impact that PyTorch does.

So I have decided to leave Affirm to focus on building more open source AI infrastructure including but not limited to Feast.

Farewell Fintech

I am sad to leave my team, a company I love, and a mission I care deeply3 about…but I believe we are living through an extraordinary moment in time. I want to be able to look back at the next decade and say I played a role, no matter how small, in shaping the future.

I believe Affirm is the ultimate Fintech and I know it will continue to scale into a colossal powerhouse—Affirm’s ability to underwrite risk, create exceptional products, and distribute them to their customers in a capital efficient way gives them massive competitive advantages compared to other Fintechs, so I am long the stock.

Note: this is not investment advice please do not buy stocks because of stuff a stranger wrote on the internet.

As I leave Affirm, I am so grateful for all of the incredible people I worked with—especially for my team4. The engineering talent is extraordinary and the impact is high.

A thing isn’t beautiful because it lasts.

I was surprised at how emotional I was to leave and that speaks to how wonderful my time was there and while I feel sad to go, I am so excited about my next adventure.

Hello Red Hat

I am joining Red Hat OpenShift AI. Red Hat is an industry leader in open source software. I wrote before about the need for AI to be decentralized (i.e., not concentrated to a single few large entities) and by working on open source and OpenShift AI, I will be able to enable other companies, startups, and engineers to build more AI.

The caliber of engineers at Red Hat is legendary and their history in empowering and amplifying open source is inspiring. I am so excited to get to work with and learn from such talented people and accelerate production AI.

Closing Thoughts

If you’re in Fintech, building in AI, working on an early stage startup, or all three please reach out—I would love to help5. I want you to be successful and I want the world to have more great products, there are enough bad ones.

The future will be powered by more software, data, machine learning, and artificial intelligence, not less. So my goal is simple: accelerate AI.

It’s time to build.

-Francisco

Some Content Recommendations

Jason Mikula shared the latest on Crypto Fraud and Money Laundering.

Nik Milanović announced Fintech Fund II and if you are a Fintech startup looking to raise you should reach out to him.

Nik also shared some incredible wisdom about unnatural product experiences.

Simon Taylor shared his thoughts about The Rise and Fall of Fast. 🥲

Alex Johnson wrote a wonderful piece about The Risky End State for Consumer Lending. I respectfully disagree with his conclusion though. I think a lot of lending and risk management can be replaced with machine learning and software (i.e., AI) but certainly there’s an irreducible human component defining which objectives to optimize. As I wrote before, lending is a math puzzle (mostly).

Post Script

The views expressed here are those of the author and the author alone, they do not reflect the views of his employer or previous employers.

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

Give me your tired, your poor.
Your huddled masses yearning to breathe free.
-Emma Lazarus

These United States

America is a nation of immigrants—and it has a complex history with them.

I am the son of two of them and I am grateful to have had the privilege of being born and raised in this amazing country.

Both of my parents are from Mexico, which means I am both Hispanic and Latino. For those of you that aren’t familiar with the idiosyncrasies of our great nation’s confusing ethnic categorizations, someone who is “Hispanic” is a person from, or whose parents are from, a Spanish-speaking country (this includes Spain, which is in Europe) and someone who is “Latino” is a person of Latin American origin or descent (this includes Mexico).

This distinction, while not terribly obvious to people outside of the demographic, is important only because it maps to a question on the US Census, which ultimately maps to important policy decisions our government makes and the people of this great country vote for.

Affinity Marketing

From a business and analytical perspective, this maps to a segment of the population that may, or may not, be interesting to cater to—at least, that was the premise behind my failed startup (a Personal Financial Management tool for Latinos).

The strategy of targeting a set of customers by some set of criteria is not novel; it is simply Affinity Marketing.

The Partitioning of Data

Analytically, the goal of finding Affinity groups is to find the fewest attributes possible that maximizes the similarity between a group of people and provides a simple description of the group in the hope that this increases the likelihood that they will buy your product or whatever.

This reduces to a form of dimensionality reduction and beyond a few dimensions it starts to get wacky. In fact, this is just Customer Segmentation, which I wrote about previously.

Being the pragmatists that we are, it is natural to try and find the most obvious groups that are the most similar; for example, using demographic information (e.g., age, income, education, geography, sex, or ethnicity). And Latino/Hispanic Americans fit that criteria quite nicely. In fact, Hispanics are the second largest demographic in the US.

Hispanics in Data

So we know that Hispanics represent a large segment in the US by the sheer number of people, which according to our data would estimate something around 63 million people (~19% of the US).

But, from a business perspective, we probably care about understanding this segment more; i.e., getting answers to the following questions:

• What do Hispanics do?

• What are their educational backgrounds?

• How much do they make?

• Where do they live?

And we can answer these questions using the open source library I wrote—PyIPUMs.

Note— Some of the data that follows may lead some to extrapolate and make value judgments about the data, which would be silly because correlation is not causation. Remember, it is useful to understand the world (and its data) as it is and not waste time judging it—even if relationships aren’t causal they can certainly tell you something interesting.

What do Hispanics do?

It turns out, labor. Relative to non-Hispanics, Hispanics are much more likely to work as construction workers, cooks, truck drivers, and in other labor-intensive occupations.

What are their educational backgrounds?

Hispanics, on average, have much less education which may explain the selection into labor-intensive occupations. Nearly 80% of Hispanics’ highest education is a high school diploma or less. For non-Hispanics this is closer to 60%.

It’s worth noting that Hispanics are, on average, younger, so this is an extremely important data point but I would exercise caution in making conjectures about the root cause of the differences in educational outcomes.

How much do they make?

A lot less than the average non-Hispanic but there’s a lot of variance! The median total income for Hispanics in the labor force was $34,901 and for non-Hispanics it’s $50,791. While the median is low, Hispanics in the 95th percentile have a total income of $123,300 so there are still some high income earners in this segment.

You may ask yourself, “Why not prioritize other higher income segments?” It’s a reasonable question but it’s worth remembering that software is best for high scale problems. That’s because the marginal cost of servicing another consumer through software is effectively zero. So, a large audience that is very similar is quite convenient, as their preferences and behaviors will likely be similar.

More importantly, Latino Americans have the 5th largest GDP in the world, so, uh, olé. 😉

Where do they live?

Approximately 33% of Hispanics live in four states. California and Texas used to be Mexico, so that’s not a huge surprise. 🤷‍♂️

So what?

So we’ve learned that Hispanics are:

1. A large chunk of people in the US (and apparently GDP)

2. Tend to work in labor-intensive jobs

3. Mostly have a high school diploma or less

4. Lower income (relatively speaking, of course)

5. Heavily concentrated in California, Texas, and Florida

Which is illuminating in understanding what sort of consumer preferences and challenges they may have…particularly for Fintech.

The Latino American Fintech Opportunity

Marketing isn’t enough to capture the attention of an affinity group, you have to deliver real value to them. That means going beyond the shallow and providing services that your target customers need.

In the case of Hispanics and Latinos, one foundational need is financial services in their primary language (a fairly obvious idea) and another is providing simple, transparent products that cater to a population that, on average, is less educated1.

No one has done a better job at serving this community more than Carlos Garcia from FinHabits.

Why? Because he never followed the bandwagon and has been patiently building useful products for his community since 2015.

He started with simple content, educational tools, and ETF/Roth investing. He’s since expanded to offering a 401k product and even helping consumers find health insurance.

Several years ago I had the privilege of having coffee with Carlos and I got to hear how he built this profitable Fintech firsthand and it was truly inspiring. I should note that I am not an investor in FinHabits but I sure wish I was.

Closing Thoughts

More startups have emerged since my failed foray into the Latino Fintech Market back in 2019 and I am bullish on the space since this segment of the population doesn’t seem to be getting smaller and I’d argue their needs still aren’t well met. While the product offerings have improved since then, I still think there’s a lot of work to be done.

More generally, while many Neobanks failed to be successful in delivering a successful Affinity strategy during the most recent Fintech Reckoning, I believe there are great consumer businesses left to be built and some of them will likely start from an Affinity group. Of course, I could be wrong. I suppose only time will tell.

Happy building! 🤠

-Francisco

Some Content Recommendations

wrote a brilliant piece on BNPL being good. It’s brilliant, just like him. You should read it.

wrote about the CFPB’s proposed role to close Overdraft Loopholes. Overdrafts are, for the most part, bad.

Alex Johnson from Fintech Takes covered three interesting fintech stores in a recent article.

Jake Gibson wrote a guest post for This Week in Fintech and shared great wisdom on building a startup the good old fashion way.

It is not from the benevolence of the butcher, the brewer, or the baker, that we expect our dinner, but from their regard to their own interest. We address ourselves, not to their humanity but to their self-love, and never talk to them of our own necessities but of their advantages.

-Adam Smith, The Wealth of Nations

Currency is a medium of exchange.

Nothing more, nothing less. It enables the exchange of goods and services in a denomination other than chickens, beef, flour; or something else you would have to barter with if you did not have these silly coins.

Those weird—nay arbitrary—set of units provide the world with efficiency.

It is a rather under-appreciated thing how little we have to care about not having to pay for a chicken sandwich1 by fixing my butcher’s computer, car, or whatever.

These arbitrary units play an extremely important role in our society and are more than an elegant mechanism for driving efficiency, they are the implicit oxygen enabling the breath of progress.

The elegance of these silly tokens is rooted in the profound yet simple exchange of labor for capital.

But not everyone has these coins and the accumulation of them is a nontrivial aspect of our economically bifurcated society.

These facts lead to at least two important questions:

1. How do we get these coins?

2. What do we do if we don’t have enough of them?

I will do my best to answer them with data.

As a brief note, I will be explicit about when I condition data and mention where I do so, otherwise I will not. Filters that you apply to data dramatically change the conclusions you make from them and I aim to provide not a perfectly contextualized story of the labor force but instead one about money in America.

How do people make money?

Wealth is the accumulation of positive cash flows generated from income.

Typically that income is generated through working and earning wages, sometimes through investments, sometimes through inheritance or gifts, and sometimes through government subsidies.

That last one is often forgotten but a substantial share of Americans do not have a large savings to pull from nor do they have a FICO score greater than 750 (but more on that later).

Wages

Approximately 85% of employed people work for the private sector and the remaining 15% work for the public sector (i.e., federal, state, and local government).

The government is the single largest employer in the US.

Walmart is the largest private sector employer with 2.1 million people employed by them; Amazon trails behind; and the rest of the major employers are much smaller.

Knowing where people work is helpful but it doesn’t tell you how much they make, unfortunately we don’t quite have that level of data publicly available.

But we do have IPUMS—i.e., the Census.

From the Census we know the median annual personal income in the US was $40,080 and the median household income was $74,580 in 2022.

And while the median income of both households and individuals is informative, it certainly doesn’t give us the whole picture.

Fortunately, the Census data is open to the public and we can explore the data together.

In the graph above I plotted a smoothed histogram of the Total Income reported in the Annual Social and Economic Supplement (ASEC) issued by the Current Population Survey (CPS).

It is a relatively well known empirical fact that higher education tends to lead to higher income and we see that in the graph above. Of course, as you can see there is significant variation and while that empirical fact is true on average it certainly is not true for everyone. Looking at the entire distribution2 gives much richer context about that.

I will likely do another article diving deeper about income by various dimensions. I chose to display income by education here because it is (1) a quite well known and studied average phenomena and (2) I have struggled to find a proper visualization of the distribution of income by educational attainment.

Beyond Wages

We can do more.

The ASEC data allows us to dig into income sources further and breakdown what makes up a person’s total annual income (shown in the graph below). As you would expect, most people’s annual income is from some form of earned income—approximately 67% (wages, business income, etc.). The next largest category is from government sources at 23% and the third is investments at ~9%3.

Investments

The average investment income by educational outcome tells a rather dramatic story, especially when you make a modest assumption that the investment income is 10% of the actual capital required to generate that income (this is obviously wrong but likely understates the total capital).

In short, you will find that educational attainment maps dramatically to investment income. On average, those with less than a high school diploma effectively have no investment income (notice the concentration of the red density function around 0) and those with a graduate degree have an average of ~$7,200.

Some people earn a meaningful share of their income from investments alone. This is money at work but only represents a small share of Americans.

Inheritance

The median inheritance in the US was ~$12,353 according to the Survey of Consumer Finances but that is not uniform across income percentiles. UPenn provided a great set of data and I have graphed it below.

As you can see, high income earners have high income earning parents who leave them a meaningful inheritance but 90% of Americans get ~$13,000 or less. Of course, a one-off capital injection of ~$13,000 is very meaningful to an average income earner of $40,080.

Government Benefits

It turns out that ~25% of Americans have some form of Government provided income and it constitutes an average ~65% of their total income for those that actually do have Government income.

The variation by educational outcome further highlights differences in economic outcomes, where Bachelor’s and Graduate degree holders have a nearly ten percentage point difference in Government income relative to those with a High school diploma. Those with less than a high school diploma have the lowest Government income and this result could warrant a further deep dive in its own right.

Government, Investment, and Wage Income

Averages hide variation, so I attempted to provide a single graph that aimed to show as much as possible in as few lines. The chart below takes a bit of time to digest but it is worth reviewing. It shows a more holistic picture of the income sources of America. I should note that because the Census does not offer inheritance information, I cannot plot it here.

The conclusion is simple: most people make their income from working, second from the government, and some from investments.

How terribly obvious.

But now that obvious statement is grounded in data and a little more rigor.

How do people manage liquidity constraints?

They borrow money.

As previously shown, 46% of Americans do not have any investment income, which means they likely don’t have a stockpile of capital they can trivially access if they have a large emergency expense. That means that nearly half of Americans are liquidity constrained—i.e., they have low liquid assets relative to disposable income or recurring expenses. We can infer that if people have no investment income then it likely means they have limited savings or disposable income, which would correlate well for a liquidity constraint.

To properly estimate the magnitude of how many Americans encounter a liquidity constraint, we would need both banking data (e.g., cash account balances and other assets more generally) as well credit data (e.g., loan types and sizes). Unfortunately, the Census data does not provide either and other alternatives, like the Federal Reserve do not give a particularly useful picture either (this is me explicitly saying headlines like “Credit card debt reaches all time high” are stupid).

Fortunately, Experian published a report on the state of Consumer Credit in Q2 2023.

And the key data points are:

1. The average non-mortgage debt balance was $23,317 in 2023

2. The average non-mortgage debt balance increased by 4.7% from 2022 to 2023

3. The percent of accounts 30 days or more past due increased 0.36 percentage points

4. The average monthly payment required to service debt increased by 8.5% to $1,148

Overall, from the single lens of FICO, the consumer appears to be credit-wise healthy. That may not map to financially healthy overall but it is certainly something we can look at.

FICO Score

So what does the FICO distribution tell us?

Number go up. Yay. 🙃

This distribution would have you think that 53% of people with a FICO score have FICO < 750 but this omits the ~19% (49 million) of people in the US who do not have a score. In practice, these consumers are declined from applications of credit because there is insufficient credit history/data which is only marginally different than treating4 them as having a score of zero which means nearly 63% of people in the US actually have FICO < 750.

The conclusion here is also simple: most people don’t have perfect credit and a lot of people don’t have what’s considered “good” credit.

It is worth stating this because your credit score impacts how much you can borrow and the price at which you can borrow, which obviously drives your ability to manage a liquidity crisis.

In Summary

Most Americans don’t have big piles of cash stocked up and large amounts of credit they can access in case something happens.

So what can Fintech companies do? 🤔

Build financial products that help your customers.

Finance reduces to the borrowing, spending, saving, and investing of money. Fintech is about using technology to efficiently do those things.

So, for any single Fintech company that means executing efficiently at high scale so that you can (1) be more cost effective than your competitors, (2) improve your profit margins, and (3) pass on some share of your cost savings to your customers.

Times are challenging. The past year has been turbulent and there is more turbulence on the horizon, so please go and work your butt off so you can help our fellow Americans5.

I have provided you with sufficient data to show you that people need our help, so we must rise to the challenge—isn’t that the whole point of Fintech?

There is ample business opportunity in providing value to customers of financial products. But if benevolence does not compel you enough, consider the massive economic opportunity that exists.

Not long ago both public and private markets understood how massive that opportunity was…and then the reckoning came. It appears that the pendulum has swung probably too far in the other direction. Remember, be greedy when others are fearful.

Closing Thoughts

It took me roughly six months to write this article6. Largely because I spent so much time pulling and cleaning data and making revisions. I also created my first open source Python library (PyIPUMS) to work with the data, which was one of the goals I had this year and I’m happy to now mark it as completed.

The code and GitHub repository is available here. You can also find a much more elaborate set of graphs in the Jupyter Notebook used for the analysis here.

I first worked with IPUMs in 2011 (👴🏻) during my first master’s. Back then I was coding in STATA and I had mostly no idea how to work with data. I decided to write this package to help graduate students in Economics or Statistics that may not be as familiar with Python, how to work with XML, and with data in fixed width format—as it can be quite terrifying for the uninitiated. Hopefully it saves them some time and helps them do great research.

Over the coming years, I hope to make lots of updates to this software so that others can use it to explore the extraordinary Census data.

There is much that can be learned from it and I sincerely believe that a lot more constructive conversations about our country would happen if we better understood our fellow Americans in data.

Happy learning! 🤓

-Francisco

Post Script

I would like to thank Alex Johnson, Dan Kahn, Jared Franklin, and Bruno Werneck for taking the time to review this article and offer their wisdom.

Several people made the great suggestion to provide some of the data segmented by Age. I found that doing so didn’t result in anything that was terribly novel but I’ve provided the univariate density and distribution functions below.

Some Content Recommendations

wrote The Techno Optimist Manifesto. It’s a little intense but it is a useful call to action to not sit idly by and, instead, build the future. And it is a wonderful reminder that outcomes only happen if you force them to.

President Biden issued an Executive Order on Artificial Intelligence. It is long, has good intentions, and is the wrong approach to making the US competitive in Artificial Intelligence. We should be leaders. The future will use more AI, not less. The Government should find ways to incentivize innovators to build more foundational models to build the future, not add laughable friction to the inevitable. You cannot stop compute, AI is inevitable.

and published a podcast recapping Money 2020. It is always great to hear from them individually and jointly is all the more fun. I enjoyed hearing about what’s fashionable and not this year at Money 2020. I’ve never been, nor do I ever think I would go but it’s great to hear about.

Alex Johnson also wrote an absolutely wonderful piece on Small Business Lending. I actually have worked on SMB lending during my time at the Commonwealth Bank of Australia and learned quite a lot. It’s an incredibly complicated area to operate in analytically, which makes it terribly fun for someone like me.

from also wrote a more detailed recap of Money 2020 and highlighted the wrecking of Crypto and attention refocusing to AI. People should be cautious to say that AI is the new Crypto, Fintech and Artificial Intelligence have a rich history, I even wrote about it before.

from wrote a great piece about Twitter’s/X’s foray into becoming the most valuable financial company in the world.

The goals in statistics are to use data to predict and to get information about the underlying data mechanism. Nowhere is it written on a stone tablet what kind of model should be used to solve problems involving data.

-Leo Breiman, Statistical Modeling: The Two Cultures

The Two Cultures

It has been over twenty years since Professor Leo Breiman wrote the famous “Two Cultures” article. For those of us machine learning practitioners working in industry, it is Gospel.

It’s regarded so highly because, not only did he invent Classification and Regression Trees, Bootstrap Aggregating, and Random Forest, but because he changed the nature of statistics.

It was Dr. Breiman’s pragmatic perspective and thoughtful writing that shaped my approach to modeling data. I am neither Bayesian nor Frequentist, I am a Pragmatist1—choosing the tool that best works for the data.

A Decade of Chaos

I have spent one third of my life modeling data and applying machine learning to various forms of quantitative financial problems.

This past April I was invited to speak at Tecton’s apply(risk) conference and I was quite flattered by the opportunity. When I thought about what I could talk about, I settled on two things: (1) all of the mistakes I’ve made during my time doing this sort of thing and (2) demoing Feast—an open source framework maintained by Tecton and used by some of the best technology companies doing applied machine learning.

So, if you’d like to attend my talk on Tuesday May 30th at 1PM EST, use this link—it’s free. The slides are also freely available here.

Update: You can find the full recording here freely available on YouTube.

And I’ll share my notes in the rest of this post.

Introduction

I’ve written about myself before here. In this talk I aim to discuss six things.

1. Taking ML from 0 → 1

2. Building Models like building Software

3. Data Lineage

4. Common Mistakes

5. Risk and the Engineering of Chaos

6. Demoing Feast

Machine Learning: 0 → 1

If you’re just getting started with machine learning in your product, it’s worth keeping in mind a few things.

The most important of which is that you will probably spend 90% of your time at the start on the data and software work and 10% on the actual machine learning.

Some other useful things to keep in mind:

• Data is the foundation of all machine learning models.

• Great software produces good data.

• Machine learning is only as good as the data it is trained on and only as useful as the software surrounding it.

• Product is the foundation of customer value and machine learning will not succeed without customer value.

• ML can amplify a great product experience but it won’t make a bad product experience great.

• Software is the core engine and it reinforces the flywheel. ML without good software is doomed.

The Machine Learning Flywheel

Great software leads to great product experiences, which lead to lots of data, and lots of leads to great machine learning. Accomplish all four and you can unlock the Machine learning Flywheel.

Build Models like you Build Software

I’ve written before about how 87% of machine learning projects fail and that it’s mostly driven by poor engineering practices.

Data is generated through code. Models are generated through code. Not having lineage of both will result in pain. Obviously, without those things reproducibility is impossible.

More importantly, this will result in either models not getting shipped or blowing up production. So make sure you have the right engineering standards in place.

Data Producers, Data Consumers, and Data Lineage

Data Producers generally don’t know who Data Consumers are, so data transformation pipelines often break. This is an industry-wide problem, not only limited to machine learning.

Data Contracts are an increasingly popular approach to mitigate this problem by having Data Consumers set a schema to test against in lower environments to prevent the Data Producer from breaking their input data before hitting production.

This is the only way to manage downstream breaks and it is necessary friction.

Some Common Mistakes

Here is a non-exhaustive list of ways I have blown up a server over the last decade:

• Featurization Errors

  • Quite literally just a bug when creating a feature (e.g,. accidentally dividing by zero before trying to execute a multiplication will make your computer go boom 🤯)

• Machine Learning Library Errors

  • Most practitioners are using the same machine learning libraries (e.g., pytorch, glmnet, xgboost, sklearn, tensorflow, keras, lightgbm) but often those libraries have very specific C++, Fortran, or other dependencies that are run on someone’s laptop and not on the server being used to run the model. Docker fixes this.

• Loading Model Errors

  • Make sure your model literally fits in your server.

• Service Errors

  • Make sure you have some exception handling for when your model breaks. Don’t give your customer a bad user experience because of poor planning.

• Business Logic Errors

  • If you do something to your model after you execute some multiplications, make sure you test the behavior.

• Statistical Errors

  • Make sure your training data was a representative sample and if it’s not, monitor the performance closely.

Overall, many of these errors can be avoided with some simple choices but ten years ago this was not as obvious to me.

Risk and the Engineering of Chaos

I wrote an entire article about this.

Modeling risk means you are modeling chaos. Rules impose order on chaos. Rules tend to spiral into chaos themselves. Know that your model plays an important role in a large ecosystem of entropy.

The Demo

As mentioned, the code is available here.

The main focus is on how Feast can play a critical role in mitigating some of the significant challenges that I experienced in the past and the demo aims to highlight that.

I won’t go over all of the many things Feast supports out of the box but it is a rather comprehensive tool that is used by some of the best technology companies in the world (*cough* Affirm *cough*).

The demonstration shows what a real ML risk system could look like for onboarding a driver to a driving app and then monitoring their risk daily.

There are two models, an onboarding model and a daily batch2 model. The onboarding model takes an input form with four fields:

1. State

2. Social Security Number

3. Driver’s License

4. Date of Birth

Since all of the data is in the form, we have to use that data to construct the features, I do this in Feast using On Demand Feature Views, which works great. Then I toss those constructed features into my model, get a score, and then make a decision.

For the daily model, I use six features calculated from a batch run from “yesterday”:

1. Yesterday’s Conversion Rate

2. Yesterday’s Acceptance Rate

3. Yesterday’s Average Number of Trips

4. Yesterday’s Average daily trips < 10

5. Yesterday’s Acceptance Rate < 0.01

6. Yesterday’s Conversion Rate > 0.80

And here, because we can precompute the features, there’s no need for a dynamic computation so we just look up the features at request time. This is called a Feature View in Feast and the batch use case is the most basic one. Similar to the Onboarding model, I fire the features to the model after retrieving them, get a score, and then make a decision.

I also included a clunky little UI to show what the user experience could look like—the UI was unnecessary but it makes it feel a little more concrete.

The whole thing (i.e., the ~660 lines of Python code) boils down to three APIs.

def get_onboarding_features() -> Dict:
    # done via Feast
    ...

def get_onboarding_score() -> float:
    # done via a fake regression but you could imagine using a 
    # more sophisticated machine learning model
    ...

def get_onboarding_decision() -> str:
    # done using the score and a simple threshold operation
    ...

And the three APIs can be reduced to one with two being called behind get_onboarding_decision(), which is the API you would want to expose to some client, probably.

Maybe one that would do something like this:

In practice the get_onboarding_features() part involves a lot but the point is to highlight the separation of responsibilities for the purpose of understanding where common things explode.

If you look at the code, Feast handles a lot for you. Especially many of the gotchas that come up when working in data and machine learning. I won’t elaborate on all of the ways it handles it because their docs really are much more useful than this post but suffice it to say that it’s a powerful framework.

Closing Thoughts

Machine Learning is quite popular in the media today due to LLMs, and this past week it appears to be 5x for the chip makers but people seem to be forgetting that model training and inference are the last steps in the ML pipeline. Data and software are the foundation and lifeblood.

If GPUs are the picks and shovels then data is the mountain.

Happy mining!

-Francisco 🤠

Some Content Recommendations

wrote a fantastic piece on The Future of Generative AI in Fintech and it’s, you guessed it, Decentralized. It’s a great piece and Simon articulated my thoughts 1000x better than I could.

at wrote an exceptional piece on SoLo Funds deceptive practices and some of Revolut’s current troubles.

wrote a great piece on what Data Engineering has gone through over the last decade.

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

Anyone who considers arithmetical methods of producing random digits is, of course, in a state of sin.

-John Von Neumann

A clever thing about generative models and ChatGPT is that they give you different results for the same prompt (or input). This is done, presumably, by setting the random seed to the computer’s current system clock time just before computing an extraordinary number of matrix multiplications (or some equivalent in a distributed sense when all of those parameters do not fit in the RAM of a single computer).

This trivial detail gives the illusion that ChatGPT is non-deterministic, but it’s not.

You can see this for yourself in HuggingFace’s open source ChatGPT alternative: HuggingChat.

Large Language Models (LLMs)—just like all machine learning models—are an estimated static equation, which means that for a fixed input you will receive a fixed output. Generative models do some probability weighted random sampling to provide a little flair and the mirage of sentience.

But LLMs are nothing more than a bunch of numbers, multiplications, sums, and a splash of pseudo-random sampling.

Centralized Artificial Intelligence

OpenAI encountered some troubles recently, and they’ve done some incredible work to overcome them.

But OpenAI, MetaAI, Google Research, DeepMind, or anyone else can’t solve the core problem, which is that true Artificial General Intelligence (AGI) needs truly Open AI; that is to say that no single entity or research lab should be trusted with the power of AGI.

After several months of reflection, I’ve come to only one conclusion: a cryptographically secure, decentralized ledger is the only solution to making AI safer.

I’ve thought for quite some time that blockchain and crypto the technologies—not necessarily the digital currency—had incredible implications but I didn’t know what for…and it turns out the answer was hiding in the next hype cycle.

I am neither a crypto maximalist, nor even necessarily a crypto advocate. I am, however, a technologist who sees the value of the technology used by most crypto currencies.

As a brief aside, my biggest skepticism with cryptocurrencies is that a non-trivial share of their advocates seem to treat it as a long term store of value which creates an economic disincentive to transact, which then renders it a poor medium of exchange. Regardless, that’s how lots of people have treated Bitcoin, Ethereum, and other tokens.

So why do I believe “a cryptographically secure, decentralized ledger is the only solution” to truly Open AI?

Because it solves some core problems.

Some of AI’s Problems

As I said, no single entity should be the sole owner of any true AGI. It creates far too much power in the hands of only well-capitalized institutions (i.e., those that can afford the compute necessary to train a giganto model).

There are other challenges outside of this, too.

Reproducibility

The academic literature is ridden with examples of state of the art (SOTA) models that weren’t reproducible and while there’s an ongoing effort to improve this suffice it to say that a lot of models aren’t reproducible (maybe even most).

That’s bad science, but the incentives in academia are what they are.

ML industry practitioners have come quite a long way in model reproducibility (i.e., model version control) but in the early days many forgot about data version control.

Quite obviously, a model cannot be version controlled unless the data and the code that constructed both the model and the data are version controlled, too.

Thinking otherwise is dumb (and, remember, that’s not you!).

So in order to have reproducibility in general we need model and data reproducibility, and it turns out that a decentralized database that records every version posted to some chain is a very good candidate for that.

Data Privacy

Most people don’t really care that you use their data for things so long as it (1) serves the right product experience and (2) isn’t malicious.

But some people care a lot! And some countries (e.g., Italy or European countries) care 100x more than that.

A benefit of cryptography and decentralization is that you can estimate “local” models without sending data and still contribute back the estimated gradient to the network. Additionally, you could encrypt the data as well to secure it. This is known as Federated Learning and is an active area of research.

That said, this approach isn’t actually what I think should exist, which are two separate ledgers: one for data and another for learning.

Stale Information

A frequent complaint that users of ChatGPT have is that the model was only trained on data up to September 2021, which means that the data and model are stale. Because it used large scale web data this makes a lot of sense as a practical limitation but for AGI to work, we need streaming data and continuous learning.

Both of these problems are non-trivial and require quite sophisticated large scale distributed computing and streaming data infrastructure...or they can be solved through decentralization and gradient mining.

Massive Compute Requirements

I would like to underscore the “Large” in “Large Language Models”, they are very big and costly to run. Which is one of the main reasons why people or labs outside of the technology industry can’t really build these state of the art models1.

As a brief aside, academics, in their attempt to develop novel algorithms, most iterate on novel-ish architectures rather than try to update existing models, which is arguable a lot of wasted compute. LoRA is an an extraordinary example of the exact opposite of this.

Over the last decade, large scale machine learning models benefited greatly from using GPUs instead of CPUs because they are much more efficient at executing matrix multiplications (an embarrassingly parallelizable mathematical operation).

They also turned out to be incredibly useful for Bitcoin mining.

Miners could decide to compute gradients (i.e., train a model) instead of transactions on the blockchain and, theoretically, this would be a straightforward migration.

Incentives

Miners don’t mine for the sake of increasing our carbon emissions—they mine to make money. Therefore, there needs to be an economic incentive to make miners want to mine Gradients.

This could be a digital currency or whatever.

There also needs to be an incentive to contribute training data. People should be rewarded when they choose to contribute their data (DeSo is doing this) and even more so for labeling their data.

Model Forks

Crypto currencies are often forked and expanded upon for different goals. If we have a decentralized system where computed model weights are published to a decentralized ledger, then we can not only recover models at any point in time but we can also fork them and train them with different goals in mind (e.g., new architectures).

Beyond creating an extraordinary lineage of models, in an extreme case of models misbehaving (i.e., humanity’s doom2), we could find the point in time and the data that led to a chaotic AGI.

Enterprise Value

Who would benefit from a decentralized AGI?

First and foremost, uh, humanity.

Secondly, I think there would be a lot of implementation opportunities in embedding these new decentralized models, similar to how ChatGPT plugins are now all the rage. If you make the crypto analogy it was exchanges that were useful to users, so one might think an exchange for the usage of these models could ultimately be the answer.

As the world of technology evolves rapidly over the coming years, I actually think a marketplace for different types of AGI could be a thing. I know the obvious flaw here is “A true AGI would be able to have intelligence across a broad set of use cases” and while maybe that’s true in the future, it’s not true now and I imagine there will be lots of capturable value between now and when that future comes.

Crypto Fixes This

As mentioned throughout this article, a new approach needs to be taken to decentralize LLMs and AI more broadly so that we can attempt to control the inevitable “singularity”. The approach I propose is analogous to Proof of Work but instead of arbitrarily wasting compute, we can use the compute to estimate gradients.

I also mentioned that we would need two ledgers: (1) for the model weights and (2) another for the data used for training those weights. These could be treated in the same way as candidate transactions being added to the Blockchain where signatures are used to verify the chain of transactions.

Closing Thoughts

All of this may sound a little ridiculous but it’s not. In fact, the work has already begun by the former CTO of OpenSea.

At the moment, many people (especially on Twitter) are pointing and laughing at crypto enthusiasts after the recent fall in cryptocurrency prices and that is a potential indicator that people have gone too far the other direction in their thinking about the space.

In general, it’s good to not jump on the bandwagon.

Happy mining!

-Francisco 🤠

Some Content Recommendations

wrote a banger on Goldman's Offloading of GreenSky.

wrote an excellent piece on Fed expectations this week.

Alex Johnson at as always shared some great thoughts on Winning in Embedded Lending.

Simon Taylor at wrote an excellent piece about needing narrower banks and the latest Fintech drama.

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

A bank’s attitude to risk is not passive and defensive; a bank actively and willingly takes on risk, because it seeks a return and this does not come without risk. Indeed risk management can be seen as the core competence of a bank.

-Alexander J. McNeil, Rüdiger Frey, & Paul Embrechts, Quantitative Risk Management

A Brief History of Chaos

In 1961, the prolific engineer and meteorologist Edward Lorenz pioneered modern Chaos Theory1, originally named “Sensitive Dependence on Initial Conditions”, while simulating sequences of water molecules to predict weather patterns.

Lorenz discovered that the predictions of his deterministic simulations were extremely sensitive to minuscule changes in the initial conditions. The profound consequence being the realization that some systems in nature are unfathomably volatile.

This, too, is true of the quantification of risk.

Borrowing Short to Lend Long

In the wake of Silicon Valley Bank’s collapse, there’s been much attention on how banks “borrow short to lend long”, while this is true the important details are in how they lend long.

The "how" dictates a banks very survival and while there is variance across banks I’d bucket them into two categories:

1. Manual

2. Algorithmic

Alex Johnson nicely characterizes the first as “Relationship Lending”, which served humans for most of our existence until Discover, Capital One, and a handful of others shook the finance industry with automation in the 1990s.

In recent decades, Fintech enhanced this approach further through advances in software and quantitative underwriting—all in the service of increasing conversion2.

Lending

To be a little reductionist, lending reduces to: a decision, a price, and a duration.

At a high level, these three things rely on answers to four important questions

1. How much does it cost to lend?

2. What’s the maximum you are allowed to charge?

3. Can this person or entity pay you back?

4. Will this person or entity pay you back?

There are many important details like how these things evolve over time, how price sensitive a consumer is, how the price impacts whether the consumer can afford the loan, how the duration of the loan impacts the risk, and maybe even some consideration for the cost of acquisition, but this gross simplification provides a nice starting point.

So how do we get answers to our questions?

1. The Fed

2. State and Federal regulations

3. Data and math

4. More data and more math

The first two are reasonable, we look at the interest we give to our deposit customers and any interest rate we charge above that for our loans is expected profit (minus some other stuff).

The latter two is where the competition begins.

Financial Data is Beautiful Chaos

In a previous article discussing Fintech and machine learning, I stated that “Financial Data is Chaos” and outlined why. In short, “everything surrounding this money stuff was kind of made up by humans to keep track of things. That is a profoundly different phenomenon than your visual cortex, linguistics, or the laws of physics.”

Financial data is more than just data.

You are observing real people with real lives, and real life is beautiful chaos.

We fall in love, get our first car, go off to college, move away from home, get married, grow our families, say goodbye to loved ones, and sometimes even buy a home—and all along the way the financing of our purchases with money stuff is an annoying detail we sometimes have to think about.

Crude representations of all of that life are stored in a bunch of stupid databases that banks use to try and make reasonable business decisions about whether or not we can afford to pay for those things on some schedule.

All of that life—encoded in data—is what makes modeling risk so fascinating…and also terribly complicated.

So how, then, do we do something useful with all of this chaos?

Lending is a Math Puzzle

In the aforementioned article where I was focusing on discussing Artificial Intelligence and Fintech I also stated that “consumer finance is basically a multivariate constrained stochastic optimization problem.” and lending is actually what I was referring to.

To be explicit, we have data (i.e., a random process), the 4 questions I mentioned above (i.e., variables and constraints), and something we want to optimize (lifetime value of a customer, profit, or maybe something else). In order to solve this optimization problem we need two things:

1. An underwriting model

2. Really good engineering that enables (1).

What does that look like?

Underwriting and the Fight to Conquer Entropy

In banking, underwriting is the detailed credit analysis preceding the granting of a loan, based on credit information furnished by the borrower…Of late, the discourse on underwriting has been dominated by the advent of machine learning in this space. These profound technological innovations are altering the way traditional underwriting scorecards have been built, and are displacing human underwriters with automation.

-Wikipedia

In the modern world, we use data, statistics, machine learning, and software engineering to do credit analysis.

This is what I’ve spent a significant share of the last ten years doing. I’ve worked at AIG, the Commonwealth Bank of Australia, and Goldman Sachs building risk models for commercial and consumer underwriting (among other things) and I can confidently tell you that over that decade, while the questions remained the same, how we approached answering them changed dramatically.

Underwriting (i.e., the approach) has two components:

1. A risk model

2. A policy.

What does a risk model look like?

For credit risk models, this equation just says that the probability of Y (maybe previous defaults or delinquencies) depends on some information X (a bunch of features/attributes that may be useful in predicting Y) and the thing on the right is just an old fashion Logistic regression. A logistic regression is a classic approach and while modern-day practitioners may swap the gibberish with something a little fancier (e.g., a machine learning algorithm) it has served banks well for quite a long time.

Note: Mathematically, the Logistic regression solution is a maximum entropy solution and minimizes the cross-entropy loss function. So when I say the “fight to conquer entropy” I’m making a play on words that underwriting is about building good regression models because it is…and it’s a fun math pun3 (*winks*).

I won’t go into the details of how to build a credit risk model but suffice it to say there’s a lot to it. McKinsey has some opinions (lol).

What does a policy look like?

In the diagram above we see rules, data, a scorecard, and a decision (the “Reject” or “Set “Limits”). All key ingredients for underwriting.

But the simple model and the policy above are two trivializations. In the real world this gets more complicated and the rules in a policy spiral into anarchy as different teams look to hit often competing KPIs and these simple rules need their own system.

Cue the Rules Engine.

Rules Engines and the Engineering of Chaos

I wrote before about Fintech and the Anarchy of Rules where I stated that “Rules/Decision engines are ubiquitous” in Fintech.

They codify the world's regulatory policies, represent the risk appetites of banks, and encapsulate the hopes and dreams of analysts grappling with an existential crisis.

More seriously though, a “Rules Engine” is software that quite literally executes rules4 to make a decision about something and they’re an important piece of software in an overall complicated architecture necessary to handle risk problems at scale.

Note: I know these systems quite well. I’ve worked with too many of them: GDS Link, Provenir, FICO (yes, they offer a decision engine), Sliderule, and I even wrote one myself (🥲). I often get asked “Should I buy or build?” and, in general, I’d say that it really depends on your organization and goals.

In practice, decision engines can have thousands of rules, elaborate decision paths, complex flows of data (e.g., calling TU for some customers and Equifax for others), and an intricate coupling with machine learning models. In the diagram above, I’ve outlined the key components used in the architecture of a decision engine, probably.

This diagram is useless5.

All of the magic (i.e., hard work) is in the code that powers this Chaos Machine6, and I want to explicitly over-communicate that a silly diagram is useful for intuition but it is not useful for anything beyond that. In fact, it is so useless that I happily share it here because there is nothing special about diagrams. There is, on the other hand, a lot that is special about good code. Few companies produce that.

Why do I say this? Because pretty much every bank or lender is doing a version of this diagram. So where does the differentiation happen?

Distribution, Product, and Speed…and also how you do the thing!7

In lending, the how is in the data generation, policy development, machine learning, and the software around the money stuff.

So if you want to build a lending business that compounds over time, you have to be executing well on all of these areas—that’s how Capital One became the Goliath it is today.

Note: I would be remiss if I didn’t mention that once a bank has a portfolio of loans and other financial assets generating liquidity over different time horizons there’s still lots of quantitative work to do and while Basel II provides guidelines for the calculation of minimum capital requirements based on risk weighted assets, managing your capital and liquidity expectations is both a challenging and interesting area of work at a bank.

Closing Thoughts

The optimal amount of fraud is non-zero8.

-Patrick McKenzie

The banking industry has come a long way since relationship based lending, but there is still ample opportunity for innovation. In the modern world, all of that innovation is driven by software, analytics, and machine learning that reinforces the value of that software.

Lending is the business of losing money and, ultimately, effective risk management means recognizing that the optimal amount of losses is non-zero, indeed.

How you diligently manage those losses is what separates the good lenders from the bad and the best way to control that chaos is through great software.

There are no shortcuts. No single tool will fix brittle and broken systems, so my advice is simple: fall in line or fall by the wayside. The future of money depends on it.

Happy lending! 🏦💸

-Francisco

Some Content Recommendations

Ron Shevlin wrote a great take on the numerous bad takes from Twitter on the SVB collapse.

wrote an excellent review of Hindenburg's research report on Block. He also wrote about BNPL (where I, of course, have biases). As always, Jason was very thoughtful in his writing. He wrote a lot but I will only share a few thoughts that I have: (1) survey data, while useful, is never as good as measuring actual behavior (what people say they do and what people actually do is, unfortunately, often inconsistent) so the data should be taken with a grain of salt; (2) to quote Jason, “One thing the report does demonstrate is that consumers who choose to use BNPL are heavily indebted across a variety of credit products…it is possible that BNPL is a net positive for these consumers, even as they struggle to manage their debt load.”; and (3) my experience as someone who grew up in a low-income area in South Chicago is that people tend to manage their cash flows very tightly when they have low income and when they experience liquidity shortages, the lender of last resort ends up being family members and that can be very consequential. Lending regulations in the US are very pro-consumer compared to other countries and have many great protections for when people, unfortunately, aren’t able to make repayments. It’s important to understand that financial hardship happens to a lot of people and it’s all about what tools they’re offered to get themselves out of that situation9.

Alex Johnson from Fintech Takes wrote “Unlocking the Next Wave of Innovation” and emphasizes that embedded lending is the future. I very much agree with him. He also highlights the challenges for lending and focuses heavily on servicing and he is spot on.

wrote "Attention is All You Need" and discussed the history of the algorithm and the paper that led to GPT. It’s really quite a fun read.

Patrick McKenzie wrote “Banking in very uncertain times.” It’s excellent. Please read and learn an extraordinary amount in ~7500 words.

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

 “The illusion that we understand the past fosters overconfidence in our ability to predict the future.”

-Daniel Kahneman, Thinking, Fast and Slow

The excitement behind ChatGPT and the latest innovations in AI continues. Bing has launched an integration with ChatGPT, Google is on its way with Bard, and Amazon is stepping up to the plate by partnering with HuggingFace.

What a time to be alive and to be in machine learning!

A lot of folks in the Fintech space have been writing about Generative AI in Fintech (I did back in late December) but, rather than write, I thought I’d actually try something I discussed in one of my previous articles.

In short, I called out the problem with ChatGPT being wrong and I gave an example of asking questions about a checking account. Being the empirical person that I am, I decided to give it a try to see how ChatGPT actually does.

Can ChatGPT Learn Fintech?

Yes and no.

Note: I am being imprecise in my language intentionally. I am using “Fintech” and “transaction data modeling" interchangeably even though “transaction data modeling” is one example of a Fintech use-case. This is because transaction data modeling is a great consumer Fintech example that just about everyone can relate to, but the conclusions generalize more broadly…at least, that’s my opinion.

The current model architecture was trained on raw internet data, which is mostly written language. I’ve said before that Financial data is chaos (and tabular) due to the exhausting number of rules at play, which is a very different underlying structure than language.

This fact makes it non-trivial for ChatGPT to learn the same things that Machine Learning Engineers (MLEs) working in Fintech usually construct using their expert knowledge and general intelligence (*winks at audience*).

As an example (and in my tweet above), I gave ChatGPT ~80 of my credit card transactions and asked for very little…and it said a lot!

It was able to compute aggregations, summations, and even an arg max on the data. This makes sense because ChatGPT was trained on lots of code itself so it probably had seen lots of examples of how to do exploratory data analysis (EDA). Note that some of the computations were wrong but I could probably be more explicit in my prompts to get the results I wanted.

Asking ChatGPT to summarize a sample of transactions is doable because it has access to a small amount of memory (approximately 4,000 tokens ~16Kb) which means it can remember this sample of data.

For something other than a trivial example, this becomes a distributed systems problem—maybe easily handled by Snowflake or BigQuery but it is irrefutable that ChatGPT can no longer fit large amounts of data in memory.

What needs to happen for ChatGPT to learn Fintech?

You’d need a hybrid architecture where ChatGPT also learns to model a specific objective.

In technical terms, that means that you need to have an explicit ML model that can be optimized for this task, which means you need to take transaction, ledger, lending, and other financial data and pair it with textual descriptions. This is a form of Fine-Tuning where you’d use the pretrained model from ChatGPT and build on top of it specifically for large scale transaction summarization or other financial tasks. In this particular case though, you’d need to update the architecture to support the data as it is not just raw tokens (i.e., a word mapped to a number) it is mostly dense, numeric “tabular” data.

This explicit architecture makes sense because ChatGPT won’t learn the idiosyncrasies of finance as a subproblem because it doesn’t have access to terabytes of sensitive financial data. If you want to optimize it for that domain, you need to train it on the appropriate data and express that formally with a model that uses the best of both worlds.

This is a fairly common practice and using pretrained models really started with Word2Vec by Google back in 2013, so I’m not proposing anything novel here.

To briefly elaborate about what a “”hybrid architecture” means, LLMs and Neural Networks, at their core, are a series of matrix multiplications on some input data. A hybrid architecture would use the learned weights from ChatGPT as starting values for training and add an additional set of weights and layers to handle the banking features (specifically to express the numeric and dense nature of money stuff). The obvious limitation here being that while ChatGPT does let you run Fine-Tuning, they definitely don’t give you access to the underlying weights.

It’s worth noting that ChatGPT or FinanceGPT as I’ve named it here will never be perfectly accurate (humans aren’t) but this would help reduce the error in a much more structured way and probably within a reasonable tolerance.

Closing Thoughts

The conclusion is simple: ChatGPT will learn superficial stuff about finance, but in order for it to become a financial expert, it needs to be explicitly trained to be.

Intuitively this makes sense because ChatGPT is a generalist about internet knowledge and a specialist in programming because it was fed that data and reinforced. Once trained on and optimized for financial data it can learn that, too.

Theoretically you could try to feed the data as is to ChatGPT’s Fine-Tuning, but you’d probably do better by changing their neural architecture to explicitly model the structure of the financial data, which, as said before, is much different than text alone.

For example, I could feed in this entire article to ChatGPT and the sequential nature of language and contextualized multi-representation of words allows for some fuzzy notion of an agenda (written by me) with some variance in interpretation (by the readers)…but I couldn’t feed in a raw database from a bank and expect the model to learn anything (mostly because the data is just random numbers) so finance is often missing context that is well understood by humans, which is why you need a language model complimented by a financial one. The first model learns linguistics and logic, the second understands the behavior of financial systems—just like quants/MLEs do.

At least, that’s how I’d do it if I had tens of millions of dollars to burn through on Azure credits.

Happy Learning!

-Francisco 🤠

Some Content Recommendations

Alex Johnson had an incredible interview with Frank Rotman. Frank is one of the brightest minds in Fintech. His balance of technical/quantitative acumen and pragmatism along with his extraordinary legacy building Capital One to the Goliath it is today, makes 45 minutes with him feel like an extraordinarily high yield usage of your time.

Alex Johnson’s article on Generative AI is largely what inspired me to write this article and I highly recommend reading it!

Ron Shevlin wrote an incredible piece on the alternative to VC for enterprise Fintech

wrote an extraordinary piece at Every on how Stripe Can't Lose. I highly recommend subscribing to his newsletter (Batch Processing) as he does very thorough analysis and has the impressive background to justify it!

wrote a pretty hot take stating "There is a strong argument to be made that the CFPB could use this authority to bring Apple and Google’s app stores under its supervision." I don't entirely disagree but I do think there are significant tradeoff decisions. Neither Apple nor Google would want that level of scrutiny as it would result in significant friction for developers creating new apps, which would ultimately decrease the utility of their respective app stores and thereby make their customers less happy. I think it's a worthwhile discussion but my guess is it'll be a very litigious one.

Macroeconomist wrote how The Fed alone cannot bring inflation down. I loved it. She’s a former Federal Reserve and White House economist and provides a terribly thoughtful view of our current economic environment.

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

The mathematics of least squares would have been so trivial for Gauss that even had he come upon the method he might have passed it over as but one of many, not noticing its true significance until Legendre’s book prodded his memory and produced a post facto priority claim.

-Stephen Stigler, Ph.D

There have been many extraordinary equations that changed the world (whether they were discovered or invented depends on whether you subscribe to mathematical Platonism—I do) but among the 17 equations that changed the world, the legendary Ordinary Least Squares (OLS) wasn’t listed among them (though it is heavily related to both the Normal Distribution and Information Theory).

It’s a shame because the article and tweets referencing the “17 Equations” have been floating around for nearly ten years.

So I will tell you about the magic of OLS, a little about its history, some of its extensions, and its applications (yes, to Fintech too).

Gauss and the Invention of Least Squares

One of my favorite authors and historical statisticians Dr. Stephen Stigler published a wonderful historical review in 1981 titled Gauss and the Invention of Least Squares. He argued that the prolific Carl Freidrich Gauss discovered OLS in 1809 and fundamentally shaped the future of science, business, and society as we know it.

But first, what is OLS and why is it important?

OLS is often referred to by many things across several different disciplines, some of them are:

• Linear Regression

• Multivariate Regression

• The Normal Equations

• Maximum Likelihood

• Method of Moments

• Singular Value Decomposition of Xw−y=U(Σ′w−U′−y)Xw−y=U(Σ′w−U′−y)

But all of them ultimately reflect the same mathematical expression and the same solution to the sum of squares of the regression equation (in scalar notation).

A little mathematics yields the famous OLS estimator (i.e., equation)

Or in matrix notation:

This equation is pure sorcery.

From it humans have been gifted the knowledge and ability to make inference about drug treatments, forecast the weather, predict genes, make empirically grounded recommendations about public health, understand the galaxies, and, yes, even develop artificially intelligent chatbots.

But it all started with OLS.

The equation basically says "Look at data about x and estimate a linear relationship with y by minimizing the average error."

As a concrete example, suppose you wanted to know the relationship between Age and Income (a simplification of the well-studied Mincer Equation, how would you figure this out? A simple linear regression could estimate that relationship and the β1 would represent the partial-correlation (sometimes called the marginal effect2 or coefficient estimate) and it exactly represents the slope of the line below.

Isn't that just amazing?

To think Gauss had discovered OLS as a method of calculating the orbits of celestial bodies (i.e., he discovered regression as a sub-problem) and that today, over 200 years later, humans would use it to for so much of what we do is astounding.

Over the years statisticians, economists, computer scientists, engineers, psychometricians, and other researchers have advanced OLS in such profound and unique ways. Some of them have been used to reflect data generated from more non-standard distributions (e.g., a Weibull distribution), or to frame analytical problems to use prior information in a structured way (e.g., through Bayesian Inference), while others have enhanced these equations to learn high-dimensional nonlinear functions (e.g., via Neural Networks or Gradient Boosting). All of which are extensions of Gauss’ extraordinary work.

Function Approximation

I showed above that regression (OLS) can be used to fit a line through two variables (e.g., Age and Income), which basically assumes you want to estimate a linear relationship…but wait there’s more!

As I said, the statistics and machine learning community did a lot of great work over the last 200+ years and now we can model more sophisticated relationships.

Let’s go back to our example of Age and Income.

In the graph above we saw that the relationship fit the data quite linearly (by the way this is by construction because I simulated the data)…but what if it wasn't linear?

What if we knew Age only increased Income to a degree and that the marginal return was decreasing? Well, maybe we'd see a plot like below.

What if things were a little less intuitive and, after another point, your Income (on average) started to go back up again? We could see something like this:

Or what if we saw something that was just, well, plain weird?

The graph above is my favorite example because it shows a piecewise linear function and, while strange looking, relationships like these are very common phenomena in the wild. That is to say that in real life (particularly with humans) correlation between things is rarely continuous, smooth, and fully linear. It’s typically discontinuous at weird points, sometimes inflated at a random point, and not as easy on the eyes.

This is because we are usually modeling behaviors, decisions, or processes by human generated systems in the world, and those systems often have weird boundary points/thresholds that dictate or mean something. In fact, I wrote about this before where I said rather explicitly that Financial data is Chaos.

Now going back to the example of Age and Income, weird trends in the data arise in the US because states have certain age minimums, some people decide to seek post-secondary education without working, some people are physically unable to work, and a bunch of other unknown things that mean something…so you see partial correlations that aren’t so clean as my simulation above or that they show you in graphs in textbooks.

Scatter plots lack nuance because they are by definition two dimensional and are only meant to elucidate a crude understanding of two things coexisting in the world. Simple things are often truisims at the aggregate level but rarely true at the individual level.

While the correlation in this example isn’t strictly linear, if you account for the nonlinear nature of it you actually end up with a fairly strong correlation.

Simpson’s Paradox, Interactions, and the Curse of Dimensionality

So we know that correlation can exist between things and that it can be both nonlinear and strong, but sometimes there’s more to the data. Often we’re only able to scratch the surface on what’s available. This is well understood in the statistical/econometric literature and is referred to as Omitted Variable Bias or Simpson’s Paradox.

In summary, when looking at data, you can always be missing one piece of information that could flip your conclusions. While that is always theoretically possible, sometimes this isn’t that likely but that doesn’t change it from being a possibility.

Said another way, what looks true in one dimension may not be true in more dimensions and the reverse.

In general, as you increase the dimensionality (i.e., the number of attributes you’re looking at for some set of data) there’s just a lot more uncertainty. Unfortunately, our brains can’t really comprehend high dimensions too well but computers absolutely can and that’s really where the fun starts.

The additional challenge here is that often, though not always, we’re faced with more information than data points. More attributes than samples or more variables than records. In Statistics, this is called the Curse of Dimensionality.

For OLS in particular, it means we can’t actually compute the OLS estimator because the underlying matrix is not invertible and, in general, the amount of data required to get robust function approximations grows exponentially with the dimensionality. In a way, more dimensions/attributes means you have to make inferences in higher dimensional spaces with less confidence.

But if you throw out your regard for the coefficients/weights (i.e., the solution to the OLS equation) explicitly and treat the whole thing as an optimization problem, you can predict something very well. This is the power of Gradient Boosting and Neural Networks.

Gradient Boosting and Neural Networks are two broad classes of algorithms that are powerful tools in learning high-dimensional, nonlinear functions.

Let’s see this with two examples.

How Computers Recognize Digits

In the picture above we have a two dimensional drawing (which I did) of the integer “2” being run through a simple Convolutional Neural Network.

Note: It’s two dimensions because there is a length and width. Most modern images are 3-dimensional because there is an explicit dimension for color (known as the RGB coordinate) but there is no color here (only grayscale) so we don’t need the RGB dimension.

The network that decided my drawing of a “2” was actually a “2” learned a nonlinear function and interactions of the pixels/data-points through the core operations of neural networks: matrix multiplications, hidden layers, activation functions, the convolution operation, stochastic gradient descent, and backpropagation.

In less jargony words, the network reads chunks of the image, makes some initial guesses about what the function is, does some multiplications and summations to try to predict what number it is, and updates the network based on how good it did at guessing.

These core ingredients are the magic that make Deep Learning so powerful. The mathematics behind it is surprisingly simple and that’s kind of what makes it so wonderful: intelligence is rooted in elegant simplicity.

To make this a little more concrete with a numerical example, the gif below shows the behavior of the convolution operation with 2 filters (weights) being applied to an image with color. Notice that the convolution is applying the red filters (W0 and W1) and bias (b0 and b1) to each RGB dimension (the 3 large left gray grids called Input Volume), which results in the 2 green grids called “Output Volume”.

This is how local correlation of pixels is exploited in the convolution operation— by sliding through the image and applying the filters. This movement helps the network measure data in images near each other and build some meaningful representation from it, similar to how humans use our eyes and visual cortex to see where a cliff ends so we don’t walk off of it.

Things can get much more sophisticated from here in Deep Learning but I won’t discuss that as I’d need to write a book to scratch the surface. I do recommend reading the Deep Learning Book for those interested.

How Computers Mine Data

I wrote before about how Neural Networks, while fantastic across many problem spaces, have not been nearly as effective on tabular data as Gradient Boosted Trees. In summary, it’s because tabular data is rather discrete in nature and typically constructed by arbitrary boundaries and Gradient Boosting Machines (GBMs) joyfully exploit this property of the data.

The underlying algorithm powering Gradient Boosted Decision Trees has 4 core ingredients:

• Greedily sorting the data and taking local averages

• Recursively partitioning the data across dimensions

• Sequentially updating the predictions by using the residual in each new iteration

• Aggregating across hundreds (or thousands) of weak models

Restating this again with less jargon (and being a little reductionist), GBMs are basically really, really intense decision trees. There are some important implementation details to make it so powerful but the essence of it really can be reduced to fancy sorting.

And this fancy sorting works wonders.

Outside of Deep Learning, XGBoost really has been the algorithm of choice when building in ML (at least in the private sector and data mining competitions) and it’s because it is just so effective.

As an example, I simulated a trivial discontinuous nonlinear two dimensional function (shown above) and trained a Neural Network and a GBM to learn it. You’ll see that XGBoost learns this function extremely quickly (after about 6 iterations) while the NN spends a lot of time oscillating back and forth.

Note: I’m sure I could tune the NN to learn this but I used both algorithms out of the box and I actually had to tune the GBM step size down to slow how quickly it learns the function. The point is how easily GBMs work out of the box.

The examples above show how GBMs and NNs can trivially learn nonlinear functions and the exciting thing is that this generalizes to higher dimensions too. For Neural Networks they can learn amazing (often smooth) structures or rich high dimensional embeddings while GBMs quickly learn complex discrete surfaces.

And the applications, as I said before, are everywhere.

Fintech and the Anarchy of Rules

If you’ve worked in Fintech, you’ll know that Rules/Decision engines are ubiquitous. During my time at Goldman we had 3 (just for one business vertical 🥲).

These systems are typically used to define rules applied to some piece(s) of data that usually map to some outcome. For example, “If Credit Score >= 630, Approve the customer for a loan”.

Inevitably, risk teams start to add lots of stuff here and what starts out as a simple set of rules becomes tens of thousands of lines of complexity. Engineers know that this is the natural evolution of any service but those not familiar with software engineering best practices may be surprised at how hard it becomes to make changes to these systems, and the inevitable cost is the speed of execution.

This spiral into chaos and anarchy happens little by little as one business analyst after the next finds a segment that can improve some metric by some percentage, and another analyst finds another subsegment, and so on…until you have a monstrosity of rules that impact your latency and your understanding of what’s actually happening in your business.

More importantly, what you have actually done is built a large decision tree—with maybe more (or less!) human intelligence reflected in some areas—which is basically to say that you’ve used several humans to define a decision boundary over multiple dimensions.

Is that optimal?

Maybe! There are a lot of regulatory constraints within Fintech that make humans defining the logic and math of these systems fairly pragmatic.

But, as we saw before, what can be true in one dimension may not be true in several dimensions, so I recommend you be extremely data driven in your meta analysis—i.e., the analysis of your analysis. This will help you learn, adapt, and iterate much faster. More importantly, it will help you balance pragmatism and good statistical machine learning practices.

To be explicit, make sure you actually go back and quantify whether the rules you’ve constructed actually ended up being statistically significant and useful, or if they were just guessing. Guessing is dumb and it will ruin your business, and we don’t want that so make sure to bust out the big guns (i.e., XGBoost). You will thank me when your vintages mature.

Conclusion

We humans try to make predictions, inferences, and guesses about the world. Often based on limited data.

This is both good and natural but we have to keep in mind that we look at data and the world through few dimensions, with imperfect information, and with—probably—a biased sample.

So exercise a healthy amount of self-skepticism and ask yourself what information you could be missing that could flip your conclusion or whether you’re just looking at a narrow segment of data where your conclusions won’t extrapolate. Over the long term, you will make better decisions and better predictions.

Happy function approximating!

-Francisco

Some Content Recommendations

and held another wonderful podcast this time covering the dismantling of Goldman’s Marcus, among other things. As always, the dynamic duo have a really great discussion. This time covering Gen Z Banking, Crypto Crashes, and some 2023 predictions. These are two of my absolute favorite writers in Fintech so getting to listen to them on a podcast is real joy.

fromwrote a fantastic article on Payment Companies and Profitability and how Adyen is crushing it.

Simon Taylor from authored another brilliant piece, this time covering The Fintech Hype Cycle. Some of my favorite parts of this were his thoughts on Generative AI (which I 100% agree with), his take on POS Payments, and his analysis of Goldman’s lending losses (which contextualizes the $1.2 billion in losses with revenue because that is the only thing that actually matters in lending).

Alex Xu from, the best newsletter for software engineering content, wrote about Designing a Chat Application, CI/CD pipelines, Visa Disputes and Chargebacks, and how to deploy services to production. It was great as always.

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.

-George E. P. Box, Ph.D.

Some History

The last ten years of machine learning and engineering have been extraordinary. We have seen unimaginable progress in computer vision, natural language processing, and tabular data modeling.

I am lucky to have started my career at a time before much of the excitement started and to have seen the innovation happen first hand. To briefly elaborate, I finished my first master’s in economics and statistics in the fall of 2011 and worked in what was called “predictive analytics and data mining”. I decided to pursue another master’s part time in data science in 2014 where I focused all of my studies on machine learning, computer science, and math. So getting the opportunity to study the field academically while working in it professionally during its glorious rise felt like (1) dumb luck and (2) a terribly good time.

So, how did we get from machine learning being mostly laughable to an extraordinary internet craze?

Things improved for three technical1 reasons:

1. Better hardware

2. Better software

3. More data

Many of the impactful algorithms that have been successful were developed in the decades before but lacked the juice necessary to do some massive computations.

As the three magical ingredients collided, a new era of machine learning was born and the world has changed significantly for it.

In the past year the release of novel models by some of the biggest AI labs (OpenAI, Meta AI, DeepMind, Google Brain) have raised the stakes further, providing what feels like yet another colossal change in machine learning.

Stable Diffusion, Multilingual Machine Translation, GPT3 (Generative Pre-trained Transformer 3), and other state of the art ML models are quite simply astonishing. Of course, they have their flaws but the good news is researchers at these labs are constantly iterating to improve them. So these models and neural architectures will get better over time, but it’s important to understand that going from the intelligence of an ant to the intelligence of a dumb human is a much greater leap than getting from a dumb human to a smart one…but these models don’t quite have the intelligence of an ant. You wouldn’t be able to take GPT-3 and plug it into a robot and suddenly watch it learn how to walk or become sentient. It relies on an extraordinary amount of engineering, so we’re not really at the “dumb human” level. We’ve just trained this model to be very good at language.

The innovation coming from these open sourced generative models is exciting. People are changing their online profile pictures to ones generated using Stable Diffusion, posting screenshots of GPT conversations, launching AI startups, but few people have mentioned any useful Fintech applications.

As a brief aside, if you are building a business with a small product on top of an open sourced model you’ll probably get copied because there are no barriers to entry—so make sure to create some if you want your business to thrive. 😅

Fintech and AI

David asked a great question (a16z wrote about this topic more broadly) and the replies to him mostly centered around synthetic data generation for fraud. Unfortunately, I don’t think that would be a great idea2 as it could be generating samples that give you useless, or worse, wrong data (and wrong data will ruin your business and we surely don’t want that in this economy 😉).

ChatBots and better customer experiences seem like a natural option, right? Sadly, no. ChatBots are often quite terrible and I’m skeptical that they would outperform basic search and curated documentation, which is mostly the illusion of a ChatBot with some slick UX and, in my opinion, is a much better alternative. (Why? More on that later.)

So what, in particular, can generative models be used for in Fintech? And what, in general, can machine learning be used for in Fintech?

Financial Data is Chaos

Finance is very different from language (NLP) and computer vision (CV). Not to trivialize all of the CV research in the past decade but the mechanism that we use to quite literally see objects in the world and represent them in Red, Green, Blue pixels stored in Tensors is rich in its structure…and also for the most part a complete and accurate representation.

By which I mean that if you see a picture of a cat and I told you “this is a cat”, that information is unambiguous. The image your brain processed can see the image again and say “this is a cat”. You don’t even have to know what a cat actually is, you just know it’s this thing. Isn’t that kind of amazing?!

With language this becomes more challenging as there is latent context missing and implicit in writing.

As an example, if I told you “Donald Trump sold NFTs this week” you may have a lot of contextual knowledge about Donald Trump the person, businessman, and former president. You also might have knowledge about what an NFT (non-fungible token) is and that “this week” means the week of December 18th-24th 2022 because you understand time. That’s a lot of hidden data in six words. Luckily word embedding and self-attention (some of the key ingredients to the success of Large Language Models) handle this information reasonably well but they have their limitations.

So what about finance data (i.e., tabular data)? The structure of financial data (especially consumer finance) is rich but in a much more non-obvious and kind of arbitrary way—i.e., the Data Generating Process is chaotic.

How? Well, we use arbitrary numbers for rules in systems that encode more numbers and rules with lots of implicit information about other random rules about dates and thresholds, which makes the structure of the data really just kind of weird.

For example, the credit bureaus store information about loans (but only from lenders that both choose to give a loan to the customer and furnish that data…and only when they want to) in a database that represents how much a person borrowed and paid back at different points in time. That data represents how the interest accrued (but not for everyone!), the dates when someone is supposed to pay it back, and the amount of credit someone has available—which are all made up numbers in a made up system with somewhat arbitrary (though governed) rules. 🥲

This isn’t to say that these numbers don’t mean anything, of course they do. It’s just everything surrounding this money stuff was kind of made up by humans to keep track of things. That is a profoundly different phenomenon than your visual cortex, linguistics, or the laws of physics.

Why does any of that matter?

1. The set of problems you can use AI for in Fintech is more narrow than other industries

2. Generative models are often wrong in silly ways and that is dangerous and potentially expensive in Fintech (maybe even criminal 😳)

3. Most industry practitioners will tell you that deep learning on credit/fraud data underperforms compared to XGBoost

Interestingly, (3) is mostly because consumer finance operates within a system that, as I mentioned, is rather discrete in nature and constructed by arbitrary boundaries which is a property that gradient boosting machines joyfully exploit.

Some fun history here: Dan Steinberg, founder of TreeNet (aqcuired by MiniTab) was, to my knowledge, the first to professionally release an implementation of Gradient Boosted Machines authored by Jerome Friedman himself 🤯. Dan first sold the software to Capital One many years ago (late 2000s?), who had been aggressively using TreeNet since. So GBMs have been in use in Fintech for over 10 years. I had the pleasure of getting to spending time with Dan back in 2013 and 2015 at both AIG and the Commonwealth Bank. He’s brilliant.

Consumer Fintech is a Math Puzzle

I still haven’t answered the question yet.

Credit and Fraud modeling are what everyone uses AI/ML for in consumer Fintech, so naturally that’s the obvious answer. Will Generative models make them better? No, they won’t. Don’t waste your time or money on it. Generating fictitious data that can harm your currently accurate models is dumb (and that’s not you!).

Will innovations in ML algorithms improve risk models? Yes but those are increasingly incremental and short lived because people catch on quickly.

Where’s the opportunity then? Engineering. 😇

The data, systems, and software surrounding artificial intelligence are rapidly changing and that’s where there’s a massive opportunity to innovate in consumer finance for risk modeling because consumer finance is basically a multivariate constrained stochastic optimization problem (that’s a mouthful).

Companies like Databricks, dbt Labs, Prefect, and Tecton are changing the industry and I think there’s still a lot of opportunity here.

I’d say the same things for consumer insurance. There are innovations happening in how data is measured (e.g., in auto insurance with tracking devices) but much of insurance still uses actuarial models for their risk modeling. This is even more true for commercal insurance and that makes sense as machine learning really isn’t the right tool for low frequency high dollar risk pricing.

Back to ChatBots

What about ChatBots and customer experiences? Surely we can use GPT3 to answer basic questions about a customer’s account? No. A pathologically extreme example would be if a consumer asked about whether they could apply to get a loan and the ChatBot said “Yes” but then didn’t give them the loan or gave them inaccurate information about the loan. The consumer would be upset and I’m sure this would result in a world of legal pain for the lender.

Suppose you asked a ChatBot for your checking account balance and it gave you the wrong number? Maybe it led you to make an overdraft, wouldn’t that make you absolutely furious? You’d be mad at the bank for using this technology on something so important (yet technically trivial)—and rightly so.

GPT3 and Neural Architectures don’t really care about individual facts in a database, they care about on average phenomena and structures that are on average true. This is why it reflects the structure of a right sentence but gets the details wrong.

Fun internet ChatBots without legal ramifications can do that, but not financial services companies, and that’s a good thing.

In short, the cost of being wrong is high and having your customer's shout (read: type in caps lock) chaotically at a ChatGPT is just pure guessing. Manually curated workflows masked behind a Conversational Form based ChatBot is a better option because you dictate the end state, not some random matrix multiplication machine3. Your legal teams will thank you.

Artificially Intelligent Marketing

Fintech and the battle to reduce CAC (Customer Acquisition Cost) is a well known challenge in the industry. Most build propensity models for Direct Mail, Email, Ads, and maybe even in-app events if you’re fancy enough. Can new generative models add value here?

Potentially! But more through enhancing copy and the ad creatives more so than the other machinery. Copy.AI, Jasper.AI, and others do a lot of this and it’s broader than Fintech alone but it does seem like a possibly useful option, though I’m not sure the gains will be significant over things people can create…unless you make an excessive combination of your ad creatives and copy…and that feels like less of a strategy and more like throwing stuff at a wall and seeing what sticks.

Churn and SQL

Customer attrition is quite well understood and we actually have enough ML tools for the problem. More importantly, I think the tools we currently have are more than what we need for the job. For understanding why your customers are leaving, I recommend (1) talking to your customers, (2) looking at your data, and (3) creating a funnel to see what in your product is broken.

No need for ML.

Recommendation Engines and the Path to BananaCard

Several years ago I spoke with a few different founders who wanted to build a recommendation engine to help consumers improve their credit so that they could get approved for a mortgage, a credit card, or whatever consumer credit product. This came after the announcement of the Path to Apple Card.

That’s not actually a recommendation engine in the technical sense but the goal is to give the consumer recommendations about how they can change their credit profile to get approved by some sort of rules system, which makes the misnomer more palatable.

This is actually much more like an optimization problem commonly solved by Newton’s Method for optimization (though I should note this is more challenging as it’s actually a multivariate optimization problem and the function isn’t always guaranteed to be twice differentiable—the approach is actually much more simple in practice).

The point is that this product feature is well solved already so I don’t think there’s much novelty here, maybe the algorithms to solve the optimization can run faster but beyond that I don’t see anything wildy new happening.

For actual recommendation engines, the machine learning systems that exist today are actually quite good already so the gains I suspect will only be incremental.

Automated Insights

Personal Financial Management Applications are heavily reliant on ML, business logic, and a miserable amount of data transformations/pipelines.

I built one myself so I know this in excruciating detail. There are companies that now offer this product as a service (e.g., Pave) and I think this could actually be a good opportunity for generative models because there’s a general meta-pattern to these things and they can assist ML Engineers in creating insights faster and maybe even more accurately. In many ways this is similar to GitHub Copilot but for Data Scientists, and I think that could be really useful.

It’s important to highlight that the Data Scientists have to be steering things as getting data insights wrong about a consumer’s data (that they have kindly permitted you to use) would be potentially harmful and we don’t want that! A concrete example would be misstating something about their current balance and making the consumer believe they could spend more than they should—that would be disastrous.

…But generating faster insights from standard transaction data could be an extremely powerful feature for Generative AI. I should note that this, again, could be used more broadly than this single use case (and is actually an active area of research called Meta-Learning).

Algorithmic Trading and Asset Management

Algorithmic trading has been using ML for a very, very long time. Two Sigma was founded in 2001, D.E. Shaw in 1988, Renaissance Technologies in 1982, and Quantitative Managament Associates in 1975. So the fight for alpha continues, maybe with short term battles left to be won, but I don’t think this area will be disrupted substantially.

Using generative models to create synthetic data is only marginally different than Monte Carlo simulations which is basically the old school approach of sampling from a multivariate Gaussian. Since there’s already published work on using Generative Adversarial Networks for trading strategies, my guess is that these aren’t that impactful. People don’t rush to publish the quantitative strategies that are the most effective. 😉

Asset Management has long used quantitative methods (i.e., statistics and machine learning) to understand risk exposure and optimize their portfolio. Some places use even more indirect ways of using machine learning to find emerging companies (I actually did some of this work back in 2015), so I expect this space to see only modest improvements as well.

Closing Thoughts

At the moment founders and venture capitalists seem to be jumping on the AI bandwagon (have we learned nothing from the NFT craze?), largely driven by the recent and dramatic innovations in AI, but that innovation was driven by large research labs funded by the largest technology companies that put in a decade (arguably more) of work and billions of dollars.

Startups will be able to use these open source models and iterate quickly with some slick UI but I am extremely skeptical that there’s a valuable venture scale business model from that alone4. The cost to build/train one of these open source models is extraordinary (i.e., millions of dollars). So tread cautiously around those selling you their AI startup or AI products as they’re either (1) using an open source model and their business is trivially exchangeable, (2) don’t know what they’re talking about (probably because of the Dunning-Kruger Effect), or (3) are lying.

Hot takes aside, the future of AI is extremely bright in Fintech5 and more broadly. Breakthroughs are continuing to happen and I have never been more excited about this ever evolving field. I am particularly bullish on the innovation in the software, data, and infrastructure surrounding these large models—that’s why I have dedicated my career6 to it.

If you are interested in staying up to date with the latest innovation in the field and less on the hype, I recommend following Yann LeCun (Chief AI Scientist at Meta, Turing Award Winner, Silver Professor at New York University, and a lot of other impressive stuff too!). He is probably the best person to follow for a broad set of information about AI. Not only is he one of the Godfathers of Deep Learning, he’s also quite active on social media giving a wonderful perspective about the history of many innovations (back when AI was boring).

I will leave you with his brilliance.

Most of human and animal learning is unsupervised learning. If intelligence was a cake, unsupervised learning would be the cake, supervised learning would be the icing on the cake, and reinforcement learning would be the cherry on the cake. We know how to make the icing and the cherry, but we don’t know how to make the cake. We need to solve the unsupervised learning problem before we can even think of getting to true AI.

-Yann LeCunn

Happy innovating.

-Francisco 🤠

Some Content Recommendations

’s latest from reviewed his 2022 predictions, most of which centered on crypto getting absolutely wrecked, was fantastic. His predictions were quite accurate. Certainly at the time the consensus opinions were that Crypto was going to be fine but things have only gotten worse. That said, I’m certainly optimistic about crypto’s future (those that do real work in the space and not embezzle money, of course).

Alex Johnson at wrote about Chime’s attempted acquisition of DailyPay, a new Fintech focusing on founders’ retirement, and Goldman layoffs at Marcus. He makes some good points about why Marcus struggled and some of the fight for attention was true based on my experience working there. Things changed quite dramatically under Solomon’s leadership compared to Blankfein’s (who initially sponsored the initiative). Broader GS didn’t really like a product that lost money whatsoever and they’ve been pulling back on it over the last few years, so it’s unfortunately not surprising. It’s a shame as Marcus’ Personal Loan product was truly innovative and consumer friendly when it launched, but Solomon is facing a lot of market pressure and it’s mostly because GS struggles to be as nimble as other competitors due to typical internal corporate bickering. During my time there, Lloyd had some pretty explicit sheltering to avoid too much internal interference—that changed dramatically after launch and probably caused much of the problems. To be explicit, there was just bad partnership across the firm due to competing incentives, and my honest take is that Marcus just stopped innovating as it grew. C'est la vie.

by Simon Taylor wrote a beautiful piece about the 2023 State of Fintech. I absolutely loved it and highly recommend reading it as it’s so thoughtful and balanced about where the industry is and where there is opportunity.

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

“All we have to decide is what to do with the time that is given us.”

— J.R.R. Tolkien, The Fellowship of the Ring

On the Finiteness of Time

Not even the world’s greatest fortune can buy back a second of time.

We are all commonly bound by 24 hours, the question then is: what shall we do with our precious minutes as they reduce ever-closer to zero?

Maybe that’s dramatic…maybe not.

The Work and the Soul

Unproductive strategy meetings and corporate political drama wear heavy on the soul, and I often found myself drained early in my career when I felt that I was spending 80 percent of my time planning and twenty percent of my time doing actual work. This is because I don’t care to waste my time on things that don’t matter.

Throughout my career, there have been many moments where things, committees, or childish corporate drama have interfered with my goals and the goals of the institutions for which I’ve worked. When things like this begin to occur too frequently, I generally change roles because I don’t particularly care to work with children, as that was not my professional aspiration.

So, I care about execution a lot: my execution, my team’s execution, and the execution of the company for which I work. But not everyone does, and my conjecture is that over time companies that forget to focus on what matters (i.e., a pretty good strategy and relentless execution) will fold to a competitor who is more nimble and focused on eating the complacent’s lunch.

Your Margin is Someone’s Opportunity

Start-ups live and die by execution. If you can’t execute to deliver a differentiated product or acquire new customers—and do so quickly—your business will fail. I saw it happen first hand when Fast imploded.

Product market fit is forged, not found.

-Keith Rabois

This hyperfocus on execution is a direct consequence of venture capital because, for the most part, start-ups are quite literally running out of money the entire time they’re operating, which is a great incentive for getting something done.

Established incumbents, on the other hand, don’t have the same incentive to execute. Instead, large corporations tend to focus on the following:

• Protecting existing revenue

• Minimizing short-term risk

• Strategizing against competitors

• Dealing with internal corporate bickering

When a company is large enough and has sufficient scale, these priorities seem quite reasonable. But the executives focusing on them have forgotten that the world is changing—quickly. Software and innovation compound in a way that lets businesses—particularly technology companies—leave their competition in the dust. So, while risk-aversion may be an acceptable strategy in the short term, it’s certainly not a good business strategy in the long term.

This behavior is not true of all big companies, and I believe there is a new era of management focused on disrupting the legacy model of death-by-endless-PowerPoint-meetings.

In fact, we are living through the transformation of one company at this moment as Elon Musk, one of the greatest capitalists of all time, takes over Twitter and explicitly addresses meetings and work output.

But this all leads me to a natural question: why exactly do big businesses place so much emphasis on strategy and meetings?

Alignment Feels Good

When you have a large organization, you typically have to spend a lot of time checking off boxes, building consensus, strategizing, attending meetings, and managing stakeholders. 

These are all reasonable things within the context of a large company that’s trying to achieve alignment across several different teams and departments, and it certainly feels good to show off a fancy PowerPoint during a meeting or cultivate a sense of “togetherness.”

But what feels good and what is actually good are rarely the same thing (in fact, I’d generally argue that the opposite).

This becomes clear when you consider that planning and alignment related tasks are never free. They cost the business time, energy, and often employee satisfaction. Moreover, these tasks are often completely unrelated to the core skill sets of the people doing the work, and they also may not even be correlated to the success of the work in question.

That’s not to say planning, alignment, and strategizing are bad—that would be stupid. But there is an optimal balance between planning and execution that most teams haven’t yet reached.

The Execution-Planning Tradeoff

This diagram says a lot but, in short, you are probably over-planning.

To elaborate, there is a 1-to-1 tradeoff between planning and executing. If we think about the work that we produce, we have to do stuff to generate it. As I mentioned before, we certainly have to gather requirements, find customer pain points, and then build a product, feature, ad creative, or something else. I would categorize all of this work as executing.

Everything else that is focused on getting consensus from some other body of people (e.g., managers or executives), socializing an idea, or appeasing some person who relents about how they don’t like something about what you’re doing belongs in the “planning” bucket. This categorization is not perfect of course, but I think you get the idea.

Obviously doing zero planning would be reckless and doing zero executing would probably get you fired but the ambiguity of all of the time between is where souls go to die.

And, as I said before, you have to really be reflective and evaluate whether or not the meetings you’re holding or are in are actually constructive to the output you’re trying to produce or whether it’s just time wasted trying to make people feel good.

Work that is hard is frustrating, mentally taxing, and stressful. This is good. This is the body telling you that what you are doing is worthwhile and not trivial.

Closing Thoughts

Existence is a perpetual search for meaning and I long to be a part of something beautiful.

Building meaningful relationships brings me joy but when I’m at work, I want to do work that is impactful—not really socialize1.

So I ask that you reflect on how you allocate your time, how you take time from your colleagues, and how that impacts the work you do because time is all we have and we have to be relentless with managing it.

Happy executing.

-Francisco 🤠

Some Recommended Readings

Now is the Time to Double Down on Fintech by is perfect. It's also a great reminder of why I'm excited to be a limited partner of The Fintech Fund, if you’re a fintech company raising money, talk to Nik.

Goldman Sachs Research published their analysis of Powell’s latest Federal Open Market Committee (FOMC) speech where they estimate the federal funds rate will likely peak in 2023 between 5-5.25%. It’s worth reviewing and it’s also worth listening to the speech itself by Powell.

and Alex Johnson recorded a great Fintech Recap podcast and talked about the latest drama in Fintech, it was a fun listen and they always give great insight.

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter!

What is Customer Segmentation?

Customer segmentation (sometimes called Market Segmentation) is ubiqutous in the private sector. We think about bucketing people into k mutually exclusive and collectively exhausting (MECE) groups. The premise being that instead of having 1 strategy for delivering a product or experience, providing k experiences or strategies will yield much better engagement or acquisition from our customers.

Generally speaking, this makes sense; it's intuitive. Provide people a more curated experience and they will enjoy it more...and the more personalized the better.

Netflix, Spotify, YouTube, Twiter, Instagram, and all of the big tech companies have mastered personalization by using robust, computationally intensive, and sophisticated machine learning pipelines. But the world has been doing this for a long time—albeit much less sophisticated.

So I thought I'd give a technical demo of what customer segmentation looks like in a basic way using a fun trick I learned a long time ago.

Approaches to Customer Segmentation

The phrase "Customer Segments" tends to mean different things across different industries, organizations, and even across business functions (e.g., marketing, risk, product, etc.).

As an example, for a consumer products retailer, they may refer to customer segments using both demographic information or their purchase behavior, where a lender may refer to their segments based on credit score bands. While very meaningfully different from a business perspective, the same algorithms can be used for both problems.

Analytically speaking, I've seen Customer Segments defined really in two main ways: (1) Business Segments and (2) Algorithmic Segments. Usually executives refer to their segments in the first category and data scientists focus on the second. The first is really important organizationally because 99% of the people working with your customers don't care about how you bucketed them and customers are the most important thing, always.

...but how do you actually (i.e., in code and data) get to those segments?

Business Segmentation

Business segmentation tends to be defined by heuristics and things that make common sense. They are often built on things that are aligned with the goal of the business.

Here are some examples of segments defined by common sense:

• The age of the customer (in years)

• The income of the customer (in dollars or thousands of dollars)

• The amount of money a customer spent in the last year

• The likelihood a customer will spend money at a given store (propensity to buy)

• The customer's geographic region (e.g., zipcode, state, region)

In data, some of that customer information would look something like this:

And so on.

We could apply some logic/rules/code to create segments like:

• Age Segments

  1. < 25

  2. 25-35

  3. 35-55

  4. 55+

• Income Segments

  1. < $25K

  2. $25K-50K

  3. $50K-100K

  4. $100-150K

  5. $150K+

• Purchase Propensity Segments

  1. Low: [0, 0.25]

  2. Medium: [0.25, 0.75]

  3. High: [0.75, 1.0]

And map that logic into our data, which would look something like this:

And so on.

Pretty simple, right? The code for this categorization is simple too (assuming you're using Pandas and Python; though it's also simple in SQL).

# Here's one example
import numpy as np
import pandas as pd

cdf['Income Bucket'] = pd.cut(cdf['Annual Income ($K)'], 
    bins=[0, 25, 35, 55, np.inf], 
    labels=['<25', '25-35', '35-55', '55+']
)

This is a really helpful and simple way to understand our customers and it's the way that most businesses do analytics, but we can do more. 😊🤓

Algorithmic Segmentation

Segments defined using simple business logic are great because they are so easy to interpret, but that's not free. By favoring simplicity we have to limit ourselves to (potentially) suboptimal segments. This is typically on purpose but, again, we can do better.

So, how do we do better?

Cue statistics, data mining, analytics, machine learning, or whatever it's called this week. More specifically, we can use the classic K-Means Clustering algorithm to learn an optimal set of segments given some set of data.

To skip over many important details (that you can read more about here), K-Means is an algorithm that optimally buckets/splits/partitions your data into K groups (according to a specific mathematical function called the euclidean distance). It's a classic approach and tends to work quite well in practice (there are a ton of other neat clustering algorithms) but one some challenges are (1) choosing K and (2) explaining what a single cluster actually means to literally anyone else.

Solving (1) is relatively straight-forward. You can run K-means for some number of K from [1, m] (m>0) and choose what appears to be a K that sufficiently minimizes the within-cluster sum-of-squares. In the graph below, notice that the the majority of the variation of the clusters can be capture by k=6.

Now to (2), which is the harder challenge; i.e., telling a non-technical person what the hell your K-means cluster means other than “idk a group or something”.

Let’s start by visualizing the data to look at the clusters.

How cool, right? This little algorithm learned pretty clear groups that you can see rather obviously in the data. Impressive! And also useless to your stakeholders.

More seriously, while you can see these clusters, you can't actually extract a clear description from it, which makes interpreting it really, really hard when you go past 3 dimensions.

So what can you do to make this slightly more meaningful?

K-Means 🤝 Decision Trees

Cue decision trees. Another elegant, classic, and amazing algorithm. Decision Trees basically split up your data using simple if-else statements that are learned through sorting on steroids. So, a trick that you can use is to take the clusters from K-means and run a Multi-classification Decision Tree to predict the segment and use the learneed tree's logic as your new business logic.

I find this little trick pretty fun and effective since I can more easily describe how a machine learned a segment and I can also inspect it.

If you run the decision tree on this learned K-means, this is what the output looks like:

There you have it, now you have a segmentation that is closer to optimal and somewhat easier to interpret.

It's still not as readable as the heuristic approach but you could actually read through this and eventually come up with a hybrid which is why I've used it in the past. Importantly, you can dump this into some SQL logic and use it for all sorts of other impactful work (I’ve shared the code below for reference1).

import pydotplus
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import export_graphviz
from sklearn.externals.six import StringIO  

optimal_clusters = 6
# 6 clusters 6 colors
xcolors = ['red', 'green', 'blue', 'orange', 'purple', 'gray']

# Chose 6 as the best number of clusters
kmeans_model = KMeans(n_clusters = optimal_clusters,init='k-means++', n_init = 10, max_iter=300, tol=0.0001, random_state= 111, algorithm='elkan')
kmeans_model.fit(X1)
cdf['pred_cluster_kmeans'] = kmeans_model.labels_
centroids = kmeans_model.cluster_centers_

display(
    pd.DataFrame(
        cdf['pred_cluster_kmeans'].value_counts(normalize=True)
    )
)

# Train Decision Tree Classifer
clf = DecisionTreeClassifier()
clf = clf.fit(X1, cdf['pred_cluster_kmeans'])
# Predict the response for test dataset
cdf['pred_class_dtree'] = clf.predict(X1)

display(
    pd.crosstab(
        cdf['pred_cluster_kmeans'], 
        cdf['pred_class_dtree']
    )
)
dot_data = StringIO()
classnames = cdf['pred_cluster_kmeans'].unique().astype(str).tolist()
export_graphviz(
    decision_tree=clf, 
    out_file=dot_data,  
    filled=True, 
    rounded=False,
    impurity=False,
    special_characters=True, 
    feature_names=xcol_labels,
    class_names=classnames,
)

graph = pydotplus.graph_from_dot_data(dot_data.getvalue())  
graph.write_png("./decisiontree.png")

What can you do with your new segments?

Now that you have your customer segments you can do all sorts of different things.

You can create A/B tests for site experiences, test the impact of changing prices to certain customers, or send different email campaigns to improve engagement.

In general, you can iterate differently for different groups.

How do I know if my segments are accurate?

The metric we used in the example above (i.e., the within cluster sum-of-squares / inertia) was a reasonably straightforward way to measure the accuracy of your segments from an analytical perspective, but if you wanted to take a closer look, I'd recommend reviewing individual users in each segment.

That sounds a little silly and can lead to the wrong conclusions in some cases but I firmly believe that in data science, you just have to really look at your data. You learn a lot from it, and it’s, surprisingly, often forgotten as a useful step in the work.

Unfortuantely, with clustering and segmentation there isn’t a single metric that you can use to say “these are the best”, most approaches have some caveats and work pretty well but there’s a bit of “hey, this looks good enough 🤷.”

How do I know when my segments need to change?

Segments typically change; your customers are always evolving so it's good to re-evaluate them time and again. The emergence of new segments should feel very obvious, since it may be driven by product, acquisition, or business changes.

As a concrete example, if you noticed that important businesss metrics split by your segments are starting to behave a little differently, then you can investigate whether it's driven by a change in the segments; sometimes it is, sometimes it isn’t.

Closing Thoughts

Customer segmentation is a powerful tool and can be made more efficient with some pretty standard algorithms so I hope you’ve found this overview useful. I want to restate the importance of defining your segments because they should always be specific to your business and problem.

I also want to mention that here we’ve discussed segments where you don’t have an explicit metric to optimize. If you had a specific goal in mind, e.g., understanding which segments have lower risk or higher likelihood to buy something, then you’re usually better off building a statistical/machine learning model to explicitly optimize it (assuming you have the historical data available).

Lastly, I've stored the code to reproduce this example in a Jupyter Notebook available on my GitHub (note to render the interactive 3D visualization you have to run the notebook). To get it up and running you only need to download the notebook, download the data, install Docker, and run:

docker run -it -p 8888:8888 -v ~/path/to/your/folder/:/home/jovyan/work --rm --name jupyter jupyter/scipy-notebook:17aba6048f44

And you should be good to go.

Happy segmenting!

-FJA

Some Content Recommendations

What is Data Engineering? by and is an excellent piece for those wanting to learn more about one of the most important roles for technology companies.

Thanksgiving Mailbag by was an excellent response to some fun questions on Twitter. Thanks for answering mine!

Payment Cards Deep Dive by was awesome and I learned a lot from it, I highly recommend it!

Data Systems Tend Towards Production by Ian Macomber writes an exceptional piece on how good data work inevitably ends in a production system. It’s a great piece and an important topic I’ll write about in the future.

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter! 🤠

I previously wrote about how Fintech recently got wrecked and, while I ended the article rather optimistically, I didn’t elaborate much on specifically which areas of Fintech I think will be exciting in the near and medium term.

So I wanted to spend today’s article on some predictions of things that I think will be successful in 2023 (and maybe even a little after).

Consensus in Fintech

I find myself often disagreeing with many consensus opinions in Fintech—largely because they are pinned to a current point in time and because both venture capitalists and founders are not immune to whatever is trendy this week (*cough* NFTs *cough*).

A couple of years ago the industry was raving over consumer Neobanks and e-commerce, then B2B fintech/corporate credit cards, and then Web3/Crypto/DeFi. Money 20/20 was, apparently, quite heavily representative of B2B Fintech.

So consumer has gotten a lot less excitement. In the wake of public consumer Fintech companies taking a beating, I understand…but that doesn’t make it rational. It’s worth remembering that all of these B2B companies will need to have a consumer Fintech customer that reaches sufficient scale to make themselves VC worthy.

I think much of current consensus is based on the learnings from the last 2 years of IPOs and acquisitions, which is useful information but the businesses that had successful exits weren’t launched recently so taking the learnings from the exits and applying them to the current selection of startups doesn’t make sense.

To be explicit: the companies that are successful today from an IPO or major acquisition were built years ago under extremely different technology and market conditions—in many cases when no one was paying attention.

So, it seems that the Twitter consensus isn’t very tempered with its thinking, which is good news because that means people are missing out on an opportunity.

Natural question: where’s the opportunity?

Opportunity in Context

To identify opportunity in startups I like to contextualize the company and the macroeconomic environment, which will drive incentives and aggregate consumer behavior.

With respect to the company, it’s important to know: (1) what product offerings exist in the market today, (2) which products are missing or underdeveloped, (3) which products are novel to the target consumer, (3) which products have significant market penetration, and (4) if the company can create barriers to entry in the medium term.

These components will dictate how impactful a startup can be in the face of incumbents.

Then you need to consider the broader macroeconomic environment which, in the near term, means there will be reduced consumer spending, a preference for saving money and investing in lower risk assets, and a rise in consumer delinquencies.

This does not mean that the economy is shutting down but it does mean the aggregate macroeconomic trends will move slower than before.

People will still spend money, just not as much on leisure goods but technology will continue to permeate throughout our society, and there will be growth in many sectors—just not on average.

So with that context, here is my evaluation of some of the areas in Fintech and what I think will have successful business outcomes in the years to come.

Consumer Banking

Ah, the excitement for Neobanks seems to be dead, which feels rather short-lived.

My honest opinion is that it’s silly for people to pull too far back from them, which effectively represents a direct relationship to the consumer. But in a high interest rate environment where cost of funds provide margin pressure, I certainly understand this reaction. But looking at interchange as the only revenue source is the mistake here.

Neobanks need to innovate because offering a PFM, debit card, or credit card won’t be enough if all you’re doing is adding a different colored UX and some emojis on top of your BaaS provider.

Neobanks that solve a hard problem for their customers (e.g., offering a bank in another language, which is surprisingly challenging) will thrive. Anyone who doesn’t do anything novel will probably explode, unless their marketing is truly legendary (since acquisition costs are notoriously high in Fintech).

So I’m bullish on some Neobanks (e.g., Daylight and Maza are two I’ve personally invested in that I’m extremely excited about) but the aforementioned challenges are important to consider.

Consumer Lending

In Fintech Reckoning I wrote about how great lending businesses are built during downturns but it really depends on your portfolio mix (because math, lol).

If you’re a new lender with few loans on book or have a short average loan duration and loan size (plus thoughtful underwriting obviously), you’ll be in a great position. But if you have a long loan duration and high average loan size, you will probably be in for some terrible vintages.

If you’re a lender without great distribution, no product novelty, or deposits reducing your cost of funds, I’m skeptical you’ll do very well in this environment.

So the opposite obviously applies; i.e., if you have cheap costs of funds (via deposits and a banking license) or product novelty and utility (e.g., BNPL—note that I have a clear bias here), you’ll probably do well.

Consumer Real Estate

The obvious conclusion is that “interest rates go up, housing go down” is a little too simple in my opinion.

That said, Fintechs in the real-estate business have faced significant pain as interest rates increased, one may argue a little too extreme, but as home prices cool and the economy stabilizes I actually think mortgage buyer activity will start to grow as homes become more affordable from a price perspective (though this means consumers with more cash will have an advantage).

It’s worth remembering that the majority of Americans became priced out1 of homes in the most recent home pricing chaos (heavily driven by investment homes). Middle and lower income groups have seen a strong labor market and increases in wages, which reinforces my belief that they will do better than consensus suggests, so I’m slightly optimistic here and I really hope consumers are able to afford homes as prices finally decrease.

Commercial Neobanks (aka Corporate Cards)

This space was quite exciting before and it will certainly get more competitive but is there an opportunity for new startups? I’m not so sure but I think the market will continue to get competitive for scale-ups (e.g., Ramp, Brex, Mercury) as they look to compete with the giants of the world (e.g., AmEx). So I’m optimistic for the scale-ups but less so for the startups.

One interesting and exciting change about this area is that some of these players (e.g., Ramp) have begun to introduce vertical products that are eating into other parts of the industry (e.g., expense management). This makes me especially optimistic for scale-ups as they continue to innovate against the established behemoths.

Consumer Trading, Investing, and Asset Management

We are in a post-meme stock era and I think that’s probably healthy. Meme-stocks and Reddit’s Wall Street Bets will probably still stir the pot with some occasional chaos in public markets but I imagine traders and quant-shops will incorporate this chaos into their models and that the frequency will be far lower.

Beyond the obvious, retail investors are moving away from equities as rates rise, so my advice to startups working in retail trading is to offer the ability to buy bonds and other low risk assets, because consumers are going to want them. So, while I have been impressed by the innovation in this space, I’m a little more pessimistic because of the current incentives in the market.

Collections

Historically collections has been an underfunded area of Fintech but during economic downturns it becomes increasingly important.

To elaborate, as people go delinquent (i.e., don’t repay their loans) lenders have to handle it with some process, this is called collections.

For many lenders it’s often done in-house poorly, or in predatory ways by existing shady third party providers, but TrueAccord (and some places that I’ve worked!) are much more consumer friendly.

TrueAccord gives dignity and control to their customers and offers an incredible service for those managing debt (full disclosure: I’m a proud investor). For those unfamiliar, I highly recommend Packy McCormick’s incredible piece on TrueAccord.

There are two reasons I think this space will be big in the years to come: (1) as mentioned, delinquencies, unfortunately, will increase and consumers will need a way to manage their debt and (2) most of collectors have notoriously lacked compassion when working with consumers in delinquency which means the space is especially ripe for disruption.

Most consumers aren’t taking out loans with the diabolic intention of charging off or going delinquent (outside of fraud of course), they just end up in challenging circumstances and make hard choices about who they can pay back. So empowering the consumer to manage their debt better and increasing repayments is a win-win, so I am optimistic on this space in general and TrueAccord in particular (okay enough plugging).

Crypto

Two things can be true: (1) there are lots of scams in crypto (*cough* Sam Bankman-Fried *cough*) and (2) there’s a lot of incredible opportunity/innovation in the space.

I’m not an expert in crypto, there are far smarter people that I’ll trust to give insights here but I will say people who think crypto is going to die after the latest fiasco will probably be wrong as that’s a pretty extreme view.

Consumer and Commercial Insurance

The insurance business is great because you end up cash rich as your customers pay their premiums (excluding your reserve requirements), which performs favorably in a high interest rate environment but if you can’t underwrite well you will surely be in a bad place.

Hippo and Lemonade Insurance, like other public Fintechs, have taken a colossal beating in the market.

Hippo operates within the commercial market and Lemonade in the consumer. I spent the early part of my career working at AIG building machine learning models to underwrite commercial businesses and I can say it’s an extremely challenging problem, one I’m not entirely confident Hippo will be able to manage.

I am much more enthusiastic about consumer insurance because the volume of policies and data tend to be higher and the claims tend to be smaller (at least compared to commercial), which lends itself to benefiting significantly from the advances in software and machine learning.

I think startups can be disruptive in commercial insurance too but I’m much more bullish on consumer.

Fintech Infrastructure

This is a broad space with lots of sub-areas (from payments, core banking, servicing, rules engines, and more) and lots of money recently.

I think so long as each company is able to empower their customers to grow during these turbulent times it will be a successful space. But if their consumers’ business (i.e., the customers of their infrastructure) aren’t thriving it’s unlikely they will too.

So I’m excited for some new companies in the space but skeptical on some of the older players that are notorious for failing to innovate and moving like a sloth.

Data Providers

Plaid, Argyle (disclosure: I’m an investor), Pinwheel, and others will continue to flourish because, in general, you need good data to send money on the internet.

As Alex Johnson excellently pointed out, Fintech needs to aggressively defend itself from fraud. As profitability and sustainable growth becomes a greater focus for the entire industry, fighting fraud becomes increasingly more important (and it’s quite hard!). So I’m especially bullish on this area.

Existing Banks

Banks that take this time to double down on technology, acquire startups at bargain prices, and hire top talent looking for stability will do well. I cannot scream this next point loud enough, so let me bold it and italicize it for emphasis.

The world is changing, technology will not go away. Every piece of the banking interaction will be run by some piece of technology and the worse you are at the software, the more likely your customer will churn. Be warned.

Banks holding off on their technology investments will regret it.

Lastly, it really isn’t enough to buy tech startups and try to internalize them and make them good products, you have to invest in the people and culture of innovation. Goldman Sachs did this and pretty much ruined the products they bought (lol).

Closing Thoughts

Some clear word for startups:

If you do not solve a hard problem you will die. The End.

That’s a little dramatic but it’s actually the truth in my opinion. There’s a lot of competition in the market these days and many companies look very similar under the hood, whether it’s from having the same BaaS, underlying data for underwriting, or sharing a bank partner. All of that similarity to me means that the companies are exchangeable and if they’re exchangeable then there’s little incentive for a customer to prefer one company over the other.

Capitalism tends to be unforgiving when companies lack a competitive advantage, which is why I emphasize solving a hard problem. Companies that successfully IPOd in the early 2020’s all focused on and solved hard problems many years ago, which gave them their moat.

Having a provider solve most of your problems for you is great but it means you don’t have a business that is well differentiated and that’s really not sustainable for long term success, so make sure your work is hard! Plus that’s the only work worth doing anyways. 😉

I know it’s a challenging time for many of us working in this area but, as I said before, it’s an exciting time to build. So I’m sending everyone best wishes and I’m rooting for you to launch amazing stuff.

Thanks for reading and Happy Thanksgiving. 🤠 🦃

-Francisco

Happy Turkey Day!

I wanted to take a moment to say thank you to everyone who reads this newsletter. I find it brings me lots of joy to scribble my thoughts down and I’ve received a shocking amount of support from the Fintech community and people that I don’t even know so I wanted to express my gratitude to all of you that read and share this newsletter: sincerely, thank you. I’m blown away by how fast this newsletter has grown. 🥹🙏

Some Recommended Readings

Reggie Young at Fintech Law TLDR wrote a terribly thoughtful piece on the CFPB’s recent Open Banking proposal. I cannot recommend reading it enough.

I cited Alex Johnson’s article above but I wanted to call it out explicitly. He wrote about the Fintech Steroid Era and discussed the recent history of lots of capital incentivizing fraud in consumer banks. As always, it’s just great writing.

Jason Mikula at Fintech Business Weekly covered the latest startup (Tellus) taking on more UDAAP risk than they should as well as Nirvana Money and Stilt’s shutdown 🙁.

Alex Xu wrote a short and intuitive post on how gRPC works, it’s a web framework that has been growing in popularity since Google open sourced it and it’s just so well done (I’m a big fan and I think it’s exceptionally cool).

Ron Shevlin’s Digital Banking Didn’t Kill Bank Branches but Chatbots Will is excellent. A bold and accurate statement “…Digital banking isn’t that good.” Which is implicit in what I’ve tried to say above but I’m optimistic that we can continue to improve it and I think chatbots actually will play a huge role in that. Maybe I’ll go over why chatbots have failed in a separate article.

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter!

Do not pray for easy lives, pray to be stronger men.

-John F. Kennedy

It has been a painful week, indeed, for those of us working in the technology sector. Workforce reductions left and right, public valuations continue to endure violence, and the rest of us left with a sense of fear and helplessness.

According to Layoffs.fyi, technology companies had ~103,906 employees laid off in 20221. That’s a lot.

For those of us in technology or corporate America we may see a number like this and be fooled into thinking it’s reflective of the broader economy—it’s not.

In September 2022, there were roughly 159 million employed people in the US.

The unemployment rate for the month of October, announced this past week, was estimated to be 3.7% which was a 0.2% point increase. This is what the Federal Reserve is aiming for but with increases in wage growth it‘s not enough cooling.

The Federal Reserve raised interest rates by another historic amount and signaled it would likely continue raising rates higher than previously expected, which means there is further pain to come.

Which leaves one major question: what should you do?

Maximum Effort

Now is not the time to panic, be afraid, or be frozen with anxiety.

Now is the time to build like you’ve never built before.

The only thing we can do to weather the storm ahead is focus on our customers and provide them with enough value that they choose us as their budgets tighten.

I previously wrote that challenging times are on the horizon but companies, careers, and fortunes are made during hard times. Building businesses and innovating when times are good is great but thriving during hardship is what separates great from legendary.

Now is the time to be bold while others are fearful, whether that is capitalizing on an incredible talent pool, launching your own startup while dry powder is waiting to be deployed, or picking up the pace at a bogged down organization.

I’m confident that businesses, technology, and people in general will come back from this painful moment in time stronger for having endured it—something I feel this country did after the 2008 financial crisis.

So get ready for maximum effort.

Extraordinary Talent and Who’s Hiring

My heart goes out to all those impacted through job loss or economic hardship.

For those that are looking for roles, I’ve curated some job boards below and I invite you to review them or share them with folks who will find it useful.

• This Week in Fintech’s Talent Collective

• ByteByteGo’s Talent Collective

• The Pragmatic Engineer’s Talent Collective

• Better Tomorrow Venture’s Job Board

• Empire Startups Job Board

For those that are hiring, Patrick Collison (the CEO of Stripe) has provided a way to contact former Stripe employees, Rex Salisbury has provided a great list of some of them, and there’s more talented people from Twitter, Stripe, Chime, Opendoor, and Lyft that are on the market.

Recommended Reading

Alex Johnson knocked it out of the park with his latest What is a Bank?

FT Partners released the latest installment of their "CEO Monthly FinTech Market Update & Analysis", which can be downloaded here.

Jason Mikula’s Money 20/20 Pre-Cap and Money 20/20: Fintech Forced to Grow Up were excellent as usual and well worth your listen and your read. As an engineer, I probably won’t ever go to Money 20/20 as it’s a much better fit for founders, venture capitalists, business development and sales executives but it’s great to get a gist of what’s going on.

This Week in Fintech’s latest article highlighted that Fintech funding dropped 38% this past quarter, which is intuitive given the cooling market.

Congrats to the Fintech Update folks on their first 1,000 subscribers! An incredible milestone!

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter!

“That's one thing Earthlings might learn to do, if they tried hard enough: Ignore the awful times and concentrate on the good ones.”

― Kurt Vonnegut, Slaughterhouse-Five

Some history

I was 20 years old when I used my first credit card. One night in college I was working as a pizza delivery driver when my car broke down. It was late, there was a pothole, and my tire quite literally snapped off. I was sitting on the side of the road trying to call a tow truck and thinking about how I would be able to work tomorrow–worried if I’d lose my job–and how I’d afford rent if I didn’t work. Finally, the car got to the shop and they told me it would cost two thousand dollars to repair it. I didn’t have that kind of money.

I remember how scared I was not knowing how I could afford to repair my car, but then I remembered my credit card could provide me with the liquidity I needed. It wasn’t the cheapest money I could have used but I didn’t know any better and, regardless, the resolution of my crippling stress, anxiety, and fear was well worth the price I paid in interest.

I didn’t know much then but I knew that money helped me.

Fast forward to 2017. That fearful moment set me on a path to understand how finance worked and make it more accessible for others. I graduated and went on to get a masters in economics and statistics, and eventually another in data science and machine learning. I worked at financial institutions like AIG and the Commonwealth Bank of Australia. 

Then a moment came that felt rather significant: I was approached by Goldman Sachs to join their consumer bank, Marcus.

It was a great opportunity not because it was Goldman but because I believed in the product they were building: a simple loan with transparent costs and no fees. What a wild product idea: transparency. Somewhat shockingly, this was novel at the time as many lenders had hidden fees and costs but Marcus is consumer-centric and I loved that.

I truly did love my time there and I actually thought I’d never leave...but I couldn’t help but notice a common theme in postmortems we were doing on non-performing loans: many of them were originated by Hispanics/Latinos. To be fair, they were not the worst performing loans, but as a Latino myself, these data points stood out to me.

As someone who grew up working class with immigrant parents from Mexico in the South side of Chicago, I thought to myself “Maybe I could build a product to help my community.” On paper, given my finance, technology, and personal background I was (arguably) uniquely positioned to do it.

So in 2019, I left Goldman to start my own company: Unidos. I built a personal financial management tool in both Spanish and English that aimed to educate and empower my community with a simple understanding of finance and interest. After 3 years, I can objectively say it was an utter business failure.

Which is the primary reason I am writing this article: to vent all of the mistakes I made so that, hopefully, you (and maybe even future me) can avoid them.

Remember, most startups fail which means that in probability if you are going to launch a startup, you will probably fail, too.

Be bold. Reject this data anyways but avoid the mistakes I once made.

All of the ways I failed as a founder 🥲

Some important facts about my experience as a founder:

• I did not have any previous experience as a founder

• I did have lots of experience at big companies

• I was a very technical founder

• I had significant misalignment with my cofounder

• I was working full-time while doing my startup

• I did not ever try to raise any venture capital1

I launched the first version of the product (a web app) in 2020 and you’ll see in the photo below that I wrote a lot of code in that time…all of which only aimed to solve for what I thought my parents and I needed when we were learning about the US financial system, which was an abhorrent waste of time and obviously a very narrow target market.

I used to rationalize not connecting with customers before having a product by telling myself I was being product-centric but that was a lie. At the time I felt I had conviction and I was focusing on solving a real problem with a real solution (look at all that code! that must be something, right?!), but this was dogmatic and wrong…in reality I should have been both product-centric and customer-obsessed.

Eventually I made some course corrections and launched the mobile app in 2021, but I still made most of the same mistakes.

So my startup failed twice, slowly, painfully, and quietly2…and that taught me a valuable lesson.

It is better to fail fast than to die slow. 😉

-Me

With that context, here’s the list of ways I failed:

1. Kept my startup idea a secret

I, like some other first-time founders, thought that I needed to keep my idea a secret so that no one would steal it. This was absolutely stupid. I should have told it to as many people as possible and gotten feedback, maybe they would have convinced me not to do it. More importantly, ideas are free. Words are cheap. Delivering something valuable is hard. Execution is everything.

2. Didn’t talk to customers first

“Build a product and they will come” may be true for some but it’s probably not true for you and it definitely wasn’t for me. I didn’t talk to my target population at all and this really matters. I was so focused on building a product first that I missed this entirely. I often told myself “If I show my target customers this great product they’ll think ‘Oh I need this!’.” Wrong.

I was in that position and not aware of it. 🫠🫠🫠

3. Didn’t listen to the failures of others

There’s a graveyard of Personal Financial Management apps that died trying to monetize their product. I should have seen this as a signal that building the same thing for a slightly different population was very likely to have the same result, but I foolishly rejected this data.

4. Did the marketing myself

I am terrible at marketing, but I had such a strong opinion about how ads, landing pages, and images were supposed to look and sound that I did it all myself. Ironically, when social ads did work it was almost the opposite of what I was expecting which was a strong signal that I really was clueless. Working outside of my skill set was a great experience as I learned how hard marketing is but that wasn’t the goal; the goal was to get customers and I failed miserably at that.

5. Didn’t manage my time well

I spent so much time coding and working on the product that I didn’t plan the work very well. Had I done better planning up front I probably would have wasted a lot less time building. This was a valuable life lesson that I have since corrected but it was a very painful one.

6. Spent way too much time on small details

You care more than your customers do, probably. I spent so much time combing over every small detail out of fear that things needed to be perfect and it was such a waste of time.

7. Wasted time deliberating stuff that didn’t matter

My cofounder and I were very different. He is a consultant and I am an engineer. In some sense it was the pairing of a builder and a critic, which ultimately led us to disagree and talk about our disagreements too much. We argued about everything and none of it mattered. The disagreements always left me frustrated as it was such an emotional drain and time simply wasted. Make sure you find a cofounder you won’t waste time with.

As a brief aside, time is your scarcest resource and as I get older I find that time wasted on things that don’t matter leaves me with a deep sense of regret more than frustration. Imagine all of the good that could have been accomplished with that wasted time! This regret led me to create the “Execution-Planning Tradeoff” framework that I recommend folks that work at big companies or startups to reflect on deeply—for startups, you need to execute to survive, for big companies you need to execute in order for a startup not to eat your lunch.

In short, you’re probably over-planning. There is a 1-to-1 tradeoff between planning and executing but I’ve seen many companies and people make lots and lots of plans because it feels good…but what feels good and what is actually good is often inversely correlated.

8. Didn’t allocate capital to the right things

Like doing the marketing myself, early on I spent too much time doing all of the wrong things. I should have hired contractors to do most of the things I did myself, especially where the cost was relatively low and the yield was high, like building landing pages, facebook ads, writing articles for content. Eventually I did hire consultants but that brought new problems, e.g., not getting exactly what I wanted, struggling to find quality at a “reasonable price” (though I should emphasize that this is kind of a stupid statement, “reasonable price” is dictated by the market, so the reality is I should have been willing to pay more), and stability in high quality contractors. 

9. Built everything myself

Coming from a data and machine learning engineering background, I already knew a lot about software but very little at all about web or mobile development. I decided to build everything myself (much of which was reinventing stuff that already existed but were going to provide a slightly better UX) and, while it was an incredible learning experience that has made me a much stronger engineer, it was not an effective approach as a founder trying to make a successful business. I should have started with ads, landing pages, interviews with potential customers, and then built something.

10. Misaligned cofounder

As I mentioned, my cofounder and I were very different. At the time I thought he complemented my skill set well but in hindsight I realized we were too different to be effective together. Adding him as a cofounder was entirely my fault, he’s someone I look up to and has been a mentor but that didn’t make him the right cofounder. If I were to do a startup again in the future, I’d look for a cofounder that complements me but also aligns with my focus on execution, which is where my previous cofounder and I really had issues.

How to avoid some massive failures

The only real failure you should avoid is your company dying, everything else is a learning experience. So, if you are a founder, get used to taking a beating (figuratively)—all that matters is that you get back up and survive.

Make sure to track outcomes and progress against them. Some argue process is everything but outcomes dominate process, especially for startups. If your outcomes aren’t happening, change your approach or business because if you don’t you won’t survive. And remember! Survival is all that matters and being attached to an ineffective process is honestly kind of stupid (and that’s not you!).

Where that led me

My experience at Unidos was lonely and miserable. I left Goldman feeling like I was at the height of my career but my startup was such a colossal failure that I felt emotionally broken, but all of that failure was an extraordinary learning experience. I learned more in the 2 years of my startup than I did in the 8 years of work before it. If I could go back in time, I would do it again.

Being a founder was a very humbling and painful experience, which made me a more empathetic and thoughtful person, and also a much better engineer. I now understand how truly hard it is to build something from scratch, which is why I’m much less likely to criticize founders who are trying to build things (though I’d say often the harsh feedback is the most constructive). If I’m to be honest, I probably would have preferred to stay at Goldman a little longer as I think I had more to learn there, but I’ve made my peace with it.

More importantly, after my startup, I joined Fast and then Affirm, which have been the highlights of my career, so I am truly happy and grateful for all of my past failures, as they led me to where I am today.

Happy failing. 🚀📈📉🧨🪦

-Francisco

Some Recommended Readings

I thoroughly enjoyed Alex Johnson’s “How Fintech VCs can Actually Be Helpful” where he highlighted the disparity in venture for founders from underrepresented backgrounds and gave actionable steps for both founders and investors.

I also recommend reading Nicole Casperson’s “Media’s Role in Fintech Diversity”, where she discussed her presentation at Money 20/20 on the very same topic.

I loved listening to the latest Fintech Recap podcast by Jason Mikula and Alex Johnson where some of my favorite Fintech policy folks discussed Money 20/20 and some predictions for 2023!

Postscript

Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!

Feel free to respond to this email or reach out to me on Twitter!

Everything was beautiful and nothing hurt.

—Kurt Vonnegut, Slaughterhouse Five

Let us bow our heads and pray for all of the lost enterprise value in 2021. Amen. 🥲

Fintech was Glorious

Let’s pretend for a moment that we’re back in 2020 and that the world and financial markets were not doused in kerosene and burning in chaos.

Wasn’t it nice?

Public and private markets were drooling over Fintech companies and it seemed like liquidity events were happening left and right

Reflecting on the timeline of events, it all really started when Visa tried to acquire Plaid for $5 billion.

After that things kind of got out of hand. In fact, in 2020 there were 2 Fintech IPOs and 6 acquisitions that amounted to $23 billion dollars and in 2021 there were 12 IPOs and 1 acquisition that amounted to $289 billion. 🤯

The Good Times are Over

But all good things come to an end and that end came roughly in January of 2022, basically the peak of the market.

Since then there’s been nothing but doom and gloom in the media1, whether it be layoffs at public companies or early stage companies shutting down entirely2—the news has almost exclusively been bad. Even this past week Goldman Sachs (my former employer) is allegedly reducing their consumer banking division3.

As the Federal Reserve gives priority to its inflation mandate, the increases in the federal funds rate has played a large role in the decimation of public markets but uncertainty across the global economy, lagging supply chain issues, expectations about tightened consumer spending, and the war in Ukraine has fueled that fear further.

The increases in interest rates have had a particularly significant impact on Fintech, largely because it impacts the cost of funds for lenders and puts additional pressure on margins, but the market has somehow forgotten that great lending and financial companies have been built during downturns before with much higher interest rates (e.g., Capital One started in 1994 and Discover in 1985).

Beyond the impact of rising rates, the CFPB has also turned up the heat on many Fintech companies. This certainly hasn’t helped the situation but overall I think it’s absolutely great! The CFPB was established to protect consumers and so long as they continue to honor that mandate the industry will be better off in the long term…that is, so long as they make sure to call out the real predatory actors.

Additionally, the OCC’s recent “Bank Supervision Operating Plan for Fiscal Year 2023” release highlights the importance that legal and compliance plays in Fintech, which along with the CFPB probably makes investors hesitant to make bets in such a highly regulated industry. I disagree with this personally but I think it may contribute to investor angst.

When will the pain end?

Who knows…probably the Federal Reserve.

I’m not a macroeconomist4 but I do think that the Federal Reserve is well equipped to get inflation under control5 and still prevent an entire economic collapse, even if most of the media would tell you otherwise. The good news is that capitalism moves quickly and the Fed can cut rates (i.e., incentivize growth) once inflation has tampered down to more regular levels (though I would expect more modest adjustments if that were the case).

As you may recall, the Federal Reserve has a dual mandate of maximizing employment and stabilizing prices. We’ve been at full employment roughly since August of 2021 and even as rates have increased we've continued to see a strong labor market, so I don't think the outlook really is as bad as many say—public markets are a different story and it's worth emphasizing that stocks aren't the economy (though an important piece of it).

Even so the IMF has forecasted significant slow downs in the global economy and the FOMC projects Real US GDP to be roughly 1% in 2023 and 1.7% in 2024, which is lower than their target of 2% - 3% in stable times. So the outlook isn’t exciting but it certainly is not a global meltdown.

Surviving

As the economy goes into a cooling state and markets continue to overreact, the most important thing for startups and Fintech companies to do is survive, obviously. While 2020 was a hot year for acquisitions, I suspect that we’ll see banks aggressively using their balance sheet to go shopping, so acquisitions will likely continue but expect much lower valuations.

Given the current market preferences, survival for public companies will only happen through profitability and actually adding value to your customers. So the question is who will weather the economic storm?

I decided to take a look at the data around public Fintech companies before the massive hype and bust cycle and it’s quite interesting to see current market preferences.

In short, it likes payments and hates everything else (except for Intuit???). Obviously the original market cap and valuation multiples in the early 2020s were unsustainable but to think that these businesses are worth less than they were two years ago seems kind of dumb6.

But, as you know:

The market can price irrationally longer than you can stay solvent. 😉

-A. Gary Shilling (mostly)

Beyond surviving, it’s actually a great time to launch a Fintech, great lending businesses are built during downturns. It’s easy to underwrite when delinquencies are low across the industry, but now is where differentiation happens. And it’s important to note that while increasing rates provides margin pressure, demand for lending increases so I believe many fintech companies will flourish during this time—so long as they add real value to their customers and manage risk vigilantly.

But where I am less optimistic is in international markets. As the dollar grows stronger international investments become harder to rationalize (from a US perspective). That and the fact that the Federal Reserve will probably not give much attention to countries outside of the US makes me rather bearish, so I expect things will be challenging for the coming years.

The Future of Fintech

Things will get worse before they get better in public and private markets.

Private markets began cooling valuations for fintech companies but early stage investing in Fintech gained a lot of momentum over the last couple of years, which means that for those that adapt and survive, we will begin seeing the fruits of their labor in the coming years.

Somewhat ironically, an unprecedented set of dry powder means that there’s capital to be deployed and eventually venture capitalist will place their bets on their favorite Fintech companies, which will further drive innovation in the coming years.

The Fintech companies that stunned the world in the early 2020s were built quietly with extraordinary effort during the previous decade when Fintech was boring. If investors and public markets find Fintech boring again so be it…the world still runs on money so innovation will happen regardless, it’s all a question of which investors will have the conviction to wait and which companies will continue to build meaningful products.

Lastly, as an engineer, my strong opinion is that the underlying technology on which Financial institutions and companies were built has changed. In the last ten years open source software, distributed computing, cryptography, analytics, big data, machine learning, and even programming languages themselves have dramatically improved how these institutions provide goods and services (i.e., value) to their customers and we are just getting started.

So my advice is simple: don’t look at your portfolio and just keep building. The future of finance depends on it.

Happy building. 👷‍♂️

-Francisco