Barnes Analytics

Contact

Blog

  • US Economic Outlook 2025

    With the 2024 election in the rear view mirror, we have a better picture of what policy for the next few years is going to look like. As such, we have a better idea on what to expect from the US economy over the next few years. That has largely colored my analysis this year.…

  • Independent Validation vs. Internal QA: What’s the Difference?

    Independent Model Validation (IMV) is an objective, external (or organizationally independent) assessment focused on whether a model is conceptually sound, implemented correctly, and fit for its intended use. Internal QA is an essential, day-to-day quality and engineering practice that keeps models working and reproducible. One ensures correctness and code hygiene; the other is the required…

  • ROC Curves Suck For Fraud Detection

    So I have been thinking of ways to communicate the value of a new fraud model to stakeholders. A data scientist working in fraud detection might produce an ROC curve. Perhaps plot the ROC curve against some benchmark model’s and tell stakeholders, “Hey, look this model dominates the output from the benchmark. We should implement…

    ROC Curves Suck For Fraud Detection

  • SR 11-7 Explained for Fintechs

    In the evolving landscape of financial technology, fintech companies increasingly rely on quantitative models to drive decisions in areas such as credit scoring, fraud detection, and regulatory compliance. The Federal Reserve’s Supervisory Letter SR 11-7, issued in 2011, provides essential guidance on managing model risk—defined as the potential for adverse consequences arising from decisions based…

  • The Cost of Skipping Independent Model Validation

    Introduction In financial services and fintech, models drive critical decisions every day. From credit risk assessments to fraud detection, pricing algorithms, and marketing analytics, organizations rely on these models to make choices that can affect millions of customers and billions of dollars in assets. Despite the centrality of models, some firms treat independent model validation…

  • How Model Validation Builds Trust with Bank Partners

    Trust is the foundation of any successful collaboration between a fintech and a bank. Banks operate under intense regulatory scrutiny and must ensure that every partner upholds rigorous standards of risk management, transparency, and governance. For fintechs aiming to collaborate with banks, independent model validation presents a powerful opportunity to demonstrate credibility, operational maturity, and…

  • LLM Validation Test: Output Similarity

    Generally, speaking for a model to be valid we want to ensure that the LLM outputs the same information for the same prompt every time. A Retrieval Augmented Generation system (RAG) can help with this consistency. However, LLMs are just fancy autocomplete algorithms. They try to guess the next word in a sentence. By appending…

  • What Regulators Expect From Fintech Model Risk Management

    The Regulatory Landscape Model risk management is not new. In 2011 the Federal Reserve and the Office of the Comptroller of the Currency issued guidance known as SR 11-7. This guidance set out a clear framework for how banks should manage model risk. It called for governance structures, validation processes, and strong documentation. Though aimed…

  • 3 Costly Mistakes Fintechs Make in Model Validation (and How to Avoid Them)

    For fintechs and banks alike, models are at the heart of decision-making — from credit scoring and fraud detection to pricing, capital allocation, and even AI-powered chatbots. Regulators know this, which is why model risk management has become a major area of scrutiny under supervisory guidance like SR 11-7. Yet despite the stakes, many firms…

  • Why Every Fintech Needs Independent Model Validation

    In an age of spectacular power of AI and machine learning models that seem to grow in capability year after year, fintechs are seeking ways to increase efficiency with these new tools. However, these tools are designed to literally have a mind of their own. So they can easily misbehave. Take the example of Hello…

  • How Well Does Real Estate Hedge Inflation?

    Last week, I built a multifactor model and ran some optimization to find good candidate zip codes to invest in. This week, I’m looking at the same data again, but this time, we are going to figure out, on a national-level, whether or not our real estate investments are a good hedge against inflation. You…

    How Well Does Real Estate Hedge Inflation?