Barnes Analytics

Contact

Blog

  • US Economic Outlook 2025

    With the 2024 election in the rear view mirror, we have a better picture of what policy for the next few years is going to look like. As such, we have a better idea on what to expect from the US economy over the next few years. That has largely colored my analysis this year.…

  • Validating Machine Learning Models in Fintech

    Machine learning (ML) is rapidly moving from R&D notebooks into production decision systems across fintech — underwriting, fraud detection, credit scoring, pricing, customer onboarding, and more. That’s great: ML can unlock better risk segmentation, speed, and personalization. But with that upside comes concentrated operational, legal, and reputational risk: a model that breaks, drifts, or discriminates…

  • How We Stress Test Credit Models

    If you build credit decisioning models for a fintech, you know the scorecard moment: a bank partner, compliance officer, or auditor asks for evidence, and suddenly your model has to survive not only production traffic but regulatory scrutiny. Independent model validation is not just a compliance checkbox. Done well, it turns risk controls into a…

  • The Role of Independent Model Validation for Fintechs Dealing with TPRM Teams

    Introduction In the dynamic world of fintech, partnerships with banks and financial institutions are essential for scaling innovative solutions. However, these collaborations introduce complexities, especially concerning third-party risk management (TPRM). Banks are increasingly scrutinizing the models provided by fintech vendors, necessitating robust independent model validation to ensure compliance, mitigate risks, and build trust. Understanding Third-Party…

  • How to Survive a Bank Partner’s Model Review

    If your fintech depends on bank partners to offer deposit rails, underwriting capital, or other regulated plumbing, a model review from that partner is not a performance review — it is a credibility and business-continuity test. Banks are accountable to regulators for models that materially affect safety and soundness, so when they review a third-party…

  • Independent Validation vs. Internal QA: What’s the Difference?

    Independent Model Validation (IMV) is an objective, external (or organizationally independent) assessment focused on whether a model is conceptually sound, implemented correctly, and fit for its intended use. Internal QA is an essential, day-to-day quality and engineering practice that keeps models working and reproducible. One ensures correctness and code hygiene; the other is the required…

  • ROC Curves Suck For Fraud Detection

    So I have been thinking of ways to communicate the value of a new fraud model to stakeholders. A data scientist working in fraud detection might produce an ROC curve. Perhaps plot the ROC curve against some benchmark model’s and tell stakeholders, “Hey, look this model dominates the output from the benchmark. We should implement…

    ROC Curves Suck For Fraud Detection

  • SR 11-7 Explained for Fintechs

    In the evolving landscape of financial technology, fintech companies increasingly rely on quantitative models to drive decisions in areas such as credit scoring, fraud detection, and regulatory compliance. The Federal Reserve’s Supervisory Letter SR 11-7, issued in 2011, provides essential guidance on managing model risk—defined as the potential for adverse consequences arising from decisions based…

  • The Cost of Skipping Independent Model Validation

    Introduction In financial services and fintech, models drive critical decisions every day. From credit risk assessments to fraud detection, pricing algorithms, and marketing analytics, organizations rely on these models to make choices that can affect millions of customers and billions of dollars in assets. Despite the centrality of models, some firms treat independent model validation…

  • How Model Validation Builds Trust with Bank Partners

    Trust is the foundation of any successful collaboration between a fintech and a bank. Banks operate under intense regulatory scrutiny and must ensure that every partner upholds rigorous standards of risk management, transparency, and governance. For fintechs aiming to collaborate with banks, independent model validation presents a powerful opportunity to demonstrate credibility, operational maturity, and…

  • LLM Validation Test: Output Similarity

    Generally, speaking for a model to be valid we want to ensure that the LLM outputs the same information for the same prompt every time. A Retrieval Augmented Generation system (RAG) can help with this consistency. However, LLMs are just fancy autocomplete algorithms. They try to guess the next word in a sentence. By appending…