Joe Ferris, CTO of thoughtbot, was a featured speaker on the Boston stop of the PAYMENTSfn meetup series. Recorded live at thoughtbot's Boston office, this talk gives great insight into some things payments teams may want to avoid when planning payments flows. Joe also shares some lessons learned from a previous role.
With great Q&A from the crowd, Joe gives a great talk on the ins and outs of technical debt in payments. Watch below, and share your thoughts with us on Twitter.
Special thanks to thoughtbot for hosting our meetup, and recording this talk for us.
Rough transcript
Joe Ferris: I'm curious, how many of you here today are developers? Okay, a few. The Technical Debt talk is going to be a little bit technical, but I think there'll be a little bit in there for everybody. What I'm going to focus on today is some of the potential pitfalls in applications that need to process payments where you might encounter unexpected technical debt. Technical debt is a problem in all kinds of applications. We're all busy paying it down or ignoring it, but there are some special cases that apply to payments that I have experienced and I would like to share with you.
Joe Ferris: But first I would like to play a little game with the audience. I'm going to show some logos and I would like for somebody to shout out a really quick description of what that company does. Here is one. Not our wonderful logo in the bottom there.
Lindsey C.: What does Thoughtbot do?
Joe Ferris: Yes. Do you want to shout out a quick description of what Thoughtbot does?
Lindsey C.: So Thoughtbot is a product design and development consultancy and if you need any help [inaudible 00:01:12].
Joe Ferris: Right. So now that we have covered the lovely Thoughtbot logo, let's talk about this other logo. Anybody have a quick description?
Speaker 3: Ride hailing company.
Joe Ferris: Yes, Yes. Lyft, a ride hailing company. How about this one?
Speaker 4: Food hailing.
Joe Ferris: Food delivery again. So an interesting thing about these descriptions and there's an audience of people here focused on processing payments, is that nobody mentioned payment at all. So there's a problem in software, which is that maybe not at the companies you work for, but at most companies, developers pretend that payments don't exist. Most of the time when you think about building a product, you think about the end thing you're delivering to the users and not the end thing you're getting from users, which is the payment. So people think about how are we going to get the cars different places. Ooh, what kind of system do I need to scale to keep track of the cars? Nobody ever thinks about, "Oh yeah, and I'm going to have to do a credit card payment at some point."
Joe Ferris: And so there is a set of behaviors that developers tend to engage in when they pretend something doesn't exist. Any of the developers here will be familiar with how people think about logging. You don't really think about refactoring your logging or managing it or testing it. You probably just copy and paste things all over the place. And the truth is that people tend to at least initially approach payments the same way. They don't treat it like a serious problem, because it's like sending email. We're just going to drop it in a thing like Spreedly or Stripe and these are great services and they can handle a lot of the problem for you, but it is actually a major product challenge that needs to be taken seriously. And so the first pitfall to fall into or not to fall into if you aren't interested in pits is payments are a real thing.
Joe Ferris: They're a thing you'll have to deal with if it's in your product roadmap, like actually include it, for example, on your project estimates when you're building an MVP and things will be better for you later. It is a harder problem than you think, even if you're offloading most of that problem. Another common problem that people run into when writing software that deals with payments is that they tightly couple their payments to their permissions. So permissions are what decide what you're allowed to do in a given system. If you're selling a subscription service like Netflix, you need to have a subscription to see it. And so intuitively you might think that those two concepts are completely intertwined. Having a Netflix subscription, paying for it is the same as being able to access Netflix content. But that's actually not really the case because there are a lot of subtleties in those two cases that are different. Like there are a lot of subtleties to how you collect a payment and a lot of subtleties to how you're allowed to use things.
Joe Ferris: And so I'm going to go to an example really quickly. This has some code. If you're not a coder, then at least it's colorful. But this is an example of a controller that decides whether or not you're allowed to view a certain page. And there's some code here that checks the current user, the person signed in. If you're using Netflix, that's you. And it looks to see if you have a subscription and if you don't, it redirects you to the path where you can get a new subscription. Then if we look at how that user works, this is a really common thing for checking to see if you have a subscription. You'll look at the user and say, "Oh do you have a token for this remote payment service I'm using?" Because you're probably using something like Spreedly or Stripe and not accepting your own credit cards, I hope.
Joe Ferris: And this is a terrible idea because it completely couples that idea of being able to access the site with the fact that you have this one subscription set up, this one way through Spreedly or Stripe or whatever. To give an example of where this bit me personally, I worked on a project where we were offering a subscription service. There was another company we were friendly with that had a similar service, and we decided to have some sweet corporate synergy. And anybody who was using our service subscribing would be able to access some of their content and anybody who subscribed their service would be able to access some of our content.
Joe Ferris: It was sweet business, but of course because we had tightly coupled our permissions to our payments, it was really difficult to give access to all those people from the other system because they had never paid. So we didn't have a subscription for any of those people and we didn't want one. We weren't going to charge them and we needed to completely refactor all of our code to handle that case. And so I would get ahead of that. It's really common for the way you accept payment, either through trials or coupons or bulk deals, whatever, to be different from the way you actually get access to something.
Speaker 5: It's like in logging processes. [inaudible 00:05:49]
Joe Ferris: When I said log, it was logging, like L-O-G-G-I-N-G, which means something really different to a developer then everybody else. Has nothing to do with trees.
Speaker 5: I was asking if permissions are... I was just asking about, and I'm sorry to belabor this. When are permissions [inaudible 00:06:12] from users [crosstalk 00:06:14] how is that separate from [inaudible 00:06:19]
Joe Ferris: So it could be depending on who you're signed in as. But the point is that for any given user or a request coming in from an app or a browser, when you check to see whether or not they're allowed to access a certain piece of content, that's called authorization, it should be different than just checking to see if the user has paid for the content. There should be some intermediate decision-making system there that allows for other possibilities. So another way that people can get themselves in trouble is to have payments coupled to persistence. Coupled is kind of a programmer term. When programmers say two things are coupled, they mean that you can't change one without changing the other. I know that also means something different to most people, but that's what I'm talking about is having these two concepts tightly intertwined when they shouldn't be.
Joe Ferris: Persistence is also a programming thing, but it's basically if you are going to know something about somebody, you need to keep it somewhere, like on a hard drive or on a database somewhere. And it's frequently easy to tightly couple those two processes is... When you're taking in a new user who signed up and storing them, you also do things for their subscription. There are a variety of reasons this happened and it does make it much easier. Your initial code will be small. It's a way to develop quickly, which everybody loves. Developers love product managers, love users, love when you develop things quickly. But doing this will result in your quick development quickly coming to a halt, because you will be unable to do anything without also interacting with your payment system. One quick example would be if you're writing unit tests, which I hope you are. It will be impossible to write unit tests that test either your payments or your persistence without testing both, which is actually a pretty hard problem.
Joe Ferris: And that goes for if you ever need to go into a production console and very quickly fix something in your database. You have to worry about the fact that changing that might also somehow trigger a payment somewhere, which is not a scenario anybody wants to be in. So I really recommend as much as possible keeping these things separate. Another common issue that developers have is that the tests they write around payments are very rigid. Unit testing is a hard problem. It's almost like a development field onto itself. But anything that interacts with the outside world is particularly tricky. And anything that is inherently scary, like taking money, or complex like taking money is even harder. And so people tend to create these really difficult domain tests. And I've seen two main ways that people do it.
Joe Ferris: One is by having their automated test suite that's running occasionally to make sure the software is working actually talk to the payment service. So that appeals to some people because a developer's nightmare is that they deploy a new change and they somehow stop taking money, which they will almost certainly get in trouble for. On some level developers feel comforted by the idea that before they deploy a change, they run some automated software that proves to them that the software still actually takes the money. That has a lot of problems with it, like you probably can't talk to the real production service because you don't actually want to pay for it when you run your tests. So you probably talked to a sandbox, so it's not even a real production test, but also it means you need to be online. Back in the day before all airplanes had Wi-Fi, that was a major pain.
Joe Ferris: It means that if the service ever changes anything, like if anybody ever pokes the piece of data in the sandbox you happen to be hitting in the test, your test will no longer pass and there'll be a great mystery. And then you'll be in this panic spot where you're about to deploy a change and it looks like you're not taking money anymore. And then you find out, "Actually it's just my test that's broken." And there is a psychological phenomenon that happens with these intermittently failing tests that you learn not to trust them. And so it's actually better not to have a test at all than to have a test that talks to your payment sandbox. And the other thing people do is once they realize that's not going to work, they save the exact response that came back from the payment system somewhere and then hard wire it in. So it doesn't actually talk to the sandbox, but they prove that last time they got this response and it worked, my software behaved this way.
Joe Ferris: So if I show that it keeps behaving that way, I know that it keeps working. But that's actually really rigid. There are a lot of details that come back in processing a payment. You'll have these unique IDs and very specific amounts in the particular transaction you are working with. And so hard coding those things rapidly results in either a very controlled un-useful test suite or a rapidly expanding suite of fixtures for all the various cases you can possibly think of for testing payments, which will be too much to maintain. And then inevitably somewhere you end up doing something slightly differently, which means you need a whole new class of responses. Or maybe the payment process you're at working with adds something new to their response and now you need to regenerate all the responses you saved. But maybe they can't be regenerated exactly because the data they were generated from doesn't exist anymore.
Joe Ferris: And so now you're in a position where you have these antique test responses that don't actually exist in the real world, which is worse than having no test at all. So testing those systems separately as much as possible is good. And if you do need to test them together, writing an integration test with some kind of a fake is, in my opinion, the way to go. I think one of the classic pitfalls that I almost didn't mention because it sounds silly, but people repeatedly implement their own payment system when they're not a payment company.
Joe Ferris: Sometimes they do this in little subtle ways, like they'll implement a subset of accepting payments like subscriptions when they don't need to. Even though they could use a payment processor that will handle the billing for them, they might be like, "It doesn't do it exactly the same way that I was planning to do it, and I want it on this monthly cycle and it's just going to be hard to customize, so I'll just do it myself." Because developers always, like everybody, underestimate how much work it's going to take to actually build and maintain something.
Joe Ferris: And then once you get it going, it's like, "That was so easy. Once a month the thing kicks off and it bills everybody." But then you realize that most of the time credit card payments have a good chance of failing. And so you have to think about, "Do I retry this? What happens if it stops working? Who's going to follow up with them? How do I expire these old requests?" You end up in this hell of probably using a payment system but then very slowly implementing it yourself as a bad copy. And then accidentally becoming liable for some kind of a compliance issue. I highly recommend offloading as much payment related operations as possible onto a good payment processor. And if it doesn't work exactly the way you pictured, just use it their way, it will be better. It will almost certainly be better not to support a very specific coupon format than it would be to implement that yourself, because it's much harder than you're thinking.
Joe Ferris: I should have advanced this slide a little sooner. It's harder than you think the problem is. It looks like that, and you will look like that. The programming concept here that is common to a lot of these problems is called orthogonality. Trying to keep it so that when you change two unrelated systems independently, it does not require a change to the other system. Like changing how you charge users for something should not change how they consume the content and vice versa and so on. And as much as you can maintain orthogonal systems, you will be able to deploy features and rapidly take payments quickly without getting into these kind of hells that I described.
Joe Ferris: And I like to have a chart in my presentations. I couldn't think of one to put in this one. So I tried to make a visualization of orthogonality. There's no labels on the axis because they don't mean anything. But you can see that as you'd expect in an orthogonal system, the payments, which are blue, don't ever overlap with the permissions, which are red. As they change, they maintain their own separate paths. That's my chart.
Joe Ferris: If you want to see a good example of technical debt in action, we actually have a service that we run called Upcase, which is a developer educational service. We used to charge for it as a product, but we have since released it for free for everybody. And as part of doing that, we actually also open sourced the code that runs the service itself. So if you are a developer and you're interested in seeing the history of all the lessons I just described, if you want to watch us learn those lessons in a git log, you can find them here. You have to go back a little bit of a ways now, because like I said, it's a free service and you won't find reference to payment anymore. But if you go back a little bit, you'll find us flailing and repeatedly learning not to implement our own payment system. Was there a question?
Speaker 6: Yeah. Do you have good [inaudible 00:15:16] messages on that?
Joe Ferris: I think we do have pretty good [inaudible 00:15:18] messages, if I do say so myself. And that's all I have. So do I have time for a few questions?
Speaker 7: I just have a real world example of something you said. So who ever uses Patreon? Anybody? I got [inaudible 00:15:39]. Who knows Patreon [inaudible 00:15:41]? There's some hands. Okay. So she wouldn't mind telling us, I've heard her tell this story in real life. Michelle Vautier is the head of payments at Patreon. When Patreon was built, they did exactly what Joe said. They run basically all their payments on the first of the month. What no one really anticipated was the fact that they would scale to doing $40 million a month on that platform. So now they have a huge problem, a throughput problem, and everything runs over just a very short 24 hour period. And in their case, it's not just that they're processing those payments, but that's somebody's paycheck. That's a creator's paycheck.
Speaker 7: So if they have a payments problem because of their debt and how they implemented three or four years ago, five years ago, six years ago, now creators are not actually getting a paycheck. So that's a real world technical debt problem that they still haven't solved yet. But it's something they're working on.
Joe Ferris: Yeah, you live with those decisions for a long time.
Speaker 7: Yeah. And they impact on the people, too.
Joe Ferris: We just gave up and made it free.
Speaker 7: Patreon is [inaudible 00:17:10].
Joe Ferris: We thought about that strategy. Any other questions? All right, thank you very much.
Download the Payments Orchestration eBook Below
.png)
.svg){kind=icon}
.svg)
