# Duckbill ### Posts #### 4 Reasons Lyft is Smart to Pay AWS $300M The internet virtually exploded when Lyft released their S-1 in preparation to go public. It contains a lot of information, but the thing most directly relevant to my area of expertise is the gem that they've committed to spend $300 million on AWS between 2019 and 2021. Some corners of the internet have lost their collective minds over this with some fairly horrendous takes. Allow me to address several of them. #1. They could save a lot of money by building out data centers! Running data centers is hard. You will take downtime, and you'll do a worse job of it than any major public cloud provider who isn't Orac—wow, a process server at my door already?! That was fast. My point is that there are more people working for AWS's data center teams than likely work at most people's entire companies (yes, there are notable exceptions). They're frankly better at handling failures, they're more practiced and polished at carrying out their procedures, and they're astonishingly capable to the point where we forget that what they do isn't easy. "Disk failures" manifest as latency spikes; networking failures are usually invisible. On the rare occasions where there's a broad systemic outage, that's the day that nothing on the internet seems to be working quite right, anyway. You own your own availability (and there’s no getting around that), but for many use cases taking an outage when half the internet is broken is understandable--and minimizes your “headline risk.” Counterintuitively, if you build an application and let it sit and rot, it'll slowly get better and cheaper over time in AWS. "Price cuts, improved disk and network performance, and upgraded equipment" tend to beat "the last spare drive just failed in your RAID array and now your site is down." Try that in a datacenter, and the raccoons will carry off your servers within five years. It's not just that data centers are cheaper (they're very much not), it's that they're not as good as the public cloud.   #2. We never used to spend that much money on data centers. Cloud is a rip off! Take AWS's annual revenue from 2018. Good. Now double it. Keep going. Okay, now we're at Dell's annual revenue in 2012. IT costs have been skyrocketing for decades.  There are just a lot more tricks you can use to hide it from view when you've got 40 different divisions, you can treat the spend as CapEx (well, easily anyway; you can still do some of this with cloud spend but that's a longer conversation), and you can spread it across multiple years rather than one gargantuan payment to one vendor. That stands out on the forms, and it empowers lazy journalism. Folks who are freaking out about a $100 million annual AWS bill apparently don't realize that a single fabrication plant for a semiconductor manufacturer costs upwards of 30 times as much. Lyft isn’t just using a bunch of EC2 instances and S3 storage; they’ve told us as much. #3. But they could save so much money if they just... It doesn't matter how that sentence ends. Look at Lyft's financial position. They're not profitable. They're focused on growth and have some ambitious milestones that they need to hit in order to get there. "Saving money" is absolutely not aligned with those goals from a strategic perspective. Spending too much money is not Lyft's primary problem. If and when it ever becomes their primary problem, they're likely in decline. Regardless, there will be opportunities to address that then. #4. Nobody ran the numbers on this! If they went with (insert company I work for here), they'd save a boatload! I find it a bridge too far to presume that people savvy enough to grow Lyft from "an idea" to "a reportedly $26 billion company on the verge of going public" somehow never bothered to run a cost/benefit analysis on a nine-figure annual budget line item. Data centers cost way more than hardware, power, bandwidth, cooling, etc. There's the human time; engineers aren't cheap. More insidiously, the cost inherent to the loss-of-focus is incalculable. "We're taking downtime in data center 4 on Thursday night to replace a generator so now we have to tell 200 service teams about it and make sure they're all okay with it" is a logistical nightmare you simply don't have with a public cloud provider.I'm not a cloud apologist; there are budget issues with cloud computing. Some workloads flat out aren't appropriate at scale; Dropbox's multi-exabyte scale storage issues realized a reported $74 million annual savings when they moved off of AWS and into data centers. So, there are definitely times when it makes sense. I've just seen no evidence whatsoever that Lyft's environment is one of them. There are certainly things potential Lyft investors should worry about, like pink mustaches. As for its AWS spend? Nothing to see here. Move along. #### A Duck Tale Life is like a hurricane here at Duckbill. In February 2020, we began building what would become DuckTools, a new AWS cost management SaaS tool, modeled off of our extensive internal tools we built for our consulting projects. We launched to the world in private beta in December 2020. This month, we shuttered it. Let’s talk about why. Race cars, lasers, aeroplanes Like many great products, DuckTools began as a set of internal tools. The Duckbill Group, being a highly technical consultancy, had built and contracted a number of clever scripts and apps to help us deliver on our mission: lowering the horrifying AWS bill. While our Cloud Economists are able to help almost any company reduce their bill, our consulting services are often not cost-effective for a company with, say, $50k/month in AWS spend. That got us thinking: What if we turned our internal tools into a public product so we could help people our consulting services weren’t a fit for? We could expand our market, help more people, build a passive revenue stream, and perhaps even create an upsell path to our consulting services. We knew we couldn’t compete directly with existing cloud tools built by large teams with their mountains of venture funding—nor did we want to. Our vision was a set of small, focused tools to help people answer very specific questions—like “what Savings Plan should I buy,” “which DynamoDB capacity model is best for this workload,” or “where’s all this Lambda spend coming from and what can I do about it?” DuckTools! I joined Duckbill as the Lead Product Engineer to make this happen and bring a new product to market. It was always a risky proposition; this was effectively a brand-new startup (albeit, within a stable company), and common wisdom says most startups fail. We believed we had some unique advantages, though: a deep pool of technical knowledge in our Cloud Economists, a pile of consulting customers eager to tell us about their problems and test our early ideas, and powerful existing internal tools to build off of. I spent my first few months developing and executing on a pretty simple plan: Let’s take a few of these existing tools and wrap them in a nice-looking app with all the standard trappings of SaaS: auth, a dashboard, team management, that sort of thing. We’d get some beta users from our existing clients, get some feedback, and start adding more tools based on what people ask for. It’s a Duck-blur The original go-to-market plan involved leveraging our existing consulting sales conversations to also pitch DuckTools. We figured that anyone with the kinds of AWS cost challenges The Duckbill Group solves would also be interested in cost tooling. Given we already had existing lead flow, sales processes, and sales staff for our consulting, we thought this would make for a strong advantage. As it turned out, this plan fell flat on its face for a few reasons: The cadence of these sales conversations was much slower than what we’d need for DuckTools. You need to talk to a lot more potential customers when you’re selling three-figure SaaS subscriptions than when you’re selling five- and six-figure consulting engagements. We hadn’t considered the incentives for our sales staff. Why spend any time talking about a sub-$5,000 TCV deal when you can talk about a $100,000 TCV deal instead? Our sales staff rightly focused on the higher-leverage deals. Spending any time selling DuckTools was, at best, a distraction. What we really needed was product feedback—not a whole bunch of paying customers. Of course, we wanted both. Given the choice, the former was much more important. But our sales staff was more concerned with closing deals than getting beta users—again, rightly so. Our existing customers just weren’t that interested in DuckTools. “Why would we want a tool when we could just have The Duckbill Group handle it for us?” we’d hear. Ruh-roh. Despite those problems, we thought that we’d at least get one or two beta customers if we gave it enough time. So we kept at it for a few months longer. Foreshadowing intensifies. D-d-d-danger By fall of 2020, we still hadn’t seen much interest in buying DuckTools from the people we were talking to. This was becoming a serious problem. As any product manager will tell you, building software without talking to customers is a recipe for disaster. As product owner, I needed to be talking directly to the people we were trying to sell to a lot more than I was Since we can’t (or, at least, really shouldn’t) build a product in a vacuum, we came up with a new plan to rapidly get user feedback: We’d launch! In December 2020, we put out a barebones marketing site with some demo videos of what we had built so far and a request for customer research interviews. We kept the launch pretty low-key, afraid of being inundated with more interviews than we could handle. This turned out to be a good idea. Ducktools.com was born and it worked fabulously; dozens of helpful, interesting, and good-looking people volunteered to tell me about their cloud cost problems. I poured over hours of notes and call recordings, condensed my findings, and came to a few sobering conclusions: What we’d built, people don’t need. Or rather, they don’t need it enough. We’d built a really nifty Savings Plan calculator that solved a problem for a number of customers. Unfortunately, that number multiplied by what people were willing to pay for it didn’t equal a sustainable business any time soon. What people want, we can’t build. When we asked what people struggled with, they described a variety of AWS cost pain points we were all-too-familiar with: forecasting spend, tagging, periodic reporting, unit economic analysis, and many more higher-order problems. Each of these could be their own full-fledged product (and some are!), but none of them we felt we could do well with our resources. We joked internally that many people really just wanted Cloudability or CloudHealth but at a fraction of the price. What customers want diverges significantly from the specialized tools our Cloud Economists need. We’d hoped to offset the engineering cost of building DuckTools by making it do double-duty as internal tools as well. That clearly wasn’t going to work now. What to do, just grab on to some… While the result of our research was concerning, we could still perhaps resolve it. Could we try to find more of the few customers that wanted what we had already built? It’s possible, but the math just didn’t add up. Plus, we expect AWS to improve their own tools such that ours become obsolete within a couple years—faster than we could recoup the investment. Could we hunker down and build the heavyweight solutions people were asking for? It’d be expensive and tricky. We could hire a team and make it happen. But where would we get the money for all that hiring? And so, we ran face-first into the wall of Company Strategy. We’re a bootstrapped company and proud of it—it lets us build the kind of place we want to work. Being profitable and having positive cash flow is of the utmost importance to us. Spending “a team of engineers” money on a product that won’t see ramen-profitability for two-to-five years doesn’t fit well with that model. And further, the more investment we make into that path, the more of a distraction it creates from our two other lines of business—our consulting and our media publications—which are both profitable and growing steadily. Bad and good luck tales It’s always worth considering what we could have done better. These are the lessons I learned; maybe they’ll help some other product builder out there: Talk to more customers and do it much earlier. This is the advice in big bold letters on every product book I’ve read, but I still didn’t do it enough. If I’d had the conversations I had in December in July, we would have realized the market fit we were hoping for didn’t actually exist. When starting a new business, you need to do sales yourself. “I’ll build it and someone else will make sure it gets sold” is a recipe for a big disconnect between Sales and Product. I needed to be pitching to customers and hearing their responses directly. But I thought we could achieve better results by separating them. Figure out how much of a problem you’re solving earlier on. Plenty of people told me that our Savings Plan Calculator solved a pain point they had. But it wasn’t until we started digging into how much they’d pay for it that we realized it wasn’t viable as a standalone product. The products you build to solve a $100/month problem and a $1,000/month problem are not the same. These aren’t particularly new insights. I could probably find a chapter matching each somewhere on my “how to build product” bookshelf. As usual, though, experience is the best teacher. Tales of Derring-do While it wasn’t an easy decision, we realized we didn’t really have a choice. We had to close down DuckTools as a public-facing SaaS. It’s a good product and we’re proud of it. But the market for it isn’t what we hoped and the path to making it viable doesn’t fit our business goals or the resources we have. While I’m definitely sad to see DuckTools go, parts of it will live on as internal tools. If you took some time out of your day to take a customer research call with me, thank you! Your input was invaluable. And if your AWS bill is still horrifying, The Duckbill Group will always be here to help. — Kevin Highwater, Lead Product Engineer, DuckTools #### A Simple, Yet Effective Cost Optimization Framework When Corey and I started The Duckbill Group in 2019, I sat down with Corey and asked him to braindump everything he looks at when assessing a client's AWS spend in the first pass. I then came up with a simple framework that's easy to memorize and still serves us today. Step 1: Turn that shit off. Whatever you're not using, get rid of it. Really not much more to say there. Step 2: Store less data. Keeping data around is expensive as hell. Maybe store less of it. If you must store it, store it in cheaper locations. Today, we think about this as a holistic data lifecycle and it's one of the first things we seek to understand with a customer. Step 3: Move less data. Bandwidth is expensive in the cloud. The more you move data around, the more you're paying. But this also comes down to things like compression too: move the same data, but compress it before transit so the size isn't as large. Step 4: Cloud-ify your workloads. Corey coined this the "cloudiness continuum." Be less like a datacenter and as serverless as you can. In generic terms, this is about lowering execution time of compute. Step 5: Pre-pay for resources. Reserved Instances, Savings Plans, private pricing contracts. These are the last things you do, since these lock in architectural decisions. Step 6: Repeat And lastly, repeat the process often. Things change regularly in customer environments, so it's always worth revisiting past discussions. You'll note that this framework isn't an exhaustive list of things to check. That's because cloud costs--and cloud architecture--can't be reduced to simple lists. These are complex problems and not every solution applies to every situation. Human expertise is necessary. #### And the CFO Wept https://vimeo.com/album/5825745/video/322944647 #### Aurora DSQL: A Technical Marvel with a Pricing Randomizer I mentioned a few weeks back in my obnoxious AWS newsletter that the Amazon Aurora team looked at how to price their new DSQL offering, and promptly gave up entirely. In a remarkably Amazonian display of Customer Obsession, the team reached out to have a conversation with me about it that somehow didn't open with "now listen here you little shit," and I learned a lot. In short: Amazon's Aurora DSQL is a technical marvel, but its pricing is absolutely baffling. And I mean just that. They're not gouging customers. It's not unfair. How they arrived at their pricing makes sense given the product's development constraints (presumably including things such as "thou shalt not lose us our corporate ass on this service, as we cannot make it up in volume"). It's just monumentally confusing. I will explain. Wherein the unofficial take is better than the official one Channeling the spirit of AWS blogs from a bygone era, Marc Bowes (Senior Principal Engineer on the Aurora team) did a small scale experiment and blogged about it on his personal site. Despite the keynotes, the heavily corporate blog posts, the copious documentation, and the "we are DSQL" rendition performed by the re:Invent House Band, it was this post that made the service "click" for me: it's effectively Amazon's second serverless database offering. Y'know, after DynamoDB. Ignore Aurora Serverless, Aurora Serverless 2: Electric Boogaloo, Keyspaces (yes, the DynamoDB shim), and the rest; this is the real deal. I followed along with exactly what he did, and sure enough! The DPUs I consumed were correct to five decimal places for the read DPUs, the write DPUs, and--wait, the compute DPU I consumed was roughly 3% less than his? It was around this point that my eye began twitching uncontrollably. What will it cost me? Who knows! The problem with the pricing is not that it's too high. The problem is instead that it’s hard, bordering on impossible, to model. I honestly don't have enough data to say for certain whether Aurora DSQL is a cost efficient way to run a given workload–and neither does anyone else. The single way to find out what a workload costs is to benchmark it on DSQL, and given that it doesn't support the full set of PostgreSQL features, that's going to resemble a bit of a migration uplift for any non-trivial workload. Given the pricing dimensions, I could not begin to guess in advance whether the economics will work out. Things that historically didn't cost anything on RDS (or rather, manifested as rising CPU utilization) now have a direct cost associated with them. Expensive is okay, “I dunno” is not If a service is very expensive, that may be okay for some use cases, for some customers. What I can't countenance, and can't in good faith recommend, is a service where the result of a modeling exercise spits out an end result of "yes, this will cost you some amount of money." When the money rises from “science experiment” to something more substantial, companies need to be able to reasonably forecast at least the broad shape of what the bill is likely to be. DSQL makes that an impossibility. Aurora has seen this before. With a 25¢ I/O charge per million operations, it was very difficult to forecast Aurora costs without running a workload to see what it would look like. This was annoying enough that AWS fixed it by offering a more expensive instance that waived the IO charge–and customers adopted it in droves. The AWS Compute Optimizer now makes recommendations to tell you, cluster by cluster, what you should be running. This is great, but the customer shouldn't even have to make these decisions. If I can’t give up, AWS can’t either AWS pricing has officially reached the point where 'it depends' isn't just the answer – it's the entire pricing documentation. The worst part is that I don't really have a better pricing model here that satisfies the constraints. I just know that this one is likely to scare the crap out of customers, and hinder near-term adoption of what looks to be a fantastic service. #### AWS Begins Sunsetting RIs; Replaces Them With Something Much, Much Better Disclaimer: In case anyone is confused, I have no special inside knowledge of AWS's plans for Reserved Instances. Maybe this marks the start of their demise, maybe it doesn't. I'm inclined to believe the former, though. AWS Reserved Instances have gone through many iterations over the years (e.g., RI Marketplace, Regional RIs, Convertible RIs, and Instance Flexibility), much of them focused on providing more flexibility and simplicity.  While RI purchase planning isn’t too bad when you have 100 instances running Twitter For Pets, it becomes an utter nightmare when the answer to “How many instances do you have?” is “Who the hell knows?” In between all of these improvements to RIs, AWS continued to launch new instance families and sizes to the point where there are over 200 distinct instance SKUs you can purchase today in us-east-1. Thankfully AWS has heard our wailing and gnashing of teeth and realized the horrific microservices-diagram-meets-Boston-streetmap of complexity they created. Today’s announcement marks an incredible improvement in compute purchasing. AWS Savings Plans Today, AWS announced "Savings Plans," which sounds like a bank's Christmas Club account offering but is in fact something far more compelling. It amounts to nothing less than a complete overhaul of the AWS compute pricing model.  Yes, it really is that big of a deal. At a high level, you no longer need to purchase RIs for a given instance type. Instead, you commit to a baseline level of spend per hour on compute that you’ll pay regardless of actual use. Anything at or below that usage level is included; anything above it you’ll pay at the existing on-demand rates.  The new model is available in two variants, though they have much in common. Compute Savings Plan EC2 Instance Savings Plan Term 1 Year or 3 Year Platform/OS Flexibility  Yes Tenancy Flexibility (Shared vs Dedicated) Yes Regional availability All, except mainland China. Yes, that means GovCloud too! Overage charges Spend above your commit is charged at on-demand rates Region-specific No Yes Instance-specific No Yes Purchase Options No upfront, partial upfront, and all upfront Discount Up to 66% Up to 72% Supports Fargate Yes No Compute Savings Plans Compute Savings Plans come with a high level of flexibility. They aren’t tied to any specific region. You decide your spend commitment per hour across all of your accounts and that’s that. You pay that much regardless of your usage; any usage beyond that baseline commitment gets charged at normal on-demand rates. You get up to a 66% discount for a three-year commitment--the exact discount percentage varies. Interestingly, Compute Savings Plans apply not just to EC2 instances, but also to Fargate. This means you can seamlessly transition workloads between EC2 and Fargate and it applies to your spend commitment. EC2 Instance Savings Plans EC2 Instance Savings Plans have less flexibility but provide you with deeper discounts in exchange. They are tied to a specific region and a specific instance family (e.g., i3 or c5 instances). In return, you get a varying discount percentage that comes in as high as 72% for a three-year commitment.  How do I work with Savings Plans? You interact with Savings Plans within Cost Explorer.  The "Savings Plans" menu option provides an overview of the program, a purchase page, and a very well-tuned recommendations engine. It looks at your historical usage for 7, 30, or 60 days and generates purchase options for you across both one- and three-year terms, as well as the usual "no upfront, partial upfront, or all upfront" payment options you've come to know and tolerate from Reserved Instances.  You’re absolutely going to want to use Cost Explorer to baseline what your per-hour on-demand costs have been; this isn’t a metric that most things track because until now, why would anyone have cared? Remember, if you spend the off-hours at a lower per-hour level of compute, you’re going to want to use that as a baseline for your purchase, as you’ll be committed to spending that much around the clock, every hour, regardless of whether you’re using that much capacity.  Are Reserved Instances actually going away? Sadly, not to our knowledge. You’ll still be able to purchase RIs, because AWS never turns anything off. On the plus side, you don't have to care about them anymore unless you want to. This brings up a wonderful question: When should you use RIs instead of Savings Plans? Here are some times when it may make sense to still use RIs: If you have internal company processes deeply tied to the RI purchasing process,If your company is accounting for RIs as CapEx and your auditor won’t allow you to do that with Savings Plans, orIf you’ve taken a sudden sharp blow to the head and enjoy the stupendous complexity of RI purchasing and management. Other than in those three exception cases, you want to migrate as soon as you can.  There's no reason not to do this. It frees up a tremendous amount of obnoxious fiddly work to calculate exactly what your RI buy should look like (hope you like Excel!) and it grants you the freedom to experiment with Fargate without paying the challenging-by-comparison on-demand pricing for it.  I just bought a boatload of RIs. Am I screwed? I wouldn't think so--remember, Reserved Instances aren't going away and offer the same discount percentages as Savings Plans. That said, if you're unhappy with AWS, I give you the same advice I give to everyone in your position: call your AWS Account Manager. They can't help you if they don't know you're unhappy. There are some losers in this change, though Unfortunately, the SaaS companies that built their businesses around managing RI purchasing on your behalf could not be reached for comment at press time. They were too busy sobbing into their AWS Partner Network Agreements.  To be honest, I kinda feel bad for VMware’s purchase of CloudHealth and Apptio’s purchase of Cloudability, both of which just presumptively lost massive market value with this one change. The new AWS Savings Plans are a long-overdue and welcomed change, to the point where about all I can mock is its uninspiring name.  "Savings Plans" sound like how you're going to cover your vacation to Hawaii. But in this context, that would just be embezzlement.  You can read more about the AWS Savings Plans on the official AWS blog, or you can ping me on Twitter to yell at me about it. #### AWS Cost Allocation Guide: Aggregating and Assigning Cloud Costs Understanding AWS spend is a big pain point for companies, but cloud cost allocation can make the process easier.  Cloud cost allocation helps you identify, aggregate, and assign cloud costs along team, product, and business lines.  Once you’ve built (and socialized!) your process for identifying cloud costs, AWS will start churning out spend data along the parameters you chose — but that’s not immediately obvious. It won’t be reflected in your total monthly spend, and it won’t show up in your PDF bill. To leverage that data, we’ll look at the last two steps of cloud cost allocation: aggregating and assigning your cloud costs. This article is part two of our AWS Cost Allocation Guide series. We’re giving you the tips to create your very own cloud cost allocation process and keep it relevant as your organization changes. Check out the first article in the series for help identifying your cloud costs, a critical first step in the process. Aggregating cloud costs with 1st-, 2nd- and 3rd-party solutions AWS is building these magnificently complex reports for you with tons of data. Now what? It’s time to aggregate that data in a way that makes it insightful and actionable. You’ve got multiple options for aggregating cloud costs depending on where you are in your cloud cost maturity journey. Excel. Let’s not kid ourselves here — your best bet may be a handful of spreadsheets and formulas if you’re just starting out in your journey or if you don’t have AWS console access.Cost Explorer. We can’t rule out this classic option. Cost Explorer has a number of default reports that can be easily tweaked to group spend by linked account or cost allocation tag. However, Finance (and Leadership) might not have access to Cost Explorer, so that data needs to be exported to be shared in internal reports.AWS Cost Categories. These let you group spend based on accounts, tags, resources, or charge types. While it’s an easy way to group these expenses at a very high level, resource tagging still provides a more granular view and metrics from dynamic architectures. The AWS Cost and Usage Report. Because you’ve already enabled your Cost and Usage Report with hourly billing and resource IDs, right? This can provide extremely granular spend and usage insights when combined with a business intelligence tool such as Athena, Quicksight, Tableau, Looker, etc.A third-party tool. Look for one that automatically groups spend for you, like Cloudability’s Business Mappings feature. While super handy, this method also requires the AWS Cost and Usage Report and a hefty price tag of 1% to 3% of your total AWS spend. Your own custom, in-house tool. Sometimes the feature set you need simply doesn’t exist in commercial solutions. We’ve seen a number of organizations with $12 million-plus annual AWS spend unable to find existing products that support their cost management strategy, leading them to build their own custom solutions. This can definitely make sense for some organizations, though I wouldn’t reach for this right off the bat. If you’re just getting started, something simple like Cost Explorer or Excel may be all you need. But consider upgrading to the Cost and Usage Report as your organization grows. The Duckbill Group leverages the Cost and Usage Report for most of our engagements, so we highly recommend this option coupled with your BI/visualization tool of choice to get granular, resource-level breakdowns of cloud costs. AWS even provides a CloudFormation stack to get you started querying the data in Athena. Assigning cloud costs with the showback model Once you’ve accurately identified and aggregated all or your cloud costs, it’s time to act on that data. Assign those cloud costs back to products, teams, and business lines to keep them accountable for their workloads. Even better, empower teams to guide their own cloud cost management journeys by sharing tailored cloud cost reports.  The easiest way to do this is with a showback model, which shows each team how much money its application(s) cost the company. If your developers consider the cloud a black box for deployments, this model provides additional information that allows them to make data-driven decisions about the cost of current features and future projects.  Once you’re allocating cloud costs, start a conversation between the Engineering and Finance teams about per-product budgets and forecasted spend. This helps both teams identify and gather the AWS spend data Finance needs to manage your organization’s money more accurately.  This is also where your holistic strategy for identifying cloud costs comes into play — you’ve already decided how to break up your costs along business lines, and now you get to see the output of that work. All of the cost aggregation solutions we mentioned in the previous section let you group costs to see how much each team, product, or business unit is spending on AWS. Finally! You’ve got some data! This is a great feedback loop moment. Is the spend data broken up the way you need it? Are you able to make data-driven decisions with this information? If not, no sweat! Your cloud cost management strategy can be flexible — and the earlier you make these changes, the sooner you will be able to make data-driven decisions with accurate costs per team, product, or business unit. The showback model can keep teams accountable to their cloud costs with per-product spend and forecasting — but not every team is incentivized to prioritize cost management work. As organizations grow to Enterprise levels, the AWS budget owner may not have direct influence over teams using AWS. And if the budget owner can’t influence teams to focus on spend data and optimization recommendations, those teams are less likely to adopt the organization’s cloud cost management strategy. Aligning cloud costs with business goals In September 2009, author Simon Sinek gave a TED talk on the difference between what you do and why you do it. Your cloud cost allocation work is the “what” in this scenario — the implementation method. But it’s the implementation method for a bigger purpose — the “why.”  Why is cloud cost allocation so important?  Cloud cost allocation isn’t about optimizing or saving money — that’s a side effect. Ultimately, cloud cost allocation lets your organization accurately predict future spend as your organization grows, which leads to better forecasting and the ability to increase profits. Once you’ve built your own cloud cost allocation model to identify, aggregate, and assign your AWS spend along business lines, there’s a critical concept your organization should explore to clearly align cloud costs with business goals: unit economics.  For those who aren’t familiar with the term, unit economics describes a business’ product in terms of revenues and costs related to a key performance indicator (KPI) that tracks closely with customer demand. That KPI can be any basic, quantifiable metric or item that creates value for the product. For example, in the airline industry, the unit economic KPI is usually a seat on an airplane. If you’re running a SaaS product, the unit economic KPI could be a single API request or a customer on the platform, depending on product architecture. Your KPI and how you break out costs to support your customers is unique to your overall business.  Once you’ve identified your product’s unit economic KPI, think about the infrastructure costs associated with it. These infrastructure costs include AWS and any other services you leverage to run the product, like Datadog, PagerDuty, Twilio, and Splunk. One simple approach to calculate your unit economic cost is to take the sum of all the infrastructure costs (including applicable third-party services) and divide that by your product’s specific unit metric.  That process works great for an organization with a single product and a single set of associated cloud costs. But let’s be real: You’re likely running multiple products across dozens or hundreds of AWS accounts. This is where your robust tagging strategy can help you accurately account for each product’s total infrastructure spend when all the cloud usage is shared. Identifying and building your organization’s unit economic model takes work. But the benefits of creating your unit economic model are enormous. Once you know what your unit KPI costs, predicting your AWS spend is only a matter of estimating growth. That in turn allows Finance to more accurately predict Engineering spend further than just a quarter or two out. How cost allocation and unit economics can drive profits There’s another less obvious benefit to setting up your unit economics model: Once you know the cost of your unit KPI and the drivers that lead to it, you can begin tweaking things to improve your margins. Imagine winning deals with competitive but profitable discounting because you know exactly what it will cost to service a new customer. That’s how you can leverage unit economics to grow your business in an increasingly competitive economic environment. Not only will you make your CFO happy with your ability to forecast the complex cloud spend, your Head of Sales will become your new best friend because you’ve given them a powerful tool to price deals more quickly and easily.  Putting your cloud cost process into action As you start identifying, aggregating, and assigning cloud costs to teams, you’ll need to set and clearly communicate your cloud cost management initiative as a priority. A team working on too many high-priority goals ultimately doesn’t accomplish any of those goals well, so your Engineering teams need to know that cloud cost allocation should be its number one focus. Next, focus on changing employee behavior to ensure lasting accountability. The Duckbill Group has seen lots of short-lived cost optimization initiatives because organizations do not invest in guided behavior change. To ensure long-term cloud cost accountability, your Engineering teams need to make cloud cost a valuable part of their conversations. This might include adding “cost” as a talking point when discussing any new architecture or feature, pulling in one or two optimization recommendations during each quarterly planning session, or spending at least one sprint per quarter reviewing cost optimization opportunities.  You can also reinforce ideal behavior by redesigning systems to promote that ideal behavior. Your technical systems (e.g., CI/CD pipelines, change management, etc.) must make doing “the right thing” the easiest path forward. For example, some teams may consider thinking about cost a major disruption to existing workflows. Use this opportunity to highlight how easily teams can gain insights from your cloud cost aggregation solution. Easy, convenient user experience in technical systems will lower the barrier to entry for most cost conversations. Similarly, leadership can reinforce and socialize exemplary behavior. Research shows that employees are far more likely to engage in an activity if they know other people in the team or department are participating too. For example, one organization we worked with had surprisingly great coverage of their cost allocation tag “product code,” which led to extremely accurate AWS spend reports for each product. These reports empowered product owners, finance teams, and leadership to make data-driven decisions with highly accurate per-product information. This also prompted a healthy level of peer pressure on product teams who weren’t tagging their resources accurately, proving that higher tagging compliance led to better and more useful data for them. All of this work is only viable so long as you maintain it. Think of it like a plant: You have to water, trim, and maintain it over time to make sure it can grow and thrive. In the next part of our AWS Cost Allocation Guide, we’ll discuss  how to keep your cost allocation plan in tip-top shape as your organization  grows and how to train teams to stay compliant. #### AWS Cost Allocation Guide: Identifying Your Costs At The Duckbill Group, our clients repeatedly tell us the same thing: We need to know why our AWS bill went up. In fact, gaining better visibility into your cloud spend is one of the biggest pain points you will encounter with AWS.  For organizations moving from the data center world, AWS is a completely different cost model and requires retraining how you think about costs. Even for cloud-native organizations, different AWS services bill usage in different ways, often unexpectedly so.  The best way to solve this problem—to create that visibility—is by improving your cloud cost allocation practices.  What is cloud cost allocation? Cloud cost allocation is the process of identifying, aggregating, and assigning cloud spend to your organization’s teams, business units, products, etc. When it’s done effectively, cloud cost allocation can have lasting positive impacts on how your business manages cloud costs.  In the next few blog posts, we’ll walk through each step of the process with you to help you understand not just what to do but why doing it is so important. Better visibility into your cloud spend starts with identifying your cloud spend—especially as it pertains to your teams, business units, products, etc. Until you know which teams’ resources live where, you won’t be able to accurately allocate costs to those teams.  Duckbill has seen three successful methods for identifying cloud costs: user-defined cost allocation tags, hierarchical account separation, and third-party tooling.  User-defined cost allocation tags Whether you only have a single AWS account or hundreds of accounts spanning multiple subsidiaries and business units, user-defined cost allocation tags are the best place to start identifying your cloud costs.  AWS won’t show your tags in Cost Explorer (or the Cost and Usage Report) until you enable them in your account, so make sure those tags are enabled in the AWS Billing console as soon as you start tagging resources. Additionally, Cost Explorer doesn’t backfill spend data for tagged resources once they’re tagged, so the sooner you add tags to your resources, the sooner you’ll have more accurate cost allocation visibility.  There’s lots of existing documentation that talks about which tags you should use. But we ultimately want to highlight the importance of the tagging strategy itself.  Tagging doesn’t have to be the worst It seems like every blog post that talks about “how to save money” tells you to tag your resources. But that advice feels much the same as when your doctor tells you to eat healthier.  Of course, we all know we should eat more veggies. But the doctor’s advice doesn’t teach us anything new about veggies. What we need to learn is how to make vegetables taste more delicious so that we’ll choose to eat more of them.  In other words, our veggie consumption (er, lack of) isn’t because we don’t know that we should eat more. It’s because we’re never looking forward to a heaping plate of unseasoned lima beans. Likewise, tagging your resources just for the sake of tagging won’t help your organization because you’re missing the crucial component: strategy. Teamwork makes the (tagging) dream work Tags are for more than just billing or finding out “who spun up this x1e.32xlarge host?” (Trick question! The answer is always you.)  For the best results, start your tagging efforts as a series of holistic conversations with different teams. Bring together Finance, Product, and Engineering/Operations to talk about how each team leverages tagging to benefit their work.  For example, Security might use tags for audits and patch compliance while Finance uses them to identify cloud spend allocated to development work for tax credits. Both are completely valid use cases for different reasons and both generate different requirements for your organization’s tagging policy.  Collaborate to create a purposeful tagging strategy—one that will incentivize Engineering teams to commit to tagging because it will actually enable them to do their job more easily and more effectively. Socializing each team’s tagging requirements provides business context for why including each tag is so important and is a great way for your engineers to create shared purpose and belonging within the organization.  What starts as two teams collaborating can turn into a grassroots effort to improve tagging across the organization, and lead to a community of practice for tagging best practices. Once you have a tagging strategy that everyone agrees on, you need to shout it from the rooftops. Track your tagging success (or lack thereof) at a similar granularity across teams or products. Celebrate teams that tag all their resources, thus improving the accuracy of your cost allocation, and socialize their workflows with other teams who may be struggling to tag or get value from their tags.  For more information, check out AWS’s “thrilling” 24-page Tagging Best Practices Whitepaper. Don’t let the length or the undoubtedly “edge of your seat” storyline sway you from giving it a quick perusal. It includes a slew of helpful use cases for various tagging strategies to fit nearly every business need. We’ve also written our own guide to tagging. Hierarchical account separation Most (but not all!) AWS usage types are taggable. This means even when you tag everything in AWS, you’re still going to end up with “untagged spend” in your cost allocation model.  It’s annoying, right?  How do you allocate that untaggable spend without hours of spreadsheets and resource-specific utilization metrics? By moving resources into separate AWS accounts for each team, product, or business unit.  First, create separate AWS accounts for each of your core cloud-related administrative functions—like security, logging/monitoring, identity, and shared tools (CI/CD). AWS accounts are free, so there are no extra charges here. Next, create accounts in a standardized manner based on your organization’s brands and products. The most common and effective pattern we have seen adopted is one account per environment per product. If your organization has multiple brands each with multiple products, you may want to create one account per environment per product per brand. Should a given brand desire more environments (staging, QA, etc.) or further security boundaries for special cases, additional accounts should be created within the same pattern.  While it’s still not perfect, AWS’s native tooling has made massive improvements in the last few years to make multi-account creation and management a breeze. Access to production or entire brands can easily be controlled with this arrangement. (Remember, account boundaries are the only hard security limits within AWS!) We recommend having some staff with full access to all accounts so organization-wide changes are quick and easy, but still secure and auditable. Third-party tools Third-party tools can accomplish the same goals as the above two methods. But the cost allocation features you’ll want are usually bundled into a larger suite of cost management tools that will cost you 1% to 3% of your total AWS spend.  For example, in addition to helping you allocate your costs, SaaS platforms like Cloudability and CloudHealth provide customizable dashboards, insights, and forecasting capabilities that may or may not be what you need right now in your cost allocation journey. And chances are you’re not utilizing your third-party tooling to its full potential. That’s not a knock against you—it’s just a common occurrence we see with powerful software that has so many bells and whistles. That hefty price tag provides you with solid features. But unless your organization is full of power users for these tools, you’re likely not getting the best bang for your buck. Plus, if you don’t have a strong cost allocation policy in place before utilizing these tools, you’re going to have a much harder time accurately identifying cloud costs within these tools. In our experience, clients have generally found the price tag to not be worth the results. Building blocks for success Whew, that’s a lot to digest!  I know tagging and account migrations doesn’t sound glamorous or exciting. But remember, you have to crawl before you can walk. In other words, you have to identify cloud costs before you can get visibility into your cloud costs. And this process takes time. It’s not just about how you break up your costs, but why you break up your costs—what are you ultimately trying to accomplish by breaking up your costs?  Talk with your engineers. Talk with Finance. Heck, talk with Product and Security and Leadership. All stakeholders should buy-in to your process for identifying cloud costs before you implement it, because all of them will depend on the results. In the next part of our guide, we will discuss implementing your cost allocation process by aggregating and assigning cloud costs. This is the place where your organization becomes empowered to make data-driven decisions on optimizing cloud spend. It’s where the true magic happens. #### AWS Cost Allocation Guide: Tagging Best Practices “Are your resources tagged?” Every engineer has heard this at some point. They may have even said it. Love it or hate it, resource tagging is a topic every engineer has a stance on, from “we’re not big enough for that yet” to “of course! we automatically tag non-compliant resources, too” — and everything in between. But why? Why is resource tagging important? It’s not just about tag coverage, though that’s important, too. Resource tagging is part of a bigger business goal: mapping cloud costs directly to your applications and projects, otherwise known as cloud cost allocation. When it’s used effectively, cloud cost allocation can have lasting positive impacts on how your business manages cloud costs. Not sure where to start with your tagging strategy? Duckbill Group has seen three successful methods for identifying cloud costs: user-defined cost allocation tags, hierarchical account separation, and third-party tooling. Keep reading to learn which makes the most sense for your business. User-Defined Cost Allocation Tags AWS provides two kinds of cost allocation tags for cost allocation purposes: AWS-generated tags and user-defined tags. As the name implies, AWS-generated tags are added automatically when resources are created within certain AWS services (e.g., resources created by a CloudFormation stack automatically receive “aws:cloudformation:*” tags). However, these tags’ keys differ from service to service, so they aren’t a great way of measuring spend across an entire account. User-defined cost allocation tags, on the other hand, are created by you (i.e., the user) and can be applied uniformly across most AWS resources in any given account. This makes user-defined cost allocation tags one of the best strategies for measuring cloud spend accurately across your business’s teams, products, business units, etc. Now the important question: What should your cost allocation tags be? Do you want to identify and measure spend per team? Per product? Per application environment? There are lots of different ways to slice and dice your cloud costs, but it ultimately depends on what’s going to be most helpful for you and your organization. Note we say organization here—not just Engineering. Finance, Product, and even Security or Leadership may have a say in which tags will help them make data-driven decisions. As a starting point, Duckbill Group recommends the following tags: Name - This tag is often built-in for many AWS resources, but it’s oddly lacking for others. Enabling this as a cost allocation tag will allow you to report costs on individual, named resources. Service - This refers to the service a resource belongs to. Along with Name, this tag can have a multitude of potential values; attempting to maintain tight control on all possible values will lead to frustration. Instead, we recommend paying more attention to the consistency of this tag application (e.g., “all resources attributed to service Foo have the tag Service:Foo”) rather than trying to enumerate every possible value from the outset (e.g., “all resources must be deployed with one of the following Service values: Foo, Bar, Baz”). Environment - This tag outlines which application environment a resource belongs to. Possible values might include production, staging, qa, and development. Owner - This tag tracks the email address of the person who created the resource. For teams using Infrastructure as Code tools like Terraform or CloudFormation, this often winds up being a shared user; we recommend ensuring that each user spinning up resources is doing so with their own account so this tag remains accurate. Product - This tag’s potential values should include your product or brand lines, plus an additional value for organization-wide resources like CI/CD, logging and monitoring infrastructure, and SIEM infrastructure. Accounting - This allows Finance to accurately report on accounting lines. Duckbill recommends three possible values: Cost of Goods Sold (“COGS”), Research and Development (“R&D”), and General and Accounting (“G&A”). G&A is usually the default value for this tag, as it includes anything that isn’t directly involved in providing service to a user (COGS) or development of new features for a user (R&D). DataClassification/Compliance - This key denotes whether the resource contains any Personally Identifiable Information (PII) as defined by regulations like GDPR, HIPAA or internal SOC2 compliance. Security will thank you later for knowing which resources contain PII and configuring restricted access on them. Possible values might include sensitive (e.g. resources containing PII) and standard (e.g., resources not containing PII). bucket-name - This key is only used for S3 resources, as it denotes each S3 bucket name. We recommend this key separately from the Name key above to make S3 cost visibility easier. There are two major caveats worth highlighting for this method. First, cost allocation tags will need to be applied to resources and activated in the AWS console before they can be leveraged for reporting purposes. AWS doesn’t backfill historical tag data, so the sooner tags are added to your resources and activated in the console, the sooner you’ll be able to accurately slice and dice your cloud costs. Second, most (but not all!) AWS usage types are taggable because—in typical AWS fashion—not everything follows the same model. This means that even when you tag everything in AWS, you’re still going to end up with “untagged spend” in your cost allocation model. You’re not going crazy; that’s expected. With that said, we’ve seen highly mature organizations tag roughly 85% to 90% of their AWS usage, meaning only 10% to 15% of AWS usage is untagged. This provides enough accuracy in the cost allocation data for teams to make informed decisions. Hierarchical Account Separation Now, we know what you’re thinking: “We’re talking about tagging—not accounts.” As your engineers can attest, tagging doesn’t always provide the level of granular cost visibility desired. However, separating your AWS resources into accounts along business lines provides additional accuracy for your cloud costs. By taking this approach, all spend related to a given business unit lives in a single AWS account and can be easily attributed back to that business unit, whether the resources in that account are tagged or untagged. Win-win. Duckbill Group recommends a set of core AWS accounts to start, plus supplementary accounts as necessary for your organization’s brands and products. Core Accounts The core of a scalable, secure account infrastructure is four accounts: The root payer account ("the management account"), A security account, A shared tools account, and An identity account. Account AliasPurposeacmegroup-rootContains the AWS Organizations configuration linked to all subaccounts, thereby functioning as the root payer account. All organization-wide Service Control Policies are controlled here. Reservations like Savings Plans and Reserved Instances should be purchased in this account to allow them to apply organization-wide to subaccounts.acmegroup-securityContains all security and logging tools (SIEM), vulnerability and audit scanners, and CloudTrail logs. Access should be restricted to security personnel only.acmegroup-toolsContains organization-wide shared tooling, such as CI/CD systems, self-hosted version control systems or configuration management systems, container or AMI repositories, centralized logging, etc. These are often systems that don’t fit neatly in a brand-specific account or an environment-specific account; segmenting these tools off improves security posture.acmegroup-identityThe termination point for all access control. If your organization leverages an SSO solution, it should be integrated with this account via IAM policies controlling access to all linked accounts. AWS SSO is recommended for this. This configuration will allow your organization to track access across all accounts and establish complete audit trails. Brand Accounts Beyond the core account structure, we recommend establishing a consistent pattern for accounts for each brand. The most common and effective pattern we have seen adopted is one account per environment per brand. That might look like this: Account AliasPurposebrandA-prodBrand: Brand A Environment: ProdbrandA-devBrand: Brand A Environment: DevbrandB-prodBrand: Brand B Environment: ProdbrandB-devBrand: Brand B Environment: Dev Should a given brand desire more environments (e.g., staging, QA, etc.) or further security boundaries for special cases, additional accounts should be created within the same pattern. Access to production and entire brands can easily be controlled with this arrangement. (Remember: Account boundaries are the only hard security limits within AWS!) We recommend having some staff with full access to all accounts so organization-wide changes are quick and easy but still secure and auditable. Third-Party Tools While Duckbill generally recommends the above two methods for cost allocation, third-party tools can provide additional value-add—but at a price. SaaS platforms like Cloudability and CloudHealth have some really neat cost allocation features, like attributing non-taggable spend to a given team, product, or business unit, or attributing spend from misspelled tag values to their correct tag value. But if you don’t have a strong tagging policy in place to begin with, you’re going to have a much harder time leveraging these tools to accurately identify cloud costs for each team, product, or business unit. And chances are you’re not utilizing your third-party tooling to their full potential. That’s not a knock against you. It’s just a common occurrence we see with powerful software that has so many bells and whistles. That hefty price tag provides you with all these amazing features, but unless your organization is full of power users for these tools, you’re likely not getting the best bang for your buck. How can you audit compliance? Tagging compliance is hard. Socializing the benefits of tagging may incentivize some engineers to champion your tagging policies, but all the socializing in the world may not help your teams implement and use those tags. For the best results, you may need to take a more targeted effort towards tag adoption. There are two main strategies to enforce tagging within your organization. The first is to enable the continuous monitoring of your AWS resources and notify the appropriate teams and users when their tag usage falls out of compliance. The second strategy, more commonly found in cloud-mature organizations, takes a hard-nose approach towards tag enforcement by stopping or terminating resources that do not meet the predefined requirements. This one requires significant buy-in from the organization because pulling the rug out from your engineers may not win you many friends at the lunch table. There is a case to be made that companies should adopt both of these strategies over time. If you’re just starting your cost allocation journey, you may want to start with monitoring and communication to get everyone on the same page before implementing tooling that stops or terminates resources. Here are some tools Duckbill has seen over the years that can help with either strategy: Cloud Custodian - Cloud Custodian is an open-source solution that allows users to scan a given AWS account for any taggable AWS resources that don’t have a given tag key/value pair and take actions accordingly. The output is clunky and needs massaging before being presented to stakeholders, but it’s the best tool we’ve seen for catching taggable, untagged resources. Plus as your cost allocation practices mature, you can leverage other Cloud Custodian’s other heavy-lifting capabilities—like automatically stopping or terminating resources deployed without the appropriate tags. Want to leverage all these features without running the software yourself? Check out Stacklet, a commercial effort by the creators of Cloud Custodian. AWS Config - AWS Config lets you set compliance rules (e.g., “all EC2 instances need the Application tag”) and tells you when resources are noncompliant. The trouble is that the only built-in rule (i.e., “required-tags”) focuses on EC2 instances. If you want to check for other tags, you’ll need to create a custom Config rule. Plus, the UI is a bit clunky. AWS Organizations Management Tag Policies - Similar to Cloud Custodian, AWS Organizations Management Policies will proactively prevent resources from being deployed with incorrect or noncompliant tags. Note that this solution requires AWS Organizations, and should only be implemented as an advanced practice after you’ve standardized and socialized your tagging policy. As the documents linked above suggest, “It's especially important to understand a tag policy's effects before you enforce compliance with any tag policy.” AWS Resource Groups - This groups resources by user-defined parameters. These groups can be created by Infrastructure as Code tools, but the application of a given tag to resources within a group requires console access—which can break scalable automation and introduce human error. Which strategy is right for you? No matter which strategy you choose to take with tagging your resources, know that the only way to ensure ongoing success is to have a plan for monitoring and compliance—and stick to it. Creating a process or even—gasp—automating your tagging remediation will help make sure you are never falling behind on tag coverage. Every organization is different, so tagging strategies that work for one organization might not work for another. The goal is to get accurate data to decision makers so they can make informed decisions. Still not sure where to start? No worries, we’re happy to help. Schedule a time to chat today. #### AWS Cross-AZ Data Transfer Costs More Than AWS Says How much does AWS data transfer cost between AZs in the same region? It seems like there should be a straightforward answer to this question.  But there’s nuance and ambiguity I keep running into. As it turns out, AWS’s documentation is wrong—or at least, incredibly misleading. For data transfer between AZs, AWS says the following:  And here’s the pricing table between regions: This table clearly suggests that data transfer is twice as much between regions as it is between different availability zones. (Data still wants to get the hell out of Ohio.) However, this is incorrect. As it turns out, they cost the same and the documentation is wrong. How can it possibly be this hard to figure out? For a while now, this topic has been the subject of some debate among those of us who obsess about AWS billing (yeah, we’re all a riot at parties).  Even trying to get clarity on this from AWS is difficult, as the question itself is complex to even articulate. It turns out that finding the handful of people who are able to state a definitive answer is asking for a lot. After a lengthy discussion with yet another AWS Principal Engineer who didn’t know the answer, I decided to just figure it out myself: Let’s chuck 10GB of data across AZs and regions and see what happens to the bill. Hurling data for fun (and...no profit?) I spun up two EC2 instances in a non free-tier account: one in us-west-2a and another in us-west-2b. I told one to listen on a TCP port with trusty netcat: nc -l 7000 > /dev/null.  I logged into the other and pelted 10GB of traffic at the unsuspecting instance with good ol’ dd: dd if=/dev/zero bs=1024 count=$[1024*1024 * 10] |nc -q 0 $OTHER_INSTANCE 7000 When the transfer completed, I shut down both instances and waited a day, since the AWS Billing system isn’t exactly real time (little known fact: there’s anywhere between a four-hour and five-day delay, depending on various things).  For completeness, I spun up two more EC2 instances and repeated this experiment—but this time between two regions (Virginia and Oregon) and in a separate account so as not to deal with bill contamination from the previous experiment. Results The following morning, my bill yielded the answers: I was charged exactly 20¢ for 20GB of data transfer in us-west-2. Wait, what? That should be 10¢, right? Well, not quite: Data transferred "in" to and "out" from ... across Availability Zones ... in the same AWS Region is charged at $0.01/GB in each direction. That last “$0.01/GB in each direction” is the misleading bit. Effectively, cross-AZ data transfer in AWS costs 2¢ per gigabyte and each gigabyte transferred counts as 2GB on the bill: once for sending and once for receiving. As for the inter-region experiment, the results show I was billed 20¢ on the sending side and nothing on the receiving side, which is exactly what you would expect per the documentation.  In summary, AWS’s data transfer pricing documentation has thus been misunderstood for a long period of time--and the cheapest data transfer in all of AWS is between Virginia and Ohio. But I think I’ve finally put this debate to rest. Now, they just need to make this incredibly clear in the documentation; it’s misleading at best today. One thing is definitely clear, though: AWS owes me 40¢. #### AWS Database Savings Plans: Six Years of Complaining Finally Pays Off In 2019, AWS announced Savings Plans. "Great, it supports EC2 and Fargate, now you need to support Lambda" was my immediate reaction--not because (at that time) Lambda was expensive, but rather because it removed a perceived economic reason not to migrate to Serverless technologies. I mean, let's be serious here: "we want to use Lambda, AWS definitely wants us to use Lambda, but the commitments we made to use EC2 and Fargate mean it'd be too expensive for us to move to Lambda" is a story that serves nobody well. In time, Lambda did indeed become supported, and my next reaction was "great, now apply it to RDS for the exact same reason." Little did I know it, but I'd spend the next six years tilting at that windmill. At Some Point It's Worth Doing Just To Shut Me Up Now, at long last, AWS has released Database Savings Plans, and I'm of two minds on it. The Good Parts This is better than it has any right to be, because it's not just for RDS! It supports DynamoDB, Keyspaces, DocumentDB ("Amazon Basics MongoDB"), ElastiCache(I TOLD you it was a database!! Note, this is for Valkey only; no Redis or Memcached), Aurora which both is and is not RDS depending upon how you squint at it, Neptune, Timestream (instance-based InfluxDB only), and for some godforsaken reason Database Migration Service. I notice at this time that it doesn't support Athena, MemoryDB, Route 53, or RedShift. It covers instance hours on a per-hour basis the same way that existing Compute Savings Plans work, and covers serverless versions of the relevant services as well. Discounts run up to 20% for instance-based usage and up to 35% for serverless. That serverless discount is particularly notable: there's mostly been no commitment savings option for serverless databases until now, so this is entirely incremental savings that wasn't previously available to you. Perhaps most notably, it's launching with full support in the Savings Plans Purchase Analyzer from day one. AWS has a long history of launching features half-baked and backfilling the tooling months later, so this signals they actually want customers to use this thing. This is again vindication that the Savings Plan Purchase Analyzer was the most customer-focused AWS launch of 2024, and I will go to the mat on that one. The Bad Parts It doesn't support EC2. I wanted folks to be able to commit to spending a certain amount of money on AWS for a given time period, and then let them reallocate it as best makes sense. Why should AWS care about whether or not a customer's dollar is going to compute or database or network, just so long as it goes to AWS? Customers don't care. AWS as a business shouldn't care. Unfortunately, it seems its feudal lords do care, and they're the ones who make these rules. If a customer wants to move from RDS to EC2 and then back again, why should there be false economic obstacles to achieving that outcome? It's only available for current-gen instances (7th generation and newer—no t4g instances), and only for one-year terms. It'd be nice to lock down database spend for 3 years, since let's face it: databases aren't the most agile of workloads to sling around between environments. It's also only available for no-upfront payment terms. I don't really care about that, but some of our customers will absolutely care about that very much because of how they structure their AWS spend cycle. I'll leave it to them to decide whether or not they want to press Amazon on the issue, and if so, how hard. The exclusion list is longer than I'd like: SimpleDB (yes, that's still a thing), Timestream LiveAnalytics, Neptune Analytics, Redis, MemoryDB, Memcached, China regions, and AWS Outposts are all left out in the cold. It also only covers instance and serverless usage—storage, backup, IO, and other usage types aren't included. And if you're running commercial database engines with unbundled license fees, those aren't covered either—but you're already bleeding money and making weird choices, so may well not care about this. The Bottom Line Despite my grumbling, this is unambiguously where AWS is taking database commitments, and reasonably close to where I've wanted them to go with it. RIs aren't disappearing tomorrow, but the writing is on the wall—and getting ahead of this now beats scrambling when you're forced into it. Go check the Savings Plan Purchase Analyzer and start modeling what this looks like for your database spend. #### AWS Finally Fixes Its Free Tier Problem For eight years, I've watched the same story play out: excited newcomers sign up for AWS thinking they have a free account, only to get blindsided by unexpected charges. I've been beating this drum for years--through public blog posts, making a scene in public, and private conversations with AWS leadership. Now, in what may be the most comprehensive cross-service collaboration I've witnessed, AWS has delivered a solution that actually works. Over the past few months, AWS walked me through their new approach. While I'm admittedly deep in the AWS ecosystem (hard to have fresh eyes after building a career around it for a decade), I channeled my inner newcomer and asked every naive question I could think of. Here's what they've built. The New Sign-up Experience The initial process remains familiar: you create an account and provide a payment method. AWS still places a small temporary hold (around $1) as a fraud check--a surprisingly effective strategy they've used since day one. This requirement also prevents the hilarious scenario of an enterprise customer spreading their $80 million annual spend across half a billion free accounts. $200 in Free Credits (Yes, Really) Here's where things get interesting. You start with $100 in free credits immediately. Want another $100? Complete these five simple tasks: Launch an EC2 instance Experiment with a foundation model in Amazon Bedrock Set up a cost budget in AWS Budgets Create an RDS instance Navigate through the Lambda console (oddly specific, but okay) I knocked out all five in under 30 minutes. More services may offer similar bonuses in the future, but this is your starting lineup. The Six-Month Countdown Your free credits expire after six months--and here's the crucial part: you won't be automatically charged. Instead, you face a clear choice: Option 1: "I'm In" You actively upgrade to a paid account (the traditional AWS account type). You'll still have access to always-free tier resources, but now you're responsible for any overages. The key difference? This is your conscious decision, not a default trap. Option 2: "Thanks, But No Thanks" Do nothing, and your account begins shutting down. Running resources get stopped. Your data remains safe for a grace period (with plenty of email warnings), and you can reactivate anytime by upgrading to a paid account. But without that explicit action, your bill stays at $0. What About Existing Users? If you signed up before this change, nothing happens. Your account continues as-is. All existing free tier offerings, including perpetual free resources, remain unchanged. Why This Actually Matters After years of defending their position, this is AWS quietly acknowledging that their onboarding experience was hostile to newcomers. But this is hardly a surprise; I've spent years telling AWS executives their free tier was broken. What's fascinating isn't that they fixed it--it's how they fixed it that reveals their evolving philosophy. This marks a return to product-led growth rather than focusing on enterprise revenue to the exclusion of all else. Let's be honest: big companies don't care about the free tier. Big companies don't give a shit. But students, hobbyists, and the next generation of developers who are currently starting on simpler platforms? They care deeply. It's also an early indicator that AWS is beginning to understand that different customers have different needs. This is the first step toward a future where big banks and students in dorm rooms are no longer forced through the same onboarding gauntlet. Customer segmentation isn't just for marketing anymore--it's becoming part of the product experience. The Hidden Cost of Doing the Right Thing Here's what AWS isn't advertising: since their billing system still puts the "eventual" in "eventual consistency," they're going to eat some costs. If someone spins up resources in excess of the free tier, AWS might take it on the chin for part of a day before the account shuts down. I posit that AWS can afford this far more readily than some poor kid who accidentally committed their credentials to GitHub. This isn't just good PR--it's an investment in winning back developer mindshare that's been steadily flowing to Vercel, Netlify, and Cloudflare. The Bottom Line This isn't just a policy tweak--it's AWS admitting that surprise bills hurt more than just customer wallets; they hurt trust. Nobody who gets burned by a surprise bill is going to come back to the platform, regardless of whether that bill is forgiven. By requiring explicit consent before charging begins and absorbing some risk themselves, they're removing the biggest barrier to AWS adoption for newcomers. It took nearly a decade to deliver on, but they clearly understood: the cost of a hostile onboarding experience isn't measured in missed $23 monthly bills--it's measured in developers who never come back. #### AWS Lambda Serverless 101 https://vimeo.com/album/5825745/video/322943628 #### AWS Tightens the Reins: New AWS SaaS Marketplace Rules Will Impact Your Commitments Yesterday, AWS lobbed a bit of a water balloon into my inbox. Specifically, they’ve made a change to the AWS Marketplace that was announced to AWS Marketplace seller accounts, but it doesn't seem any customers have been notified of this new development. Here's the email: “AWS Marketplace SaaS Policy Update” The FAQ document they link to is available here, though it has changed since the email was sent. Here's a link to the original, which contains some additional information.  What’s Changing? Purchases on AWS Marketplace count toward contractual spend commitments (commonly referred to as "spend retirement" or "commitment retirement"), with some exceptions. For contracts signed prior to 2022, 50% of Marketplace spend by dollar amount counted toward the commitment, with limited exceptions. The terms changed beginning in 2022 to count 100% of spend, but with a cap at 25% of the annual commitment (a good change!), exclude Professional Services from counting toward commitment retirement, and add a small piece of language—that seemed innocuous until now—that refers to what counts for commitment retirement: "... fees for purchases on AWS Marketplace that are deployed on [AWS services]". Starting May 1st, 2025, only SaaS products hosted entirely on AWS will qualify for commitment retirement, effectively enforcing the above clause customers began agreeing to three years ago. This represents a dramatic shift from the previous requirement, which only specified that "a portion of your application must be hosted in an AWS account that you own." If your product is focused on facilitating workload migrations to AWS, the product must, in order to qualify, pass this section: The product is specifically designed to collect data, replicate data, or migrate workloads from outside AWS to AWS. Except for clients and gateways running outside of AWS, the application and control planes must run on AWS. AWS must be the only available target. If the product also supports replication to environments outside of AWS, you must remove that capability and publish a separate product with that capability. AWS Marketplace won't consider that second product as deployed on AWS. What does this mean for customers? Any customer who relies on Marketplace to help meet their spend commitments now has to be concerned that their SaaS vendors are running entirely in AWS. On the plus side, AWS is providing a mechanism for vendors to get that approval, which will make it easier on the customer, but it is another thing a customer has to think about. No longer can a customer be assured that any vendor spend will count toward their commitment. (And to be clear, AWS isn't preventing customers from purchasing non-qualifying products via Marketplace, but they won't count toward commitment retirement.) More annoyingly is that AWS does not provide mechanisms for customers to track their commitment retirement and, at the time of this writing, does not provide a mechanism to see which purchases in AWS Marketplace are or not eligible (though this is planned to release before May 1, 2025, as mentioned above) (Update on May 5, 2025: AWS now provides a Deployed on AWS badge for Marketplace to identify which purchases are eligible for commitment retirement), so this change does introduce more uncertainty for the customer. What does this mean for AWS ISV/SaaS Partners? Yet more hoops to jump through and requirements to meet, unfortunately. It's been a long-standing complaint from AWS ISV partners that the requirements can be excessive and it seems they just got a little bit more onerous.  One thing that stands out to me are ISVs who operate in multiple providers, allowing the customer to choose which provider the workload lives in—it's unclear how these will be handled under the new rules. Likewise, it's also not clear how ISVs who run concurrently in multiple providers for resiliency reasons will be handled in this new change. Both Azure and Google Cloud have similar policies. Azure's MACC program requires products to be "primarily platformed on Azure", but does not restrict whether a product in Azure Marketplace operates on Azure (the same policy as AWS now), while Google Cloud's is actually more restrictive: the product must operate on Google Cloud to even be listed in Google Cloud Marketplace. When we asked AWS for comment on this article prior to publishing, they suggested it’s common for SaaS products to run provider-specific environments for this purpose, though our spot-checks of common SaaS products seems to indicate this is only common for the largest of SaaS companies (eg, Dynatrace, Wiz—both of which are available in the Google Cloud Marketplace) with smaller companies operating exclusively in AWS (eg, PagerDuty, Arctic Wolf, Honeycomb—none of which are even available in the Google Cloud Marketplace).  It’s also worth noting that all of the cloud providers give deeper private pricing discounts for increased commitments, and so vendors splitting their consumption across multiple clouds under these policies will decrease the discounts they could obtain. And, thus, AWS benefits here more than Azure and Google Cloud by forcing vendors to weigh the tradeoffs on splitting their environment up. The current and new guidelines are available in the AWS Marketplace Seller Guide, with screenshots below for posterity. Current guidelines effective until April 30, 2025 Guidelines effective May 1, 2025 What might this mean for AWS Services Partners in the future? While not explicitly covered in this update, these changes did cause me to think about AWS Services partners (such as ourselves). The Duckbill Group's presence on the AWS Marketplace (and the reason I got this email at all) stems from customer requests to facilitate their procurement process by paying for our services through the AWS Marketplace. While this change doesn't impact us, it raises questions about future restrictions. For example, could similar policies eventually impact Professional Services Marketplace spend unless vendors agree to certain requirements (such as not mentioning AWS's cloud competitors in their marketing)? And that “Professional Services” angle is why I find myself digging into the meat of this change. We spend a fair bit of time helping customers negotiate their AWS contracts, and for a while now we’ve seen Marketplace spend take a meaningful place in their future spend calculations, often with AWS pushing for it.  In fact, according to Forrester research in 2023 (commissioned by AWS!), Forrester found a common reason customers transact through Marketplace is to help with commitment retirement and that customers are increasing the commitments they're making to AWS with Marketplace in mind (which is something I can confirm in our work supporting AWS contract negotiations). AWS further provides a lot of incentives for AWS Partners to transact with customers via the Marketplace. While Professional Services fees don't count toward commitment retirement, it does make me wonder what happens in the future, now that it's clear AWS is making changes that only benefit itself. Where’s the “Customer Obsession”? Which brings me to my main concern: these changes don't seem to benefit customers or partners—only AWS. And they do so by requiring the partner to be running entirely on AWS—and not on a competitor's cloud—or else the customer doesn't benefit from using the Marketplace. No customers benefit from these new rules and, in fact, they're somewhat worse off than they were prior to this. If the ISV partners either have capabilities that go against the new rules or otherwise decline to submit new architectural diagrams to AWS, customers could suddenly lose out on millions in commitment retirement benefits mid-contract cycle. I can see this from AWS's perspective: why should non-AWS spend count toward commitment retirement, if they're not benefitting from the vendor’s spend? But man, how short-sighted and money-grabbing that is! AWS is already making money on the Marketplace transaction fees (which range from 1.5% to 3%, or a staggering 20% for AMIs) for simply being a broker, so why are these new restrictions even necessary? To me, it's entirely a way to gain additional points on their margin without raising public prices—and that's just unfortunate. #### AWS's Valkey Play: When a Fork Becomes a Price Cut In a move that's equal parts predictable and surprising, AWS has decided cut prices for their Valkey-based services significantly cheaper than their Redis counterparts. For those of you who've been living under a rock (or perhaps just sensibly ignoring the never-ending open-source licensing drama), Valkey is the successor fork of Redis, spearheaded by AWS in conjunction with several other players in the space and currently residing at the Linux Foundation. The Numbers Game Let's cut to the chase: AWS has launched ElastiCache for Valkey and MemoryDB for Valkey at significant discounts compared to their Elasticache for Redis version offering. These offer the same features and APIs Redis users know and… tolerate, with trivial migration – just at a lower price tag. It's almost as if AWS discovered that Redis' service margin was just taking up space in their massive bank vaults. The Strategic Play Here's where it gets interesting. By slashing prices on the Valkey versions, AWS is essentially paying customers to switch – and more importantly, to start thinking of "Valkey" as something distinct from Redis. It's a move that's both customer-friendly and strategically brilliant, reminiscent of the Day One AWS of old. Customers get lower costs, and AWS gets to shift the ecosystem towards what's shaping up to be the obvious Redis successor. It's the cloud equivalent of having your cake and eating it too, except in this case, the cake is a third off, AWS gets to keep itself in a leadership spot with regards to a technology, and still turning a profit on the whole thing. The Customer Obsession Angle AWS loves to tout their customer obsession, and for what feels like the first time in forever, they've found a way to align it perfectly with their money obsession. By offering a significant discount on feature-equivalent services, they're giving customers a tangible benefit while simultaneously advancing their own interests. It's a rare win-win in the typically zero-sum game of cloud economics. Comparing Apples to Significantly Cheaper Apples This isn't AWS's first pricing rodeo, but it's certainly one of their more interesting ones. Usually, AWS price cuts are about as exciting as watching paint dry – a fraction of a cent here, a microscopic percentage there, and generally in some subset of far-flung regions where relatively few customers actually run workloads. This move, however, is substantial enough to make even the most jaded Cloud Economist raise an eyebrow. The Technical Nitty-Gritty Let's not forget the technical side of things. Valkey isn't just Redis with AWS's name slapped on it. As per GitHub and as of this writing, it's got roughly 10 times as many contributors and is hundreds of commits ahead of Redis. It's like Redis, but with more caffeine, a bigger dev team, and a community that's suddenly not beholden to keeping requested features stuffed behind a paywall. The Bottom Line AWS has managed to pull off something rather clever here. They're providing significant cost savings to customers, pushing their strategic agenda forward, and still likely making a hefty profit. It's a masterclass in cloud economics that doesn't require a PhD to appreciate – just a willingness to switch to a product with a slightly sillier name. In the end, this move shows that even in the cut-throat world of cloud computing, there's still room for surprises. And if those surprises come with a massive third off the price tag, well, who are we to complain? #### Buying Reserved Instances just got a lot easier. Yesterday evening Amazon announced a change to how reserved instances work. You can now modify your environment on the fly while still taking advantage of RIs. For example, you have an m4.xlarge reserved instance. You can now apply that reservation to four m4.medium instances instead-- or vice versa. In the inverse (you have an m4.medium reservation and switch to an m4.xlarge) the instance reservation discount will still apply-- you'll get a discount on 1/4 of your usage of that instance, and pay on-demand for the remaining 75%. AWS "does the right thing" here-- but it's going to make unwinding your monthly bill a bit more convoluted! Benefits: This applies to existing RIs you've already got, effective for this month's billing cycle. You can rearchitect your instance size. Caveats: This only works inside of instance families. You can't use an i4 reservation for a m3.medium. As a result, you still need to determine the general performance profile of your application prior to locking in an RI purchase. This only applies to shared tenancy. If you're using dedicated hardware for your instances (generally used either for security reasons or to qualify your cloud spend as CapEx) this doesn't apply to you. "Instance Families" only apply to a single generation. When i3 instances came out last month, they offered significant cost savings over the previous generation i2 instances-- but i2 reservations won't work here. As a result, 1 year RIs continue to be my default recommendation unless you really know what you're doing. #### Cloud Cost Compass: Duckbill's FinOps Maturity Assessment Cost assessments are sort of our bread-and-butter here at Duckbill, both in identifying ways to reduce costs and, more importantly, assessing an organization’s ability to manage its own costs. About two years ago, we decided to formalize our assessment methodology into a structure we call Cloud Cost Compass. Since then, we’ve run this assessment on many dozens of organizations, with monthly AWS spend ranging from $100,000 per month up to several tens-of-millions per month. Today, we’re publishing our assessment methodology. Two interesting takeaways We learn two interesting things from our experience running this assessment on so many organizations. First, high-performing organizations are not high-performing everywhere, and vice versa. The reality is that even in low-performing organizations, there are always pockets of high performance. In high-performing organizations, there are always pockets of weakness. Second, organizations have a tendency to regress their performance during periods of business growth, creating a need to rethink their approach to managing costs. This makes sense, of course: innovation is inherently wasteful, and growth is often better than cutting expenses. As growth slows down, the need for efficiency comes back into play. Criteria Cloud Cost Compass contains six areas to assess.  Compute OptimizationCompute Optimization is a measure of compute density, intentionality of choices concerning various kinds of available compute resources, and use of ephemeral compute resources.Data OptimizationData Optimization measures how well an organization understands and leverages the available types of AWS data storage and understands and optimizes for the architectural and financial impacts of data transfer. CloudinessCloudiness is a measure of how an organization uses AWS, employs modern cloud practices, and level of continual improvement. Does the organization treat AWS simply as another data center by only leveraging the most basic primitives of AWS? Or does the organization make extensive use of the higher-order services? The more the latter is true, the more optimized cloud costs tend to be.Resource Purchasing CommitmentsResource Purchasing Commitments measures the extent to which an organization takes advantage of the significant discounts AWS provides for customers who are willing to commit to a base standard of usage. This includes Reserved Instances, Savings Plans, and private pricing agreements (EDP/PPA).Cost VisibilityMaking costs more visible across the organization is an important aspect in understanding and controlling those costs. Companies that make investments in visibility strategies fare better with understanding and managing AWS costs. This criteria also covers cost allocation.Collaborative Cost ManagementMeasures the strength of the collaborative relationship between Finance and Engineering when it comes to AWS cloud costs, particularly decisions and understanding related to unit economics, forecasting, business seasonality, and each department’s ability to influence spend. Rubric We score an organization on a 1-5 scale for each area. We intentionally leave 2 and 4 empty to provide room for in-between scores. The rubric is subjective, as it’s designed to be used and interpreted by people with expertise in the subject matter rather than run by some automated system or a junior-level team member. 12345Compute OptimizationNearly all of our infrastructure is on long-lived EC2 instances. We treat AWS as just another datacenter.Extensive use of AWS-managed services relevant to the environment’s use cases.Extensive use of IaC tools and practices.Extensive knowledge across the Engineering org with respect to AWS/cloud design and architecture principles and practices. Established architectural review processes.Instance types are intentionally chosenAnything that can have a lights-out has oneServerless is used as much as is practicableHigh utilization of low-cost process architectures (eg Graviton)Maximized compute densityData OptimizationWe almost exclusively use EBS storage or EC2 instance-backed storage. We leverage S3 only for backup purposes.No S3-IT, no EBS/S3 lifecycles. Basically, AWS defaults.No understanding of data transfer or data lifecycle.Consideration given toward use of various S3 storage classesEfforts made toward understanding and classifying data usage patternsEfforts made toward understanding data transfer patternsExtensive/predominant use of lowest-cost S3 storage classes.Demonstrated understanding of data usage patterns (instance store, EBS, S3, etc).Data transfer costs are well-understood and architectural decisions are data-driven. CloudinessWe use only the most basic AWS primitives, like EC2, S3, EBS, and RDS.Use of old-school orchestration systems (eg, OpenShift) or hypervisors (eg, VMware)Moderate use of AWS-managed services relevant to the environment’s use cases.Moderate levels of IaC tooling and practicesModerate levels of knowledge across the Engineering org with respect to AWS/cloud design and architecture Extensive use of AWS managed services relevant to the environment’s use cases.Extensive use of IaC tools and practices.Extensive knowledge across the Engineering org with respect to AWS/cloud design and architecture principles and practices.Established architectural review processes.Resource Purchasing CommitmentsWe have not purchased a Reserved Instance or Savings Plan yet. Everything runs on-demand.No private pricing contracts, though eligible for them.Ad hoc purchasing of RIs/SPs for eligible services.No assigned owner for contracts, but contracts are in place.No assigned owner for RIs/SPs.RI/SP coverage consistently >75% and utilization consistently >80%RI/SP coverage consistently >90% and utilization consistently >90%We have an assigned owner whose job is to forecast and maintain RI/SP purchases.We have an assigned owner for the AWS bill and contract and have high confidence we have the best-for-us deal possible.Cost Visibility I get a bill every month and I pay it (sometimes on-time!)One sole owner of AWS spend, but it’s at a VP/CTO level. No defined process for spend visibility/management.Tagging strategy in place but inconsistently applied with low to moderate coverage.Low-to-moderate ability to map spend to teams/products/services. Teams do not contribute to managing their spend in an active way.We have a dedicated role/team for understanding spend.We can accurately map spend to teams/products/services/projects with ease. These teams/products/projects are aware of their spend and take an active role in managing it.Strong understanding of our cost model and how that influences our AWS private pricing and contractual commitments.Collaborative Cost ManagementRelationship between engineering and finance is either non-existent or actively hostile.Finance has no data on cloud spend beyond the AWS bill.Finance has no influence on cloud spend. Engineering has no influence on budget.Unit economics are undefined.There is no cloud spend forecast.No understanding of business seasonality as it applies to cloud cost.Engineering and finance regularly communicate about cloud spend, primarily to share information.Finance periodically gets data on cloud spend from engineering.There is an established, regular way to set goals on cloud spend. It’s not collaborative, but it’s functional.You have a unit economics story that somebody set, but it’s not clearly understood how you got to it or how accurate it is. It’s not really used outside finance.You have an unreliable forecast. Basic understanding of business seasonality as applied to Engineering and Finance concerns.Engineering and finance effectively communicate about cloud spend.Finance has a self-service way of looking at cloud spend by cost center, customer, etc.Finance and engineering actively collaborate to influence/set goals on cloud spend in response to broader organizational pressures.You have a unit economics story, both eng and finance know how you got it, and either eng or finance can influence it. Unit costs play an active role in decision-making across the organization.Engineering and finance actively collaborate to continually refine their cloud spend forecast.Demonstrated ability to tie business seasonality with Engineering and Finance concerns. Want us to assess your organization? Just reach out and we’ll work to get you scheduled. #### EBS Snapshots Now Support Cost Allocation Historically, one big item in AWS cost reports that wasn't able to be assigned various costing tags has been EBS snapshots. The reasoning behind this was presumably tied to their differential nature-- it's presumably non-trivial to allocate thousands of very small amounts of data to a cost tag, nor was it a particularly high priority. After all, unless you're doing something strange you're unlikely to be spending significant amounts of money on snapshots. Of course, AWS is vast enough that many of us are indeed doing something strange! In some cases I've seen snapshot costs rise into the tens of thousands. As of today, tags propagated to your snapshots now act as cost allocation tags once enabled in your billing dashboard. Combined with a tool (I'm partial to Lambda functions) that propagates tags to from instances to EBS volumes, and then further to snapshots, you get a more accurate cost picture for various environments, business units, and projects. #### EC2 Reserved Instances are Being Quietly Deprecated Something strange started happening at re:Invent last year. Yes, I realize that absolutely doesn’t narrow it down one bit, but this one is germane to my Cloud Economics day job. Specifically: new EC2 instances started launching without Reserved Instance support. If you check the pricing pages for the trn2, p5e, i8g, f2, and i7ie instance families, you’ll note that they don’t show up with reserved instance pricing—just Savings Plan discount rates. The writing’s been on the wall for RIs for a while now. Savings Plans are the way forward; they offer the same discount percentages as Reserved Instances, with a lot less manual work to get there. True, they don’t offer the Convertible RI Double Irish with a Twist*, but I can’t help but feel that this is considered a benefit by folks at AWS tired of customers’ doing an end run around their commitment structures. I’ve wondered for a while how AWS was going to get to a point of Savings Plan adoption as the single path forward. It was pretty obvious that there wasn’t going to be a hard cut-off of no more RI purchases past [some arbitrary date], so phasing them out this way seems to be the most customer-centric solution. Note that a far more customer-centric solution would include folding RDS, OpenSearch, Redshift, and other services that use their own bespoke RIs into a unified Savings Plan too—but it’s clear that those teams are too interested in empire-building. We’ve seen where that leads, with SageMaker-specific savings plans: customers have to decide up front whether or not they’re going to build on EC2 or a more managed service. Unlike Fargate or Lambda, which quite sensibly folded their discount structures into Compute Savings Plans, these teams built their own structure that locks customers into architectural decisions via economic factors—in many cases, keeping customers from adopting services that both AWS and the customers themselves would prefer. For customers, this is unlikely to be a significant change. Case in point, it’s been four months, and I haven’t found folks to be particularly concerned about the change. Most of them are learning about it from this post. It just doesn’t matter to most customers in 2025; they’ve switched to Savings Plans years ago, and are happier for it. However, this likely is the turning point for an awful lot of businesses that model RI spending in “creative” ways. In a few years when Savings Plans are all that exist on modern instance types, you won’t need some vendor taking a percentage of your AWS bill; you’ll use AWS’s own free (and excellent) Savings Plan Purchase Analyzer, take the recommendation, and then purchase them on a reasonable cadence. Our own Reserved Instance guide will become an interesting historical footnote, as its companion Savings Plan guide becomes the only relevant document of the two. Overall this is a win for customers; they don’t have to stop what they’re doing today, but they absolutely should be keeping an eye towards the future of AWS discount vehicles—and modernizing their processes to suit. *The Convertible RI Double Irish with a Twist: Buy a tiny RI every month on a no upfront 1 year. In the 11th month, exchange that tiny RI for, say, 10K c8G.24xlarge instances. Yes, you can turn a tiny RI into many much larger ones, without resetting the commitment length. You get all the discounting of the RI, with only a one month commitment for that spend level. It's super handy for variable workloads, and isn't hard to automate. #### Every Byte You Take: How Cloud Ingestion Pricing Eats Your Budget Remember when large enterprises had to buy tech like they were doomsday preppers? "Quick, we need enough servers to survive the digital apocalypse!" Massive buildings full of blinking lights, waiting for the day your app might go viral on ... well, whatever replaced Slashdot in your worst-case capacity planning nightmares. The cloud made data management cheaper and easier for companies — right? The answer isn't actually straightforward, and it depends heavily on how your developers are moving data.  The old school way: On-Premises Let's talk about how you've probably been doing data management since before AWS decided to rent out its "spare" computers. You're basically shopping for servers like you're furnishing a mansion. "Hey boss, we need $500K for some shiny new servers! Why? Because maybe we'll need all that power ... someday ... probably. Look, just sign the PO before Dell's quarter ends and it stops pretending these are special prices." And the expenses don't stop there. You're signing up for: Electricity bills that could power a small city. A cooling system that turns your data center into a meat locker 24/7. IT folks who get paged at 3 a.m. because a disk is at 82% capacity instead of the allowed 80%. Support contracts and spare parts that make your wallet cry (because Murphy's Law is real). The best part? Whether your servers are working harder than a caffeinated squirrel or just collecting dust, you're paying the same amount for all of those costs. Sometimes it's like having a gym membership you barely use — except this one costs millions. Enter the cloud: Pay-as-you-go paradise? Then along comes the cloud, strutting in with the whole "Hey, what if you only paid for what you actually use?" sales pitch. Which sounds great until you realize the pricing models make the U.S. tax code look like a children's bedtime story. And here's the catch (there's always a catch): You might now have to deal with the world of ingestion-based pricing! If you're as old as me, you can think of it like the original cellular plans where every text, call, and cat video you stream counts. Except instead of cat videos, it's your log files, API calls, and data transfers. Take AWS CloudWatch Logs, for example. Think of it as a fancy librarian sitting in front of a huge data warehouse, except this librarian has a weird pricing model. "Oh, you want to hand me those debug logs to file away? That'll be $0.50 per GB plus $0.03 per GB per month to keep them on the shelf." Everyone gets worked up about that monthly storage fee, but here's the thing — you're sweating over pennies while dollars are flying out the door.  Here's how things can spiral out of control: A well-meaning developer adds some debug logging to their Lambda function. Just some innocent messages: function start, database query, data lookup, final calculation. Reasonable, right? But then they spot an issue and decide to log the entire database query result — a mere 10MB of debug data PER REQUEST. Suddenly your "serverless" function is generating enough logs to fill the Library of Congress, and your AWS bill looks like a phone number. Just remember that moving data around gets expensive, whether it is out to the internet or between your own servers. And don't even get me started on cross-zone data transfer costs — it's like paying toll fees for your data to cross virtual bridges.  How to optimize your cloud bill by limiting data movement Storage isn't the budget-killer; it's data movement. That means your verbose developer (you know, the one who logs every user blink) needs to channel their inner haiku master. Sure, logging is valuable, but before sending any data to services like CloudWatch Logs, ask: Do we really need this data, or are we just logging it because adding print statements is easier than using a debugger? If yes, when will we actually use this data? (Hint: "Never" is a valid answer.) Have you checked that you're not accidentally logging credit card numbers? (Because storing PII in CloudWatch is both expensive AND a compliance nightmare.) Minimizing your data movement is the surest way to slow your cloud ingestion costs and reduce unnecessary spending on your AWS bill. On-prem vs. cloud: Which is better for the bottom line? Is cloud cheaper than on-prem? The cloud is like having a DoorDash: incredibly convenient, scales to your needs, but definitely charges you for every single delivery. On-prem is like buying your own restaurant: huge upfront cost, but hey, at least you know exactly how much you're overpaying, assuming you can accurately account for the overhead, maintenance, and that one chef who keeps breaking things.  Essentially, yes, the cloud can be cheaper — but only if you're using it the right way. Remember: In the cloud, every byte counts, every AZ transfer costs, and somewhere, an AWS service team is probably inventing a new way to charge you for something you thought was free. Welcome to the future — it's billed by the millisecond. #### Figma's $300k Daily AWS Bill Isn't the Scandal You Think It Is Well, the internet did what the internet does best this week: it collectively lost its mind over a number in an S-1 filing. Figma disclosed they signed a ~$550 million contract with AWS, someone used arithmetic (the secret weapon of Cloud Finance) to determine that this was roughly $300,000 per day on AWS, and suddenly everyone with a social media account became a cloud economics expert. "FIGMA IS DOOMED!" they cried. "FIRE THE CTO!" they demanded. "They're gonna blow through their commitment in record time if they spin up another Managed NAT Gateway," I observed. Here's the thing: I negotiate AWS bills for a living. My company, The Duckbill Group, has seen more AWS invoices than most people have seen Marvel movies. And I'm here to tell you that Figma's spending is about as scandalous as "Mr. Roger's Neighborhood." Let's Talk Numbers (The Real Ones) Figma's S-1 filing revealed a $545 million five-year AWS commitment. That's about $109 million annually, or roughly 12% of their $821 million in rolling revenue. For those keeping score at home, that's supporting 13 million monthly active users and 450,000 paying customers while maintaining a market-leading 91% gross margin. But sure, let's panic about the AWS bill. While we’re at it, let’s also make sure we get the math wrong in two key ways. First, let’s ignore that most AWS contracts increase year-over-year (if they don’t, the company either isn’t growing, or doesn’t expect to grow much further, which is absolutely not Figma), so the idea that they’re currently spending $110m under this contract year isn’t reasonable. It’s far likelier it starts at something like $80m a year, ramping up to ~$150m a year in Year 5. Second, what everyone’s missing is that these spend targets are post discount. Remember that Figma didn’t sign a half-billion dollar non-cancelable contract for funsies. At this scale, with no specific knowledge of their deal, they likely have something on the order of a 30% effective discount on their cloud spend. So when you scroll through AWS’s byzantine pricing pages doing mental math, you can think of Figma spending not $550m, but $785m at retail prices. The best part? Rather than pointing out realities like this, someone on Twitter instead claimed Figma only makes "$220m a year in revenue" (actual: $749M) while spending "$110m a year on AWS." Even Twitter's community notes had to step in, which is like being fact-checked by a YouTube comment section. This Is What Normal Looks Like Here's what the pearl-clutchers don't fully grasp: 12% of revenue on cloud infrastructure for a compute-intensive, real-time collaborative platform is completely reasonable. We see this all the time with our clients. Industry benchmarks show: Compute-lite SaaS companies: ~5% of revenue Compute-heavy platforms: 10–15% of revenue AI/ML-intensive companies: Often exceeding 15% (but do note that it's so early days right now it's hard to speak definitively here) Figma is rendering complex designs in real-time for millions of users with sub-100ms latency. They're not running a blog on a t3.micro instance. This is heavyweight technical infrastructure, and it costs what it costs. The "AWS Dependency" Non-Story My favorite part of this whole saga is the breathless reporting about Figma's "risky dependency" on AWS. The S-1 contained standard boilerplate language about vendor dependencies that appears in virtually every cloud-dependent company's SEC filings. You know, the kind of language lawyers make you include about how your business could be affected if your primary infrastructure provider suddenly decides to take up goat farming instead, or your CEO is Elon Musk. Breaking news: SaaS company uses cloud provider, could be affected by outages. In related news, restaurants report troubling dependency on food suppliers. Let's Add Some Context, Shall We? Want to know what real AWS spending looks like at scale? Snowflake has a $2.5 billion commitment from 2024–2028 Apple was spending an estimated $360 million annually back in 2019 Snap got over their skis by making $3 billion in cloud commitments, spending over 40% of revenue before having to renegotiate But yes, let's all clutch our pearls about Figma's completely reasonable infrastructure costs. The Armchair Architects Have Logged On HackerNews commenters claimed they could cut Figma's costs by "at least 30%, often more than half." Sure, Steven; that seems credible. I'm certain your experience running a Minecraft server uniquely qualifies you to architect infrastructure for 95% of Fortune 500 companies. These are the same people who think they could rebuild Facebook over a weekend if only they didn't have their day job at the DMV. Here's a hint: you might have a misconfiguration costing you 20% of your AWS bill when you're dropping $80K a year, but by the time you're annually spending nine figures, somebody has asked where the hell all the money is going. Here's What Actually Matters Figma has documented optimization efforts including transitioning from Ruby to C++ pipelines, and migrating workloads that don't need it away from GPU-based instances. They're implementing dynamic cluster scaling. They're doing the work. More importantly, they're growing at 46% year-over-year with a 91% gross margin. If you're losing sleep over their AWS bill while they're printing money like this, you might want to reconsider your priorities. Once again the innovation <--> optimization continuum rears its head. If you enjoyed this analysis, subscribe to Last Week in AWS for more cloud commentary, absurdity, and actual insights that won’t make your finance team cry. 👉 Subscribe Here The Bottom Line After years of helping companies optimize their AWS spending, I can tell you definitively: Figma's infrastructure costs are boringly normal. The real story here isn't their AWS bill—it's that they were transparent enough to disclose it when most companies hide behind vague "infrastructure costs" line items. The hyperbolic reactions to this disclosure reveal something more concerning than any AWS bill: a fundamental misunderstanding of what it actually costs to run enterprise-scale infrastructure in 2025. So the next time someone tries to tell you that a company spending 12% of revenue on the infrastructure that literally runs their entire business is "doomed," maybe ask them how much they think it costs to serve real-time collaborative experiences to 13 million users across the globe. Spoiler alert: It's not cheap. And that's perfectly fine. #### FinOps Is Solving the Wrong Problem We've long been known for our hot takes on cloud at Duckbill, and here's the hottest one: the cloud isn't expensive. "Expensive" implies a judgment, a comparison. If cloud is expensive, what's it expensive compared to? Every argument we’ve heard is one considered in isolation, rather than in the full context of the business. Cloud spending isn't inherently wasteful. It’s entirely proper to be deliberately aggressive with cloud consumption if it enables faster time-to-market, supports rapid scaling, or frees up engineers to focus on product development rather than infrastructure. The state of FinOps tooling today is missing the forest for the trees. It’s like everyone started with "wow, cloud bills are big, wouldn't it be great if they were smaller?" and drove right past whether that even actually matters. We don’t think it does. We’re at war with complexity Business leadership doesn't really care whether the cloud bill is $1m or $100m in isolation. They care whether they can predict it. They care about healthy gross margins. They care about having a meaningful ability to influence the numbers. A business can easily work with a large, predictable expense. What keeps executives up at night are expenses that move in ways they can't explain, forecast, or control. This is the problem consumption-based billing created. A data center is financially boring in the best way. You buy racks of servers every three to five years. It's a known quantity. Cloud infrastructure changed that. Now infrastructure cost is a function of customer behavior, engineering decisions, market conditions. Seriously: the need for companies like ours is an indictment of the complexity this billing model created. AWS alone has 2.3 million SKUs, all metered by the hour. A one-time $1M capital expense, a $100K/year subscription, $375 per user per month: finance can model all of these. But "it depends on how many API calls we get" is something else entirely. And it's not just cloud infrastructure anymore, either. More vendors are moving to consumption-based pricing every day, making the problem substantially worse. Cloud cost management is more than finding savings When we look at the work our customers actually do day-to-day, cost reduction work is a small slice of it. Instead, they’re: building budgets, what-if scenarios, and adjusting forecasts investigating variances between actual spend and projections consulting with engineering teams on how planned product changes will affect the bill creating enablement materials advising on governance negotiating with vendors handling invoicing and internal chargebacks improving KPI measurement and stakeholder reporting The industry has built tools for finding savings but left practitioners to figure out the other 90% on their own. That’s a real shame. https://www.youtube.com/watch?v=KNnl6A0VJ3Y Skyway: Cloud Cost Management For the Nine Figure Club As we’ve hammered on in this post, we believe the fundamental problem of enterprise cost management isn't minimizing the size of bills. It's making consumption and spend data explorable, predictable, and actionable across the entire organization for a variety of use cases. And not just for cloud, but all infrastructure: Databricks, Anthropic, Snowflake, New Relic. We’re aiming for complete coverage of every vendor, not just AWS/GCP/Azure. Finance needs this data for budgeting and forecasting. Engineering needs it for optimization and architectural decisions. Procurement needs it for demand planning and negotiation. Product needs it to understand unit economics. FinOps needs it to play their role of conductor across all of these use cases. These are different questions asked by different people with different mental models—and none of them are well-served by a dashboard telling you to shut off some idle instances. Meeting these needs across dozens of providers, serving hundreds of stakeholders, requires a data platform—not piecemeal, app-based point solutions. We’re designing Skyway to be an open integration point inside of a business’s existing processes and tools, not a walled garden. We know we can never pry a spreadsheet out of anyone’s hands, so we’re not even going to try. Instead, we’re providing the entire business with more reliable data in an easier, quicker, and cheaper way. Skyway’s Contract Manager We have a lot on our roadmap and our first module, Contract Manager, is available for customers now. Contract Manager turns cloud private pricing contracts into structured data, alongside your cloud consumption and billing data. This enables you to: measure private pricing commitment performance validate you’re receiving your discounts and that your bill is correct track individual discount impact manage credit attestations identify new private pricing discount opportunities enable the renewal and negotiation process Curious if Skyway is right for your team? Get in touch with our team to get started. Let's talk We’ve raised $7.75m to build our vision To help build our vision of a better future, we’re also announcing that we’ve raised a $7.75m seed round from Joseph Ruscio at Heavybit, Andy McLoughlin at Uncork Capital, and Alex Benik at Encoded Ventures. We’re further supported by some delightful angel investors who are excited to back our mission, including Mark Imbriaco (Head of Engineering at Railway, former Sr. Director of Engineering at Epic Games, VP Tech Ops at Digital Ocean), Adam Gross (founder/former CEO of Heroku), David Singleton (CEO of /dev/agents, former CTO of Stripe), Brian Hall (former VP Product Marketing at Google Cloud and AWS), and other notable leaders from companies including AWS, Airbnb, Instacart, Block, Confluent, Epic Games, and Cribl. We're growing. We're building our San Francisco team and looking for people who thrive without a playbook, say what they mean, and aren't afraid to figure things out as they go. We take the work seriously but not ourselves—if that sounds like you (or someone you know), let's talk. Join the team #### Freedom's Expiration Date I got an email last week from AWS telling me that since my Last Week in AWS account is now a year old, its qualification for the 12-month free tier is expiring. All good things must end, and I was expecting this. "If you want to estimate your monthly bill," ends the email, "you can use our forecasting tool in the billing console." Wonderful-- I'd love to be prepared for the inevitable bill shock next month is going to bring. And this is where things went to custard, and inspired this blog post. I can see very clearly what my current free tier usage looks like. For example, for S3 I'm making a lot of GETs and PUTs-- those will cost me $0.004 per 10,000 for the former and $0.005 per 1,000 for the latter.. I'll pay $0.023 per GB a month for the little storage I use, and data transfer out is negligible, as traffic from S3 to CloudFront is free. And this is for S3. I use a lot of other services! To be very clear, I estimate my cost of free tier services to be in the tens of cents range, and I don't particularly care about the economic impact here-- but I know the AWS billing model extraordinarily well (I consult on AWS bills for a living!) and I still have to sit down with a calculator to figure out what my expiring free tier is going to cost me. My point is that it wouldn't take a lot for AWS to add a column to the billing console's "Free Tier Usage" section that tells me what this will cost me at the end of the month rather than leaving me here wondering whether my next bill is going to be 48 cents or 48 dollars. #### How ConvertKit Could Lower Its $64K Monthly AWS Bill So earlier this week, ConvertKit made an excellent post about their AWS bill. This level of transparency around their bill is, to put it lightly, unusual. Others have commented extensively (oh hi there Hacker News! What are you doing here? Go back to first principles!) on this, but given that this is what we do for a living here at The Duckbill Group, I figured I'd share my thoughts, too. In the interest of full transparency and disclosure of my own, Last Week in AWS is a happy ConvertKit customer. Go read their post; I'll wait. Back yet? Okay, let's dive into this. If they were a client, my initial impressions would be along the following lines. 1. Consider Spot for anything that's autoscaling It saves way more than buying reserved instances does, and it also means you have no commitments. That said, this isn't necessarily a slam dunk. Any workload that's comprised of nodes that can't withstand interruption is absolutely not a candidate for this, but with how much money can be saved here, it's worth investigating whether an architecture change is worthwhile so you can take advantage of Spot. 2. Reserved Instances Notice how they describe their clusters in terms of the instance types that exist in them? The i3.2xlarges are their Cassandra and Elasticsearch clusters, their t3.medium nodes are their baseline workers for a bunch of things with intermittent loads, and so on. Their r4.xlarge instances are instances “they never really needed but were stuck with due to a mistake made while reserving instances last year." It doesn't help now that they're over with it. But if this happens to you, reach out to AWS support. They're surprisingly good at helping fix mistakes like this so you don't have to eat them. Along that same line, ConvertKit keeps saying "once they figure out their baseline usage they can buy some reserved instances and start saving money." Stop that! If Spot isn't viable for a workload, go out and buy RIs today. Worried you'll overbuy? Okay, buy 20% of what you think you'll need, wait a week, and try it again. 3. Let's talk about those Elasticsearch clusters ConvertKit mentions having massive volumes of data to search through. I don't know what their workloads look like, but if they've got a bit of latency sensitivity, I'd consider CHAOSSEARCH, out of Boston. This company provides an Elasticsearch-compatible API on top of data that lives in your S3 buckets. Given that a gigabyte of data in S3 costs 23% of what a gigabyte of data on an EBS volume does, there's a storage cost win. But I'm not finished. By divorcing compute from the storage, they don't have to manage the crappy parts of Elasticsearch—by which I mean "all of it." Based upon what I've seen, they'd likely cut their Elasticsearch bill by something like 80% after paying CHAOSSEARCH, but it'd require a bit of investigation. In the interest of continued transparency, CHAOSSEARCH sponsors my podcasts and newsletters. But, truth be told, I've been recommending them long before that happened. I have no partnerships with any vendor in the cloud space; if I recommend a product, it's because it's what I'd use if it were my environment. 4. ConvertKit waited on a C4 --> C5 migration until RIs ran out Never do this! If you're letting a sunk cost dictate what you do architecturally, you're hamstringing yourself in most respects. People's time is worth more than what you spend on infrastructure. If there's a capability gap, fill it; don't wait for instances. If you want to avoid having to have this debate, go for three-year convertible RIs; all you're committing to at that point is using some amount of EC2 in a particular region. You're not bound to any particular instance families. Given that there are roughly 200 different instance types in us-east-1 alone, this is the most flexible path. It offers better discounts than a one-year standard reserved instance does. 5. Beware the Managed NAT Gateway Oh, honey. Don't use this terrible service for anything at scale. "We're sending 1.1TB of data per day through our NAT Gateway," ConvertKit says. Of course, they're paying data transfer costs on this data (depending on where it's going, that's anywhere from 0¢ to 9¢ cents a gig, under the Data Transfer section of the bill), *plus* 4.5¢ per gig in data processing fees. Replace these with NAT instances you manage yourself (or move the workload to a public subnet) and that 4.5¢ charge vanishes. Voila. Plus, if they haven't configured gateway endpoints for S3, anything they write there passes through the gateway, too. That's right, it applies to data destined for other AWS services! This is incredibly complicated—but borderline abusive from a billing perspective. 6. Enable S3 Data Analytics Go into the largest S3 buckets and turn on S3 analytics. In only three short months, it will start spitting out recommendations around tiering changes, unlocking sensible S3 infrequent access lifecycle policies that save money. The rest of this section is largely around data transfer. I'd want to delve deeper before making additional recommendations, but there's almost always an optimization story here. There's just not enough information in their post to comment intelligently past that point. Aside: The RDS RI model is bonkers This is the part of the post where I take AWS behind the woodshed for their horrible reserved instance model. ConvertKit just bought RIs in July. AWS is pretty clearly incentivized to drive business away from traditional MySQL databases and over to Aurora, their flagship database named after a Disney princess. They crow about it breathlessly in virtually ever conference keynote they give. This is fine. But because of their RI model, if ConvertKit would benefit from an Aurora migration (and while it’s highly workload dependent due to their 20¢-per-million-IO-operations, statistically they’d lower their RDS bill significantly), they wouldn’t consider it until next July at the earliest because they'd be leaving RI money on the table if they did. Smooth, AWS. Real smooth. Final Thoughts A few other thoughts that struck me about their environment, but I don't have enough detail about their environment to do a proper analysis: ConvertKit may want to take a look at what regions they're in. If they can get by with us-east-1 and us-east-2 as their two regions, cross-region data transfer is half-price between them, as even data wants to get the hell out of Ohio. Capitalize on that. If ConvertKit goes with Spot, they're not going to want to use t3 instances due to economic reasons. The t3 instance type starts with a depleted CPU credit balance, so that’d cost way too much money when they exceeded the baseline. Look at ConvertKit’s historical usage of AWS Support. If they're getting value from it, by all means keep it. If not, turn it off. They can always turn it back on later. I'm suspicious of ConvertKit’s CloudFront bills. It may be time for a quick call to Fastly, CloudFlare, etc. to see if there's a compelling alternative. To sum up, ConvertKit has done a fine job and they deserve a round of applause for the transparency. None of this stuff is easy, and there's an incredible depth of complexity here. I can’t fault them for missing a few things, given how complex this gets. They've built a successful, thriving business. They just have a few things they could do to optimize their bill. And the truth is it wouldn’t even require that much work. If you’ve found yourself with a horrifying AWS bill and confused about fixing it, let’s chat. We see dozens of these a year and we’re happy to help. You can learn more about our consulting here. #### How Storytelling Can Shape Your Cloud Cost Management Strategy Your AWS bill tells a story — about the decisions your business has made and the cost management strategy you’ve implemented. Whether the decisions and strategy were intentional is another matter entirely.  When organizations don’t communicate across teams, share their business context, and work toward common goals, their AWS bill shows it. Third-party tools can further complicate cost management recommendations by missing critical context, like suggesting shutting down a cluster with low resource utilization … that’s also your disaster recovery site.  Fortunately, you can change the narrative of cloud cost management through storytelling. Yes, storytelling. At The Duckbill Group, we’ve found that implementing these four elements of storytelling can help you effectively share the context for your business initiatives and engineering goals. This kind of audience-driven communication can put your organization back on the path toward an intentional cloud cost management strategy. 1. Understand your audience Before you can tell a good story, you have to know who’s going to be listening to it. Do you share a common vocabulary with them? Are they familiar with the same concepts you are? What will you need to explain to create shared understanding? You might already be doing this without realizing it; communicating across different teams in your organization requires understanding your audience. Leadership presents high-level organizational ideas to individual contributors at company all-hands. Engineers describe technical architecture to product owners who’ve never touched a line of code. Engineering teams with minimal knowledge of each other’s architecture discuss how their workloads can work better together.  But is everyone on the same page in those conversations? We each operate with a mental model of the way our world works, but we rarely communicate those mental models to others. John Allspaw, founder at Adaptive Capacity Labs, describes the work that happens in our coworkers’ heads as above-the-line versus below-the-line thinking. Tom Limoncelli, SRE manager at StackOverflow, talks about the tendency to hoard information as a high-context versus low-context culture. For cloud cost management to work, every team needs to assume a low-context culture and establish a shared mental model of how its actions affect other parts of the organization.  For example, Finance might want better visibility into Engineering’s AWS spend to manage the company budget more effectively. But Engineering might want Finance to understand that AWS is a completely different billing model than data centers, requires different processes, and needs atypical ways of thinking about cost. Here, both sides need to communicate their respective knowledge of financial requirements and AWS billing to reach a shared understanding of how best to manage and forecast AWS spend.  As the storyteller, it’s your job to illuminate your mental model, ensure a common vocabulary, and facilitate your audience’s ability to share their mental model with you. Only then can you tell a story about your business context that will resonate with your audience. 2. Determine your characters' goals In every story, each character wants something. Sometimes two characters want the same thing, and we end up with a protagonist and sidekick moving through the world side-by-side. Sometimes two characters want different things, and we end up with a protagonist and antagonist at odds with each other.  Other teams in your organization may not just be your audience; they may also be your characters. It’s up to you to find out what motivates them and how their goals align with yours. So what do the characters in your organization want? Ask them! Finance might want to measure AWS spend to better forecast its cost to the company. Leadership might want to keep costs lower than revenue to keep the company profitable. Management might want to make sure its teams are meeting their OKRs. Developers might want to solve problems with code. These goals are vastly different but ultimately create a shared purpose: driving the company to succeed.  Make sure everyone in your organization knows they’re working together for a shared purpose. Different departments’ wants may seem disjointed on the surface, so frame them as part of the overarching objective. When Finance, Engineering, Leadership, and Product all understand how to support one another in this larger goal, they’re more likely to help one another and perform well in their own work. Sometimes one character withholds information from another, ultimately causing unnecessary problems for both characters. Poor communication is a common trope in fiction writing — and I’ve seen this play out in nearly every organization I’ve worked with. Make sure you and your company implement policies that encourage clear and regular communication of goals, collaboration to meet every department’s needs, and alignment to ultimately move the organization forward together. 3. Let them be the hero of their own story During the pandemic, I decided to write a Dungeons and Dragons role-playing campaign. (Yes, I’m that kind of nerd.) As I started thinking about my campaign, I got really hung up on making sure players moved through each major plot point.  And then it hit me — I was micromanaging. Rather than metaphorically narrowing the players’ paths, I could give them an obvious goal and let them decide the best way to achieve that goal.  In a lot of cloud cost management work, The Duckbill Group sees Leadership present a solution and expect Engineering to implement it, even though Engineering didn’t have a say in the original cloud cost problem statement or solution selection. This works sometimes but can diminish autonomy, which is a critical component for productive teams. Instead, The Duckbill Group recommends giving Engineering teams a clear problem statement and letting them find the solution themselves. For example, during a client engagement, we discovered two teams who wanted better standards and best practices for shared resources. We brought those teams together to collaborate on a solution, giving both teams agency to create the change they wanted. This collaboration led to better adoption of those standards and best practices long-term since both teams were invested in the solution they had created.  Some guardrails will still be required to make sure security and legal constraints are met, but engineers are more committed to a solution if they have a hand in finding it and the path forward is clear. 4. Convey your story through non-verbal signals as well as words Clear communication is key to good storytelling, but great storytellers go beyond the words they say to how they say them. In December 1999, “Buffy the Vampire Slayer” aired an episode called “Hush” that’s almost entirely devoid of dialogue. When the characters in the episode lose the ability to speak, they’re forced to rely on non-verbal communication like body movements, facial cues, drawings, and written language. In some cases, they actually communicate better through non-verbal communication. Now think about the last time you presented cloud cost information to others. How did you present yourself? Did you make eye contact with the camera over Zoom? What was your body language and posture like? What did you use to convey your meaning? What was your tone?  Your audience will pick up on non-verbal cues and tune in or tune out accordingly, so make sure you engage the audience with both verbal and non-verbal communication. There are lots of best practices out there for non-verbal cues, but Ain Aissa’s blog Seek to Speak has some of my favorite tips (with handy visual guides): Emote during your presentation. If you’re excited about the story you’re telling, make sure that excitement shows on your face. Creating an emotional connection with your audience increases their attention to the topic.Emphasize interesting points with vocal variety. Varying the tone and pitch of your story helps the audience understand the highs and lows of your story. It can also indicate which points are key takeaways.Use body language to signal that you’re relatable yet authoritative. Crossing your arms or putting your hands on your hips signals you don’t want to have a conversation, which means your audience is less likely to engage in the story you’re telling. Instead, try standing up straight with your arms at your side — this signals you’re comfortable and open to discussion. Your AWS bill is the sum of your business decisions We at The Duckbill Group like to say “your AWS bill is the sum of all your business decisions.” If those business decisions were made without shared business context and common goals, they may have led to the architecture problems you face today. The way to get your AWS bill and cost management strategy back on track is to clearly communicate the business context from different departments like Finance, Engineering, Leadership, and Product.  Storytelling tools can help you shape the narrative of business context in a way that lets teams understand everyone’s part in your company’s cloud cost management journey. The more frequently teams clearly communicate their cloud cost management knowledge and goals in ways that speak to their audience, the more resilient their solutions become, and the faster cloud cost management best practices scale to the entire organization. Not sure where to begin on your cloud cost management journey? Check out our cost allocation series or contact us for actionable recommendations you can start implementing today. #### How to Properly Engage with AWS Enterprise Support The AWS support page lists four support plans (as of this writing) with corresponding coverage that ranges between "Good luck," “Developer,” “We’ll answer the phone if you call us,” and "We will take care of you with a vengeance." The cost for each support plan naturally reflects the promises, with monthly costs that start at "$0," "$29," "$100," and "$15,000" respectively. Let's focus on the expensive one: Enterprise Support. At $180,000 a year, you might think that the enterprise support program is a fantastic waste of money. You would be correct—unless you use the support it buys you properly, at which point it becomes a jewel beyond price. What Enterprise Support Buys You The key differentiators of this support level are: Access to engineers with the word "senior" at the start of their titlesA target response time of less than 15 minutes to a business-critical system down incident A consultative review and guidance based upon your applicationsIncluded Infrastructure Event Management (think big launches, Black Friday, etc.)Access to a "Well-Architected Review"Access to online self-paced labsA concierge support teamA designated Technical Account Manager (TAM) Note that these are just the additional benefits that are beyond those of the next most expensive tier. For an in depth exploration of everything offered consult the support page directly. Note also that unlike every other paid support tier, Enterprise Support applies to every linked AWS account. Every other support tier applies per AWS account. Ergo, if you have 500 AWS accounts in your organization, you'd very possibly be facing a monthly support bill for “premium” (but not Enterprise) support that starts at $50,000 a month. In other words, for some organizational configurations, you save money by switching to Enterprise Support. Common mistakes in working with Enterprise Support If your company does sign up for enterprise support, it makes sense to make good use of the financial investment. However, I have seen a number of ways that companies engage with Enterprise Support that result in poor outcomes. DON’T believe that Enterprise Support replaces staff training. You may well find yourself asking AWS Support relatively basic questions. This isn't a bad thing; that's what they're there for. That said, if your primary mode of engagement with Enterprise Support is to ask basic "How do I use this service" questions, your money may be better spent on hiring internal staff who are already up to speed on AWS concepts. Despite their best efforts, AWS cannot take ownership of your applications the way that internal staff can – and should. DON’T treat the Enterprise Support staff as though they’re the enemy. Because they’re not. I have seen people draw sharp lines between "us" and "them." These start at the level of communications failures, and continue into the realm of failures of empathy. Yes, I understand that you want to keep company information proprietary. You’re loathe to explain that the technical problem you’re trying to solve is to sort Twitter for Pets’s dog-tweets by breed at scale. But you have to find an acceptable way to share what you are trying to do, or the professionals are limited in their ability to help you. If you refuse to tell your account team what you're working on, you cannot reasonably be upset if it turns out that AWS releases a service that solves your exact problem. AWS builds a great number of services that solve common problems. I constantly marvel at how often I see an architectural problem in one client's environment that mirrors what's going on in another client's environment. If you're trying to move data from one place to another, or working around a frustrating limitation in AWS's offering (or your perception of such a limitation), start by talking with your account team. Very often the answer takes the form of, "Wait a few weeks and this will go away." DON’T be rude to the Enterprise Support team. f your first response whenever there's an AWS service incident is to berate your TAM, not only are you wasting your money, but you're being a jerk. TAMs are customer-facing support personnel. They have no ability to cause service outages (well, not without a tremendous amount of creativity!), fix service outages, or demand customer-specific answers from the engineers who are doing their best to respond to the outage. By yelling at them, you're becoming the business version of someone who calls Dell to scream at the poor phone support agent about your laptop not working properly. You absolutely have a right to be upset when a service you're paying for fails; however, it isn't a productive use of anyone's time to take out that frustration on the people who are themselves often as much in the dark as you are. DO treat Enterprise Support as more than a supercharged ticket tracking service. One common pattern is for engineers to treat Enterprise Support as if the support team’s sole job is to track the AWS tickets you opened. While tickets are indeed important (they're sadly one of the only ways that large organizations communicate internally between various units), this is the smallest piece of what Enterprise Support offers a business. If you view support only in defect-tracking terms, it would be wiser and cheaper for you to hire several entry-level people to fulfill this function. DO expect expertise. Don’t pay for Enterprise Support so you can sit in planning meetings with AWS, then use that time to imply that their engineers are morons. This is incredibly toxic behavior, unfair to the people you're speaking down to, and a hilarious misunderstanding of reality. If you do so, you are wasting your money; dead wrong; and I think I used to work at your company. I've found my fair share of AWS annoyances over the past decade. Invariably I eventually spoke with an AWS engineer about the problem who understood the problem far better than I did. The engineer also had an accessible explanation as to why things were the way that they were, and the complexities involved in overcoming it. (Of course, they were actively working to address it.) DO listen to their advice. You don’t have to follow it. You may find yourself sitting with an AWS architecture team discussing your environment, wherein they make suggestions about what you should do. You may well decide not to implement what they suggest. This is ambiguous. It could be that you're wasting your money by ignoring their expertise, or it may be that your specific business constraints make a "best practice" inapplicable to your environment. You know your shop better than they do, after all. It's hard to say without further context whether you're wasting money in this situation. Please note that I'm not suggesting you blindly follow AWS's suggestions without consideration! Ultimately, their advice is advice, not binding rules you must follow; I simply advise that you take it seriously instead of rejecting out-of-hand. DO take advantage of their knowledge. One key differentiator of AWS Enterprise Support is that it grants you access to a host of architectural review services. By all means, use them! This **lends itself to several positive outcomes. First, it's a certainty that the expert team with whom you consult has seen many more environments than you. As a result, they see patterns invisible to you, which makes them well positioned to opine upon what works (or doesn't work) in particular scenarios. They've got the kind of experience that most external engineers take a decade to acquire. As a part of this process, you absolutely should be as transparent about your future plans as you can be. It's entirely common that you'll mention a problem that AWS engineers have encountered a lot—and they may know about a new, not-yet-announced service to which you can get beta access! You'll be a lot happier than if you built your own solution and deployed it a week before the service is publicly announced. I've been on the wrong end of that story several times myself; it's not fun, and you kick yourself for all of the wasted effort you poured into something that suddenly isn't a problem anymore. DO understand that you’ll need to reset the relationship periodically. There are a lot of benefits to Enterprise Support, but they only exist if you and your team know they exist. To that end, it's important to recognize how the relationship changes with AWS ages. You hire new staff; they rotate staff on their end; and suddenly you have a team of developers who's never been introduced formally to enterprise support. Refresh and renew that relationship regularly, and especially do so whenever there are staffing changes to the team on either side. In summary, pay for Enterprise Support and use it properly, or save your money. #### Lambda Logs Just Got a Whole Lot Cheaper* On May 1st, AWS announced a significant change to how Lambda function logs are priced in CloudWatch. Instead of the previous flat rate model where you paid $0.50 per GB regardless of volume, they’ve introduced a tiered pricing structure so you save the more you use it. One downside is that any custom-written CloudWatch Log reports will need to be updated, which is great for me since I have written Duckbill’s comprehensive CloudWatch Logs analyzer—ugh.  To state the obvious, this is a welcome change that I think has been long overdue. Many of the large enterprises I’ve worked with are running Lambda at scale and have struggled to balance comprehensive logging (for troubleshooting and visibility) or reducing that logging to control costs. This changes that equation a bit, so let’s take an in-depth look at how this change will look for you based on your usage patterns. What’s changed exactly? This blog post from AWS breaks it down quite a bit more, but essentially there are two major changes to CloudWatch Logs for Lambda:  Tiered pricing for CloudWatch Logs: We used to get a flat rate for logging, now it's tiered based on usage. This is great news for those heavily logging out of Lambda, and no real change for those that are not. New logging destinations: In addition to CloudWatch Logs, you can now send Lambda logs directly to S3 or Firehose. Sounds like a great way to bypass the CloudWatch Logs tax, but alas it is not free. I want to stress this is ONLY for logs created by a Lambda function and not for application logs generated outside of those lambda functions.  Because the first tier is priced the same as the old CloudWatch Logs flat rate ($0.50 per GB for Standard Logs in us-east-1), then there really is no downside for customers for this change. If you are a heavy user of it and were to generate 60TB of Lambda logs monthly in a given account, your costs would drop from $30,000 to $12,500—a 58% reduction! Cheaper* There are a couple of ways this change will not meaningfully change your AWS bill. First, if you are using less than 10TB per month of Lambda logs in an account, you will in effect see no change. I also highlighted “in an account” for a reason, this new pricing change is bound within an AWS account, so if you are a large enterprise with multiple AWS sub-accounts, you can’t count usage across all of those sub-accounts.  As an example, if Account A is logging 8TB to CloudWatch Logs, and Account B is logging 2TB to CloudWatch Logs, both accounts will be charged the first tier rate of $0.50 per GB (in us-east-1). While it may seem like a good idea to co-locate as many Lambda functions as you can in an AWS account to take advantage of this change, there are many other factors you would need to consider (such as Lambda usage limits, security posture, etc). New logging destinations I’m a bit conflicted on the new logging destinations for Lambda functions. On the one hand, I’m excited that S3 and Firehose are now supported. The ability to send logs directly to these destinations eliminates the need for complex Lambda-based forwarders built to export logs from CloudWatch to other systems—certainly a cost saving and operational burden removed.  The new log destinations do unlock several use cases: Long-term compliance archiving: S3 with lifecycle policies is far more cost-effective for retaining logs that might be needed for regulatory requirements Advanced analytics: Sending logs directly to S3 makes it easier to run Athena queries or use AWS Glue for log analysis at scale Third-party observability integration: Firehose can stream logs directly to Datadog, Splunk, New Relic, or Sumo Logic without intermediary Lambda functions Custom processing pipelines: For organizations with specific log enrichment or filtering needs, direct Firehose delivery provides a more reliable starting point I just wish the pricing was less than sending to CloudWatch Logs directly, as it may encourage teams to fully integrate Lambda logs into their observability platforms. Hopefully the pricing for this will change over time if enough customers ask for it. What took so long? Many other AWS services (Amazon Cognito logs, Amazon MSK broker logs, etc) have used this pricing tier for a while, which they refer to as Vended logs. You can see a list of them here, but the same tiered pricing applies to all of these Vended logs. I suspect that Lambda logs were initially excluded because they are almost entirely customer-generated content, where you have control over what and how much is logged. You can still put just about anything into a CloudWatch Log group, but you still have to pay to get it back out using something like Live Tail, OpenSearch, or CloudWatch Logs Insights, so it doesn’t necessarily hurt AWS to have you pushing lots of logs if they can charge you. How much will I save in May after this change? As I was exploring this change, it dawned on me that we could use some AWS API calls to figure out how much your bill will change from April (before the price change) to May. I present to you, CloudWatchCashBack—my quick Python tool to estimate how much you will save in May based on your April usage of CloudWatch Logs!  Just follow the README file and you can generate a report that will look something like this: CloudWatch Logs Cost Analysis =========================== Region: us-east-1 Total GB Ingested: 42,095.08 GB   - Standard Storage: 42,095.08 GB   - Infrequent Access: 0.00 GB Old Pricing Total Cost: $21,047.54 New Pricing Total Cost: $11,209.51 Cost Difference: $-9,838.03 Percentage Change: -46.74% Daily Breakdown: --------------- 2025-04-01: Old: $629.28, New: $359.19, Diff: $-270.09, Usage: 1,258.56 GB (Standard: 1,258.56 GB, IA: 0.00 GB) 2025-04-02: Old: $592.31, New: $351.80, Diff: $-240.52, Usage: 1,184.63 GB (Standard: 1,184.63 GB, IA: 0.00 GB) 2025-04-03: Old: $677.47, New: $368.83, Diff: $-308.64, Usage: 1,354.94 GB (Standard: 1,354.94 GB, IA: 0.00 GB) 2025-04-04: Old: $619.86, New: $357.30, Diff: $-262.55, Usage: 1,239.71 GB (Standard: 1,239.71 GB, IA: 0.00 GB) 2025-04-05: Old: $841.35, New: $400.80, Diff: $-440.54, Usage: 1,682.69 GB (Standard: 1,682.69 GB, IA: 0.00 GB) 2025-04-06: Old: $685.76, New: $370.48, Diff: $-315.27, Usage: 1,371.51 GB (Standard: 1,371.51 GB, IA: 0.00 GB) 2025-04-07: Old: $721.96, New: $377.73, Diff: $-344.24, Usage: 1,443.93 GB (Standard: 1,443.93 GB, IA: 0.00 GB) 2025-04-08: Old: $620.83, New: $357.50, Diff: $-263.33, Usage: 1,241.66 GB (Standard: 1,241.66 GB, IA: 0.00 GB) 2025-04-09: Old: $621.21, New: $357.57, Diff: $-263.63, Usage: 1,242.41 GB (Standard: 1,242.41 GB, IA: 0.00 GB) 2025-04-10: Old: $651.58, New: $363.65, Diff: $-287.93, Usage: 1,303.16 GB (Standard: 1,303.16 GB, IA: 0.00 GB) 2025-04-11: Old: $658.98, New: $365.13, Diff: $-293.85, Usage: 1,317.96 GB (Standard: 1,317.96 GB, IA: 0.00 GB) 2025-04-12: Old: $553.01, New: $343.94, Diff: $-209.08, Usage: 1,106.03 GB (Standard: 1,106.03 GB, IA: 0.00 GB) 2025-04-13: Old: $589.36, New: $351.20, Diff: $-238.15, Usage: 1,178.71 GB (Standard: 1,178.71 GB, IA: 0.00 GB) 2025-04-14: Old: $616.53, New: $356.64, Diff: $-259.89, Usage: 1,233.05 GB (Standard: 1,233.05 GB, IA: 0.00 GB) 2025-04-15: Old: $575.86, New: $348.50, Diff: $-227.35, Usage: 1,151.71 GB (Standard: 1,151.71 GB, IA: 0.00 GB) 2025-04-16: Old: $611.29, New: $355.59, Diff: $-255.70, Usage: 1,222.59 GB (Standard: 1,222.59 GB, IA: 0.00 GB) 2025-04-17: Old: $717.70, New: $376.87, Diff: $-340.83, Usage: 1,435.40 GB (Standard: 1,435.40 GB, IA: 0.00 GB) 2025-04-18: Old: $714.00, New: $376.13, Diff: $-337.86, Usage: 1,428.00 GB (Standard: 1,428.00 GB, IA: 0.00 GB) 2025-04-19: Old: $610.92, New: $355.52, Diff: $-255.40, Usage: 1,221.84 GB (Standard: 1,221.84 GB, IA: 0.00 GB) 2025-04-20: Old: $687.97, New: $370.93, Diff: $-317.04, Usage: 1,375.93 GB (Standard: 1,375.93 GB, IA: 0.00 GB) 2025-04-21: Old: $833.33, New: $400.00, Diff: $-433.33, Usage: 1,666.65 GB (Standard: 1,666.65 GB, IA: 0.00 GB) 2025-04-22: Old: $767.34, New: $386.80, Diff: $-380.54, Usage: 1,534.68 GB (Standard: 1,534.68 GB, IA: 0.00 GB) 2025-04-23: Old: $767.75, New: $386.88, Diff: $-380.87, Usage: 1,535.50 GB (Standard: 1,535.50 GB, IA: 0.00 GB) 2025-04-24: Old: $790.52, New: $391.44, Diff: $-399.09, Usage: 1,581.05 GB (Standard: 1,581.05 GB, IA: 0.00 GB) 2025-04-25: Old: $752.29, New: $383.79, Diff: $-368.50, Usage: 1,504.58 GB (Standard: 1,504.58 GB, IA: 0.00 GB) 2025-04-26: Old: $1,015.47, New: $418.21, Diff: $-597.26, Usage: 2,030.95 GB (Standard: 2,030.95 GB, IA: 0.00 GB) 2025-04-27: Old: $854.63, New: $402.13, Diff: $-452.50, Usage: 1,709.26 GB (Standard: 1,709.26 GB, IA: 0.00 GB) 2025-04-28: Old: $785.81, New: $390.50, Diff: $-395.32, Usage: 1,571.63 GB (Standard: 1,571.63 GB, IA: 0.00 GB) 2025-04-29: Old: $720.58, New: $377.45, Diff: $-343.13, Usage: 1,441.16 GB (Standard: 1,441.16 GB, IA: 0.00 GB) 2025-04-30: Old: $762.59, New: $385.85, Diff: $-376.73, Usage: 1,525.17 GB (Standard: 1,525.17 GB, IA: 0.00 GB) What does this mean for you? For everyone running Lambda functions at scale, there’s really nothing to do but enjoy the savings, but there are a few things you should always look at in regards to CloudWatch Logs: Review your log levels: Now might be a good time to review what you are logging and consider dropping back to just warning log level to save some money. Evaluate the new destinations: If you have an existing CloudWatch to S3 forwarder, this direct to S3 integration may be a better option Adjust budget forecasts: Since this could have a material impact for heavy lambda users, look at your budget for the rest of this year and make adjustments accordingly The bottom line is that this is a customer-friendly move from AWS that directly addresses concerns with running Lambda at scale. The only thing I wish AWS would do is combine sub-accounts usage so customers won’t try and jam more Lambda functions into a single AWS account just to save logging costs, but otherwise this should be seen as AWS listening to customer feedback about their pain points regarding CloudWatch Logs.  Did you have problems running our Python code? If so, feel free to create a Github issue, let us know at hello@duckbillgroup.com or hit us up on Bluesky! Are you confused as to how your CloudWatch Logs bill got so large? The Duckbill Group specializes in helping companies reduce their AWS bills. If you're concerned about maximizing your discounts across AWS services, reach out to learn how we can help. We’ll be discussing this topic Thursday at 10:00am PDT over on Duckbill’s weekly drop-in clinic. Join us. #### Overhauling AWS account access with Terraform, Granted, and GitOps As part of our engagements with clients, we need to access their AWS accounts to see what’s what. When we first started our work in 2019, we followed AWS’s recommendation of using role assumption, which worked great, but we had a ludicrously broad access scope on the policy attached: the AWS-managed ReadOnly policy plus a few specific grants that weren’t in the AWS-managed policy. We later scoped that down to the AWS-managed ViewOnly policy after some customer feedback about the scope, which worked just fine for us. From time to time, some of our more security-conscious clients would scope our access down even more. I’ve always thought their changes were great and often implemented them into our standard setup. Lately, that got me thinking about how we could redo our access permissions entirely so that we had the absolute least access needed to do the job. Internally, we started discussing a new problem we were facing: as we brought in more specialized contractors for one-off consultations on our client accounts, we needed a way to grant a person access only to the client accounts they were working on. Our existing setup worked well for our staff, who had access to all clients, but it didn’t really support limiting access on a per-client basis. We eventually cobbled together an MVP that required the manual creation of roles and policies, but the checklist to set up a new contractor was two pages long! Definitely not a good solution. To make matters even more complicated, we had recently set up AWS SSO, but we were still using the built-in AWS identity store. And, on top of that, we found ourselves having to pass AWS CLI config files back and forth constantly. Our method of managing access to client accounts really left a lot to be desired. A new vision for the future We ultimately decided to take a step back and really think through exactly what we wanted of a new approach. We came to these key needs: Employee access and one-time contractor access should be managed the same way Authentication and authorization to AWS should be governed by a single source: Duckbill’s Google user/group directory. Use Granted.dev for assuming roles, and have a programmatic generation of the Granted config. No config files should ever be passed around again. Read client access details (account ID, external ID, internal name) from a central database. These aren’t secrets and shouldn’t be treated as such. Granting/revoking access should be easily done and traceable/auditable. A user should only be able to access clients for which they have been explicitly granted access. One look at this list and we quickly realized we were going to need some expert help. An aside about confidentiality One key thing we realized we had to decide on was the level of confidentiality of the data involved. We’ve always treated client data as the crown jewels, but different people on the team had different ideas about what constituted client data and what didn’t. We finally decided to properly define our levels of confidentiality, resulting in three levels: Public, Confidential, and Client Data. Public is exactly as the name implies: the information is public information. This mainly applies to documents that contain content for a blog after it’s been published. For example, this document will be Confidential during editing and Public after it’s approved for release. Confidential is a normal level of confidentiality. We don’t take steps to hide it from team members who aren’t working on clients, but we do expect it to be treated as non-public, not-to-be-shared information. For example, Metadata about a client’s relationship with Duckbill is a normal level of confidentiality. Metadata includes bits of information like account IDs, external IDs, the name of the client, and so on.  “Client Data” is a distinct category of confidentiality, requiring need-to-know. For example, the details of a client’s AWS spend is considered Client Data.  The way we talk about it in our client security packet is basically: “The existence of a meeting and the attendees is Confidential, while the content of the meeting is Client Data.” Making this decision was a key issue for us to implement a solution here. Treating even the name of a client as Client Data would mean obfuscating everything, which could lead to a scenario where we’re accidentally granting the wrong access or assuming the wrong role–and all for very questionable upside. The potential downside from poor UX outweighed the upside of increasing the confidentiality. Remember, folks, security is always a bunch of tradeoffs. Enter Chris Farris, IAM wrangler extraordinaire We, of course, did the obvious thing and hired Chris Farris to design and implement a solution for us. I’ll let Chris tell the rest of this story! Everything Mike and Corey sought fit well within the best practices I’ve implemented for my employers and clients: centralize your identities, use least-privilege roles, and grant access to humans only as necessary. The solution for Duckbill is three-fold. Part one is a custom Terraform module and pipeline to create the required AWS Identity Center (hereafter referred to as SSO) elements for each client. The second part is a simple way to ensure all the cloud economists are working from the same AWS config file and that the necessary account IDs and external IDs are communicated. Both part one and part were implemented as part of an automated CI/CD pipeline. Part three is right-sizing the permissions to ensure that the security issues of ReadOnlyAccess and limitations of ViewOnlyAccess are addressed.  Part one: Terraform  With AWS SSO, access is granted via the confluence of three elements: 1) an identity and 2) a Permission Set are 3) assigned to an account.  Each Duckbill client config boils down to two input elements: the internal client name and the Duckbill users who should have access to the client. A Lambda function fetches the other critical details from Duckbill’s CRM, such as external_id and account_id. When deployed, the terraform module will: Invoke a Lambda function to look up the name, payer_account_id, and external_id from the CRM based on the provided client_id. Create a new client_access group in AWS SSO. Add the provided users to the client_access group. Create the Identity Center Permission Set. This includes creating the role policy allowing the cloud economist to assume the DuckbillGroupRole if they pass the client’s unique external_id. Assign the Permission Set to the SSO Group in the Duckbill Client Access AWS account.  Generate the Granted config based on the data from the CRM and commit that to the Duckbill Group’s Granted Registry in GitHub.  The AWS Identity Store is a managed user directory that offers a basic active directory feature set. If you read the identity store boto3 docs, all API calls require obscure identifiers. Doing anything via the command line is painful, so leveraging Terraform for all this is much more straightforward than scripting.  The Terraform code boils down to: # Create the Group resource "aws_identitystore_group" "client_access_group" { display_name = var.customer_code description = local.customer_name identity_store_id = var.identity_store_id } # Add members to the group resource "aws_identitystore_group_membership" "members" { identity_store_id = var.identity_store_id count = length(var.users) group_id = aws_identitystore_group.client_access_group.group_id member_id = data.aws_identitystore_user.users[count.index].user_id } # Create the permission set resource "aws_ssoadmin_permission_set" "client_permission_set" { name = var.customer_code description = local.customer_name instance_arn = var.instance_arn relay_state = "https://s3.console.aws.amazon.com/s3/home?region=us-east-1#" session_duration = "PT6H" } # Add inline policy to the permission set resource "aws_ssoadmin_permission_set_inline_policy" "client_assume_role_policy" { inline_policy = data.aws_iam_policy_document.client_assume_role_policy.json instance_arn = var.instance_arn permission_set_arn = aws_ssoadmin_permission_set.client_permission_set.arn } # Assign the Permission Set and Group to the Account resource "aws_ssoadmin_account_assignment" "client_assignment" { depends_on = [aws_identitystore_group.client_access_group] instance_arn = var.instance_arn permission_set_arn = aws_ssoadmin_permission_set.client_permission_set.arn principal_id = aws_identitystore_group.client_access_group.group_id principal_type = "GROUP" target_id = var.duckbill_clientaccess_account_id target_type = "AWS_ACCOUNT" } Managing access But now we’re left with a problem. We only know the payer account_id and not the account_ids of all the client’s non-payer accounts, which we need to run Duckbill’s analysis tooling. Since we don’t want to have to update the AWS SSO Permission Set each time a new client account is discovered, we can limit access in the cloud economist’s identity policy via the external ID like so: { "Action": "sts:AssumeRole", "Condition": { "StringEquals": {"sts:ExternalId": "d61ea58c-foo" } }, "Effect": "Allow", "Resource": "arn:aws:iam::*:role/DuckbillGroupRole" } In the above policy, the cloud economist can assume any DuckbillGroupRole, but only if the cloud economist passes the external_id that matches the client. If they pass a different external_id for another client, their identity policy will not allow the action. If they attempt to assume a role for a different customer but provide the wrong external_id, the client’s trust policy will deny the action. One advantage to this approach is that we don’t need to treat the external ID as a secret. As long as permissions to assume DuckbillGroupRole are locked down in the trusted Duckbill Client Access account, even with the external ID, a cloud economist cannot assume a role into a client account they’re not authorized for.   CodePipeline GitOps is all the rage these days, but delegating these sensitive IAM Permissions outside the AWS account introduces an additional risk factor we wanted to avoid. When pushing files to S3 or deploying a Lambda, you can tightly scope the policies granted to your GitHub action. For our purposes, however, we are delegating the permissions to decide who has permissions, so extending the trust boundary beyond AWS into GitHub isn’t ideal for this scenario. Luckily, AWS has an underrated service that, while not as slick as GitHub Actions, does the job and keeps the scope of trust limited to just the AWS Identity Center account: CodePipeline.  The basic pattern is to create a CodePipeline with four stages. Stage one downloads the source, and stage two calls CodeBuild to run a Terraform plan. At stage three, the pipeline pauses and requires a human to review and approve the plan before executing the final stage: terraform apply.   Part two: Account IDs and external IDs Granted Granted is a tool to simplify accessing AWS accounts in a seamless manner, allowing a cloud economist to be logged into multiple AWS accounts in the same browser window thanks to browser containers. It supports SSO, chained roles, and much more. (Mike: we used to use aws-vault for this same purpose, but Granted is so much easier and feature-rich for our use cases.) Like aws-vault, Granted allows a cloud economist to either access an account in the CLI through setting the right session variables via STS or logging into the account in the browser, with a single CLI command for either. Certainly not something supported via the normal awscli means.  Rather than let each cloud economist roll their own configuration solution, Duckbill is leveraging Granted’s Profile Registries. The custom Terraform module centralizes the creation of config files for each client, and each person has a personalized config file they leverage when setting up the registry. We don’t keep the full set of client profiles in git, but rather just the basic configuration—the Granted profile registry sets up all the profiles dynamically based on what AWS SSO grants the cloud economist access to. Google Workspace and AWS Identity Center The Duckbill Group’s business is two-fold: cloud finance consulting and the media properties, each with separate staff. While the above solution is focused on the security of their consulting clients, the media side also requires certain staff and contractors to access AWS for specific uses that aren’t client related. The “traditional” method manages these by assigning users to groups inside the Google Workspace console. We needed to enable SCIM provisioning from Google Workspace to AWS SSO to provide this capability.   Sadly, AWS’s integration with Google’s identity store left something to be desired. While it was reasonably straightforward to configure the AWS SSO redirect to Google for authentication, creating users and groups in AWS SSO requires using the ssosync Lambda function from the AWS Labs GitHub.  Halfway through the project, AWS & Google released official SCIM support. Unfortunately, this turned out to be a half-baked integration. The Google-managed SCIM provisioning doesn’t support Google Workspace Groups! When SCIM is enabled, AWS console management of users and groups is mostly disabled. Under this new method, you can’t manage groups in Google or AWS unless you use the convoluted AWS IdentityStore APIs. That would be one hell of a yak shave, so hopefully AWS finishes their SCIM support in the future. Part three: Properly scoping Duckbill’s permissions Lastly, we needed to scope down the permissions of the role that gets deployed on the client side. Here, they were at the mercy of the AWS teams that manage the pre-canned AWS policies. ReadOnlyAccess is clearly over-permissive, and customers were right to ask for a more limited set of permissions. However, the AWS recommended alternative, the ViewOnlyAccess policy, is likewise insufficient.  Of the 261 AWS Services referenced in ReadOnlyAccess, only 150 are referenced in ViewOnlyAccess. There are over 110 services in ReadOnlyAccess that ViewOnlyAccess does not cover, and ReadOnlyAccess does not cover nine services present in ViewOnlyAccess.  As a third-party auditing a client, you need to see everything about the account and its resources but not read any data inside of those resources. For example, knowing how many objects are in an S3 bucket, their age, their access patterns, and their storage tier are important for analysis purposes, but we need to make sure the contents of the object can’t be viewed by Duckbill. Unfortunately, AWS provides no distinction between data and metadata in their Get, List, and Describe API calls, so we had to find them. Rather than enumerate the IAM Actions a cloud economist would need—a likely never-ending task—I decided to identify which IAM Actions provide access to customer data and credentials and then explicitly deny those. As part of this, I created the Sensitive IAM Actions collection to provide a source for the cloud security community to define which actions provide access to data, expose credentials, or permit privilege escalation. Ian McKay, Kinnard McQuade, and Scott Piper had done a lot of work to identify permissions that led to privilege escalation, credential, and resource exposure, so I built off their work to generate a list of the permissions that allowed access to data.  Some IAM Actions are in a gray area: lambda:GetFunction is required to show a function's runtime, memory, and duration, but that call also returns a pre-signed URL with access to the code zip file. The first three are critical for a cloud economist, but the code zip could be considered client-sensitive data. Denying access to that Action meant not being able to advise on Lambda, which would mean not being able to optimize Lambda, so we kept that action. We’ve also carved out an exception in S3: A cloud economist needs access to the billing CUR reports, which are just S3 objects and an exception to the rule above. By customizing the policies with an Effect=Deny on a NotResource of the CUR bucket, the DuckbillGroupRole could access only the required data. (For my fellow security nerds, this pattern would work quite well for CloudTrail event logs, too!) What’s next We’re pretty thrilled with our new setup and certainly have a lot more confidence in our security around client access now. That said, there’s one big thing we’d like to do for the next iteration: auto-discovery and configuration of non-payer accounts. Our new setup only configures payer accounts, which handles 80% of what we need, but we do still need non-payer accounts for our automated tooling. We’re not sure yet exactly how we want to solve this, so for now, it’s a manual configuration. Last but certainly not least, a huge thanks to Chris Farris for helping us sort out this mess. We had some pretty complicated requirements but I don’t think we stumped Chris even once. If you’re looking for help on AWS security, Chris is great. Check out his services at https://primeharbor.com. #### POST No AWS Bills: Cloud Cost Optimization Without APIs https://www.youtube.com/watch?v=IWyRBvKAIM0 #### Protecting Our Clients: Incentive Alignment When my cofounder Corey Quinn and I started The Duckbill Group, one of the key questions we worked through was this: "How can we be certain we're serving the client's best interests at all times?" If you've ever worked in a decent-sized company, you know firsthand that this isn't always so obvious and clear-cut. Many teams within larger companies have misaligned incentives from their customers. For example, a call center that is measured on call time may push to end calls faster at the expense of the customer's satisfaction. Even minor misalignments can lead to serious problems, and we were cautious about that. Given that we were dealing with our clients' money, we never wanted our loyalties to be in question. Instead, we wanted our clients to have absolutely no doubt that when we recommended something, it was because it was in their best interests—and not because we were getting a kickback behind the scenes or anything. I Made the Client Happy and Nearly Got Fired for It Some years ago, I was working at a technical consulting firm and was assigned to a well-known company as a subject matter expert. The client had hired my employer to do a complete overhaul of how they monitored their global infrastructure. A short while into the project, the client had a major shift in business priorities—to the point that they no longer needed a solution to the problem I was there to solve. Still, they were happy to keep me around to maintain what they had.  Knowing how much more expensive I was versus what they now needed, I wrote up a new job spec for a much more junior (and less expensive!) person that could meet their needs and assured them my firm would be able to provide such a person in my place. The client was thrilled. My employer was livid. While I left the client in good hands, my reputation inside my employer took a significant hit and I lost the trust of the executives. I resigned shortly thereafter. Despite serving the client's best interest, I did so at the expense of my employer's (short-term!) best interests. My client's interests and my employer's interests were misaligned. In making the decision to replace myself, I was serving the client's interests. In practice, had my employer not been so short-sighted, they might have seen the goodwill generated from my action was worth far more than the billable hours they lost. This event formed much of how I think about consulting and client relationships. Unquestionable Incentives When you have a client, you have a duty to protect their interests. In fact, it's in the definition! Protecting the client, to us at Duckbill, is about more than giving good advice, being proactive, and being generally good at what we do. Sure, we do all of those things. But for us to protect our clients, our clients also need to trust us. Which brings us back to the early days of starting Duckbill: How do we ensure we are always protecting the client to the best of our abilities?  We decided five things right away, which form central tenets of Duckbill today. 1. We are not AWS partners or partners with anyone in the space. Once you're in a partner program, your loyalties become divided between the client and the partner. As we want our incentives to be unquestionable, we knew that we could never partner with anyone or any company. Our independence ensures alignment. 2. We are not venture-backed. Taking venture capital means committing to hypergrowth—which means capturing as much value from a relationship as possible and lowering our cost structure to near-nothing. In such a situation, the client will inevitably be harmed. Instead, we stay small, grow sustainably, and hire experts. 3. We charge for our services on a fixed-fee basis only. While many solutions similar to ours in the market charged a percent of savings, we realized that we'd inevitably end up in a debate with the client over how realistic our identified savings were or about excluding some savings opportunities because the client's team already knew about them. Charging fixed-fee skips all of that. But it’s also important to us that we charge a fair price for our services. After all, we provide incredible results. Every single customer of Duckbill has realized savings of at least a 550% ROI. We’ve had some as high as 73,300%. No, I’m not exaggerating. 4. We have a strong guarantee on our services. If a client isn't happy with our work and we can't make it right, we simply refund the entire engagement fee. When you remove financial incentives from a conflict, the conflict has a way of resolving itself rather smoothly. (To date, we've never been in this position!) 5. We do not provide implementation of our findings. Having seen so many consulting firms provide an assessment where the results just so happen to recommend hiring their firm for a larger, more expensive contract... that didn't sit well with us. By not providing services to implement our findings, we're not incentivized to paint a more/less dire picture than reality calls for, and there's no reason to suspect our findings are a ploy for a larger-but-unnecessary project. Incentives Matter A story from the book Freakonomics has always stuck with me. Paraphrasing a discussion on the misaligned incentives with residential real estate brokers and homeowners, the authors noted that if a house were to sell for $300,000 and the broker receives 6%, the broker makes $18,000. Seems like a match made in heaven, incentive-wise: You've got a house to sell, and the broker only makes money if they sell it. However, if the house were to take an extra week to sell but at a higher price of $310,000, that extra week is worthwhile for the home owners—but not for the broker, who only makes an extra $600. Looking at this story, it’s clear that the initial perspective of incentives is at odds with reality. As it turns out, the broker is incentivized for a quick sale while the homeowners want a high sales price.  In this way, the real estate broker has an opportunity to put their own interests ahead of their client's. While I'm sure most brokers would never do such a thing, even the opportunity for it to happen raises enough doubt in the client's mind about what the incentives truly are. Incentives matter. When incentives are aligned, clients are protected. Interested in getting your AWS bill under control? Want to negotiate a better contract with AWS? We can help. Check out our services and give us a call. #### RDS Reserved Instances: Where Did All the New Instance Types Go? The Case of the Missing RDS Instance Types Every so often I get an AWS question that makes me roll up my sleeves and dive into some code to get the answers. This week’s question was posed by Justin Brodley from The Cloud Pod in the Last Week in AWS Slack channel: why couldn’t they purchase a reserved instance (RI) for a R8g Amazon RDS cluster? My first thought was that it might be just a bug or some weird anomaly for us-east-2 (where they found the issue), but I noticed it also was not available in other regions. Now I really had to know if this was widespread or just some weird quirk.  We recently found that EC2 Reserved Instances are being quietly deprecated. During that investigation, I wrote a small Python program to find all of the non-RI eligible EC2 instances in the AWS pricing file. Now, I’ve adapted it to find all RDS instance types that don’t support RI’s. Lo and behold, those same instance types showed up! Let's dive into what is going on here. Non-RI Amazon RDS Instance Types Looking at the full list of instance types unavailable for RDS RIs, we can put them into two distinct buckets: Legacy instance types that have been retired. This includes: M-series: m1, m2, m3, m4 R-series: r3, r4 T-series: t1, t2 Newer generation instance types including: m7i family—released October 2024 m8g family—released November 2024 r7i family—released October 2024 r8g family—released November 2024 c6gd family—launched for RDS in March 2024, specifically for Multi-AZ deployments AWS’s excluding retired instance families from RI purchases makes perfect sense. Why would they want you to purchase RI’s for end-of-life platforms?But looking at the newer instance types, these represent some of the most recent and powerful options in the RDS lineup. The absence of RI’s for these newer generations leads us to an important question: Is AWS simply behind on updating their RI offerings, or is something bigger going on here? AWS is forcing customers to either use the older generation instances that qualify for significant savings via RI discounts, or adopt the latest instance types and be forced to pay on-demand rates. There’s no middle ground here, and if this is the new norm, customers are going to start making crappy architectural choices to get around this. Two Theories (And Neither Is Particularly Reassuring) Theory 1: AWS Just Forgot The simplest answer is that AWS simply hasn’t gotten around to adding these instance types to the RI purchase options yet. Perhaps it’s an oversight, a SIM ticket languishing in someone’s backlog, or a product manager who didn’t realize that customers might want to save money on their newest, most powerful instance types. If these instance types had just been released, I would be more inclined to this hypothesis—just wait a little, as I’m sure they will add it soon…—but all of these instance types were added over 5 months ago at this point. If it truly is just an oversight, I’d sure hope AWS can quickly fix this for customers who surely are asking for it. Theory 2: The Great Reserved Instance Purge Continues The more intriguing possibility is that this is a deliberate extension of the RI deprecation strategy we’re seeing in EC2. Perhaps AWS is starting to phase out RIs across all services, beginning by simply not adding new instance types to the RI program. If this is true, it creates a real problem for customers utilizing RDS. Going without Savings Plans or RIs will force customers to pay full on-demand pricing for their most modern database instance types. This is sure to make customers evaluate whether they should use these newer, often more efficient instance types rather than sticking with instances that can still be reserved—presumably the opposite of what AWS should want! The RDS Savings Plan Dream If AWS is indeed phasing out RIs for RDS, then the logical next step would be to introduce RDS into the Savings Plans framework. Corey has been harping on this for years, and I agree that it is sorely needed. Implementing this would greatly simplify commitment management for customers and allow for much greater flexibility for RDS databases. A Database Savings Plan—or better yet, folding RDS into the existing Compute Savings Plans—would be a win-win for AWS and its customers. What now? Hopefully the RDS team will quickly tell us what the plan is for these newer instance types moving forward. If the lack of RI options for newer instance types is affecting your planning, reach out to your friendly AWS TAM to let them know. I have seen first-hand that customer feedback really does influence product decisions—so don’t be afraid to tell them! Have you noticed other AWS services with missing discount options? Or have you heard anything from AWS about RDS joining Savings Plans? Let us know at hello@duckbillgroup.com or hit us up on Bluesky! The Duckbill Group helps companies reduce their AWS bills. If you're concerned about maximizing your discounts across AWS services, reach out to learn how we can help. We’ll be discussing this topic today at 10:00am PDT over on Duckbill’s weekly drop-in clinic. Join us. #### RI Coverage Reports: AWS Hits You With the Shame Stick Today Amazon announced Reserved Instance coverage reports in an effort to simultaneously help you control your costs while feeling bad about your infrastructure. At an implementation level this makes a lot of sense; define a percentage threshold for instance hours you want covered by reservations (a common number for initial baseline reservations is 70%), and highlight the deviation whenever you dip below that threshold. Many of my clients target a RI coverage rate above 90%-- which you can do once you have an accurate forecasting model, and automation that handles RI purchasing. SO FAR, THIS SOUNDS GREAT-- SO WHERE'S THE PROBLEM? Stragetically, this continues to position AWS as a cloud computing environment where you need to plan your spending one or more years in advance. As they themselves say, one of the best aspects of AWS lies in its elasticity. You can have your baseline usage squared away, run a Super Bowl ad, scale to 1000x your typical volume, and then dial it back down as traffic ebbs, all in near realtime. That model's diametrically opposed to a CapEx centered world wherein you have to plan in advance what your usage is going to look like, and commit to 1 or 3 years at that capacity. (If you can do that accurately enough, the economic benefits of AWS become more unclear.) As a result, if you don't know what your usage is going to look like in three months, AWS's approach to RIs begins to take on a darker tone, where the unstated message is "you're terrible at forecasting and should feel bad about that." Further, it will be interesting to see how many of the cost analysis vendors such as CloudCheckr, Cloudability, CloudDyn, and others respond to this move. I firmly believe that visibility into your AWS bill should be something that AWS provides natively, so moves like this one are definitely in the customer's interest. How this is going to impact areas of Amazon's partner ecosystem remain to be seen. If you have questions about this or other AWS billing topics, please email me-- I'm eager to hear your perspective! #### Right Sizing Your Instances Is Nonsense Already, a good dozen companies that purport to "right size" or "adjust your instances' sizes and families to fit your workload" are shrieking and reaching for the torches and pitchforks, but hear me out. We're all on the same side here: Nobody wants to see money wasted on cloud services. The theory behind right sizing is sort of like this: You walk into a new environment, one you've never seen before. You start poking around. Ha! You see a bunch of m3.2xlarge instances running. Upgrading them to m5.2xlarge instances instantly saves 28%. "What ancient moron set this up?!" you confidently ask—invariably to said "ancient moron," who is now actively incentivized not only to fight any recommendation you might possibly make, but also to see you run over in a tragic parking lot incident. "If you upgrade to m5.2xlarge instances instead you'll save 3.4 cents per hour per instance! Plus, your systems are idle most of the time; making them m5.larges instead saves an additional x per instance per hour! It's a slam dunk! Now, here's the part where you pay me." Why Right Sizing Is Wrong On paper, right sizing makes an awful lot of sense. Your existing nodes are largely bored (the average CPU utilization of EC2 instances often hovers in the range of single-digit percentages), newer instance families are a lot more efficient (and cost effective—i3 instances are roughly a third the cost of i2 instances), and nobody wants your workloads to sit idly. That said, virtually nobody actually right sizes their instances and suggesting that someone do it as a low-effort change is almost always incorrect. Why is that? It turns out that despite what the modern best practice evangelists preach at disturbingly high volume, an awful lot of workloads are "legacy," which is condescending-engineer-speak for "actually makes money." They generally aren't terrific at handling cluster members joining or leaving, they're monolithic, and it's a near-certainty that they've got system dependencies that will bust themselves to chunks if deployed on a more modern OS. Newer instance families use different hypervisors (Xen for the old, Nitro for the new), which means two things. First, older versions of operating systems don't support the newer hypervisor. So, you're not just migrating instances; you're upgrading the entire OS as well—which includes a hideous number of version dependencies. If you’ve containerized your workload to the point where you don’t care about this, great--you probably want to look at spot fleets instead. Second, a lot of these workloads are "certified" by either external vendors or internal divisions to run on certain versions of various bundled libraries. Suddenly, upgrading them to the newest version doesn't work nearly as well as you'd hope.   Also, unless you had the foresight to buy convertible-type reserved instances to begin with, you'll likely find yourself wasting a lot of previously committed money after being forced to do this eventually. Within a family and generation (for example, m3, c4, t2...) size doesn't matter; a single 4xlarge decomposes to four xlarge instances, and vice-versa. But standard reserved instances don't carry over between generations or families. Lastly, you're almost certainly using the wrong instances. With over 180 distinct SKUs in us-east-1 alone, you're statistically almost never going to be using the proper instance type for your workload unless you spend significant time benchmarking your application. In conclusion, if a tool, person, or tool of a person comes in to take a look at your AWS environment and casually suggests migrating to newer instances as a "quick win," show them the door. ‘ It's a win. But it's certainly not an easy one for most environments. #### S3 Intelligent-Tiering: What It Takes To Actually Break Even As Cloud Economists, we're often asked when it makes sense for an object to be in Amazon S3's Intelligent-Tiering ("S3-IT") storage class. The answer, as is unfortunately often the case in the world of consulting, is "it depends". There are two primary considerations before you jump into S3-IT: an object's access pattern and its size. S3-IT and object access patterns S3-IT is an extremely handy way to make sure your objects are stored in the appropriate storage tier without having to write complicated lifecycle management policies or incur the cost of lifecycle transitions and minimum retention periods. Maintaining lifecycle management policies and making thoughtful choices about object tiering both require engineer time, which is a lot more expensive than the $0.0025 per 1,000 objects monthly management fee. For this reason, our Cloud Economists often recommend that clients treat S3-IT as the default unless their objects' access patterns are extremely well-understood. In many cases, it's cheaper to just let S3-IT figure out where to put your object. There’s one additional S3-IT caveat customers should be aware of. The S3 Standard storage class is designed for 99.99% availability, while S3 Intelligent-Tiering loses a 9 from the end of that target to offer only 99.9% availability. At large scale, you'll indeed start to see object retrieval failures more frequently on tiers other than S3 Standard. S3-IT and object size Since S3-IT is a good default option for most objects' access patterns, let's take that off the table and only look at the monthly storage component of the object's total cost of ownership (TCO). How big does an object need to be in order for S3-IT to make more sense than the Standard tier from a storage cost perspective? To come up with a concrete answer to this question, let's make the simplifying assumption that an object is written once and never read or re-written thereafter. Let's also make the simplifying assumption that a month is 30 days long, so we don't have to do fractional math to compute the average cost of an object in GiB-days. So, to calculate the TCO of this hypothetical object, we have to model the object's movement through S3-IT's various tiers over time. In all different flavors of Intelligent-Tiering, a new object's first three months of existence are the same: It spends the first month in the Frequent tier and the next two months in the Infrequent Access tier. Where it spends the rest of its existence depends on whether S3-IT's Deep Archive or Archive tiers are enabled for that object. Therefore, there are three flavors of S3-IT to consider: "Vanilla" S3-IT: If neither Deep Archive nor Archive are enabled, the object spends the rest of its existence in the Archive Instant tier. This is the default option.S3-IT + Deep Archive: If only Deep Archive is enabled, the object spends three months in the Archive Instant tier and all subsequent months in the Deep Archive tier.S3-IT + Archive + Deep Archive: If both Archive and Deep Archive tiers are enabled, the object spends three months in the Archive tier and the rest in the Deep Archive tier. To make things even more complicated, S3-IT tiers tack on an additional management overhead fee per object-month, and the Archive and Deep Archive tiers store some additional metadata that you also pay for: 8 KiB for the name of the object (billed at Standard tier rates) and 32 KiB for "index and related metadata" (stored at the Glacier and Glacier Deep Archive rates, respectively). If we think back to high school math class, it sounds like storage cost in S3-IT as a function of time is a piecewise-linear function. Luckily for us, cost of ownership over a given fixed period of time is a linear function of object size. Calculating S3-IT costs Since cost of ownership over a fixed period of time is a linear function, we can describe the relationship between object size (x) and storage cost (y) over a fixed period like this: y_1 is how much it costs to store x_1 bytes for the given time period, and m is the marginal cost of storing an additional byte for that same time period. Calculating S3 Standard costs We also know that storage cost in the Standard tier is a linear function of object size, but it's much simpler: C is the cost per GiB-month of Standard storage in a given region multiplied by the length of the time period in months. Calculating the Break-Even Point So far, we have two equations that can tell us how much it costs to store an X KiB object for a given duration in both S3-IT and Standard. The "break-even point" for a given storage duration is the intersection between those two equations, or the value of x for which: We can solve the above equation for x, which yields: For object sizes greater than that intersection point, it's cheaper to store the object in S3-IT. For object sizes less than that point, it's cheaper to store the object in Standard. Intuitively, the longer you store an object of a given size, the more benefits you accrue from S3-IT and the lower that break-even point should be. The plot below shows that break-even point for storage durations from one year to 20 years. As you can see, for all three flavors of S3-IT, the break-even point starts to level off once the object's lifetime surpasses about 10 years. The break-even point for S3-IT with colder tiers available is a little lower because the savings in storage costs in those colder tiers add up over time. In all cases, though, the object size at which your break even is very close to the 128 KiB minimum size, regardless of how long you store the object for. S3-IT has come a long way since its introduction in 2018. At this point, it's a good default option for most objects under most workloads. One big drawback to adopting S3-IT is that moving objects from other tiers into S3-IT is expensive: it costs $0.01 to transition 1,000 objects, which can add up quickly if you've got a lot of objects to transition. However, new objects created in the S3-IT tier aren't subject to that transition fee, so adopting S3-IT for new workloads won't cost you anything. [1] Because this sort of thing can get complicated, it's worth clarifying that these equations are a function of x (object size) and also implicitly of T (storage duration in months). Solving the equation for the breakeven point is calcuating the breakeven point for a single value of T, and the plot that follows is a plot of that intersection over many values of T, not a plot of y=f(x). #### S3 Reduced Redundancy Storage is Dead Once upon a time, Amazon offered four tiers of object storage: Standard S3 (this is S3 as commonly discussed) Infrequent Access S3 (costs less, but you pay to access it more frequently) Reduced Redundancy S3 (similar to standard S3, but offers 4 9's of durability instead of Standard S3's 11 9's) Glacier (long term storage that doesn't need to be available instantly) Officially, Reduced Redundancy S3 still exists. That said, it's apparent that Amazon isn't moving forward with the offering, based upon two datapoints: First, they no longer talk about it at all in blog posts blog posts or other public discussions of S3 storage classes. Second, and more relevant to optimizing costs, S3-RR no longer participates in service discounts that affect the other S3 tiers. As a direct result, while S3 Reduced Redundancy still exists, you will pay more to store objects there than you will in Standard S3, while achieving worse object durability. In conclusion, if you've got any objects living in Reduced Redundancy, it's time to migrate them to standard S3. Your bill and your data durability will both thank you for it. #### Should I Purchase AWS Through a Reseller? The Duckbill Group's clients frequently ask whether they should purchase AWS through a reseller instead of direct. Conversely, others wonder whether it’s worthwhile to leave their reseller and go direct.  There's not one simple answer to those questions, so we've taken some time to do a thorough investigation on the matter. First, some context. A reseller is a particular type of AWS Partner. There are five types of partners: resellers, consulting partners, managed services, independent software vendors (ISVs), and system integrators. Some partner companies fall into multiple buckets and provide a combination of these services — for example, it’s common for a reseller to also provide consulting services and managed services. When you should use an AWS reseller The primary reason to go through a reseller is because they _can_ add value to your organization above and beyond the AWS services they’re reselling. For companies that benefit from buying AWS through resellers, we've seen this value show up in some combination of the following four ways.   Expertise Many partners that provide reseller services also provide expertise in some fashion by way of one-time consulting services. For example, some partners are known for their deep expertise in a certain industry or problem domain, so they provide that expertise in addition to the reseller services. This expertise is bundled together with the resold AWS services. Invoice consolidation  Some companies want fewer vendors and prefer a “single throat to choke,” so to speak. Many resellers happily do this, providing the company with a single invoice that includes a wide range of other third-party services. Enhanced support Many resellers provide additional support above and beyond what AWS provides. This can take the form of supplying additional, dedicated-to-you support staff to answer tricky AWS questions or by providing some additional bundled software to help manage AWS. For example, most resellers bundle cost management products like CloudHealth and Cloudability for free or at a deep discount, alongside the resold AWS services.  Discounts (with a catch) The core value proposition for every reseller is that AWS is cheaper through them than if you go direct. There’s a lot of truth in this — up to a point. For companies not eligible for private pricing with AWS directly (those spending under $1 million/year), a reseller can provide private pricing discounts you wouldn’t otherwise be eligible for. As mentioned above, many resellers can bundle other third-party services, and these bundled services may come with substantial discounts off retail pricing, thanks to the bundling. Note that some resellers also lock the customer out from using AWS Cost Explorer as a result and force them into using a third-party product for exploring costs. When you shouldn’t use an AWS reseller Just because a reseller can add value for companies doesn't mean that they'll add value for _your_ company specifically. Ultimately, that's a judgment call that your company will need to make for itself. We've found two big reasons businesses shouldn't buy AWS through a reseller.  No AWS Organizations & Control Tower As mentioned above, some resellers block access to AWS Cost Explorer, which is decidedly not great. However, some resellers go even further and force you into their AWS Organization setup, preventing you from using AWS Organizations, Control Tower, and associated functionality. If the capabilities from these services are important to you, then be cautious with a reseller. Not every reseller places these limitations on their customers, and we've found it to be inconsistent even within a single reseller. No value-add  If your reseller isn’t providing any value-added services you want, then there’s no reason to use a reseller for your AWS services. This can be particularly true for the discounts resellers offer: If you’re eligible for private pricing directly with AWS, the discount level will generally be the same as what the reseller would provide, as they are simply passing that discount through. In other words, if your reseller is _only_ providing discounts and no other value-added services, you should consider going direct with AWS to simplify your vendor management. Vendor relationship management  Speaking of vendor management, AWS is often an organization's single largest expense after payroll and office space. It’s nearly always the single largest vendor expense for tech companies. At that point, AWS is more of a partner to your company than just any usual vendor (whether you want them to be or not!). We at The Duckbill Group believe anything that puts an intermediary layer between you and a critical partner is an undesirable thing. We recommend our customers own their core vendor relationships instead of putting an intermediary in between. Analyze your needs before buying AWS through a reseller There's not a universal answer to whether it's a good idea to purchase AWS through a reseller. It comes down to what your company needs. Ask yourself a few key questions to determine whether a reseller can add value: 1. Do you lack certain expertise internally? 2. Do you want to simplify your billing process via invoice consolidation? 3. Do you need additional customer support or software? 4. Do you spend less than $1 million/year on AWS services? If the answer to all of the above is "no," then we suggest cutting out the middleman: Work directly with AWS to purchase the services you need. #### Skyway: Cloud cost management for the 9-figure club When Corey and I started The Duckbill Group in 2019, we felt like we were late to the party and everyone was walking out the door. We were the only consulting firm focused solely on cost management, and it felt like no one really cared. On the software side, everyone was still talking about Cloudability and CloudHealth, as both had only recently been acquired. AWS was at a $29bn/yr run-rate then ($130bn/yr now, for comparison!) and GCP/Azure were tiny by comparison—in fact, we worked with the #1 GCP customer in 2019 and they were "only" $100m/yr. It felt like the peak had already happened, and we had missed it. Despite that, we quickly became the go-to for everyone who cared… but, honestly, not a lot of people cared. Slowly, then suddenly and all at once Then 2020 hit, and we were slammed with more than we could handle. We even threw our own hat in the ring on an ill-fated SaaS idea, but quickly concluded that market wasn't for us. So, we doubled-down on being the absolute best at enterprise cloud cost management. By the time we came up for air in 2022, ZIRP was over, and we were surrounded by dozens—hundreds!--of companies focused on cloud cost management. Now, there are 150+ SaaS companies working on cloud costs to some degree and more launching every day. Quite the shift from where things were in 2019. Opinions are like assholes And boy, do we have both in spades. Having watched many of these companies launching (and folding, as has often been the case), we've had a lot of opinions on the right way to solve these problems. For a while, many of the SaaS cost management companies would ask our advice but be unhappy with our feedback, and ultimately just go back to building what they were already thinking. Last year Corey asked me a question: how much do we really believe we're right about this? No other company on the market sees the world as we do and if we really do believe in our own vision, we should build it—or just shut up. Introducing Skyway Today we're launching Skyway, the first cost management platform built exclusively for those companies who are actually at-scale and spending $100m/yr+. For the past year, we've been quietly building the first piece of what is a much larger vision. The first of many product modules to come, Skyway's Contract Manager encodes everything we've learned from negotiating tens of billions in cloud contracts, enabling FinOps, procurement, and Finance to manage contract commitments, calculate discounts, validate you're actually getting those discounts, negotiate renewals, and proactively identify ways to improve the contract. Skyway's Contract Manager is available in private access now. If you're interested in learning more, head on over to the Skyway page and send us a note. It's just Duckbill now As part of all this, we're also announcing our rebrand from The Duckbill Group to simply Duckbill, as a reflection of our transition from a pure consulting firm to a services and software company. Our new branding reflects the values Duckbill has come to represent: our customers trust us to guide them to the right answers every time, enabling them to become true masters of their business. Editor's Note: Last Week in AWS and our beloved Billie the Platypus aren't going anywhere. New ways to support our customers The services you've all come to know and rely on aren't going anywhere. In fact, we've greatly expanded our capabilities: we now advise on Google Cloud and Azure, and will be expanding our reach into other major SaaS and AI platforms such as Datadog, Anthropic, OpenAI, Snowflake, and Databricks soon. (All of these will be part of Skyway in due time!) Beyond our expansion of capabilities, we've simplified how to work with us: Want assistance in negotiating your cloud contracts? We've got you—we've handled tens of billions across multiple F500 enterprises. Need to learn where your FinOps program is falling short and what the best-of-the-best are doing differently? Yeah, we can help—whether you're a one-person show or 30-strong like some of our customers. Not sure exactly, but want us around just in case? Our newly-redesigned services retainer aka "Duckbill on Demand" has got you covered. And Skyway, of course! On behalf of the entire Duckbill team, it's been an incredible 7 years, and we're looking forward to serving you all for many more years to come. #### Spending Money to Save Money with Savings Plans and Reserved Instances As a Cloud Economist, a significant portion of my day is spent staring at the AWS Cost Opportunity Recommendations and slowly muttering, “Why… but why…”  AWS offers a number of ways to save on the infrastructure you already have — such as Savings Plans, EC2 Savings Plans, and Reserved Instances for EC2, RDS, ElastiCache, and ElasticSearch.  The best thing is that they’re named so similarly and each with their own “special sauce” of recommendations that it’s deceptively easy to get lost in.  Let’s do everyone a favor and actually explain what these things are and which is actually better.  What is an AWS Savings Plan? A Savings Plan is a commitment to spend some minimum amount in compute services per hour in exchange for a discount.  When a Savings Plan is purchased from the organization account, AWS will apply that discount to the most highly used compute services in that account or its children.  What is an EC2 Instance Savings Plan? An EC2 Instance Savings Plan is priced similarly at a higher discount for a specific instance type.  If you had a specific instance type in an organization account and a child account and then purchased an Instance Savings Plan for it, that discount would apply to instances in the accounts that had that instance type—but not any others.  What is a Reserved Instance in AWS? A Reserved Instance is an instance that is yours to use for the term of your reservation.  You get a deeper discount and aren’t held to per-hour commitment levels. A child account can also use any reservations made by the organization account.  So, which is better? That depends on how much you like managing it.  If you hate managing it and you have fairly steady traffic, you can get a Compute Savings Plan. You can save up to 66% and have it cover your EC2s as well as your Lambda and Fargate Spend. It is truly AWS’s managed service savings solution.  But what if your traffic isn’t steady?  If you know what sorts of EC2 instances you’re going to be using, then Reserved Instances are the way to go. Depending on your level of commitment and reservation, you could get a 60% discount off of your spend. There lies the problem with EC2 Instance Savings Plans. For those, you need to know which instance types you’re using. Then, AWS will dynamically assign discounts to that usage while charging per hour.  If an EC2 type’s spend does not meet that hourly commitment, those dollars are wasted. You essentially lose the flexibility of a Compute Savings Plan as well as the aggregated discount of Reserved Instances. Unless you are autoscaling a specific instance type with a base level of spend, you will end up spending money on discounts you will never see.  But I have spikes!  Then get both.  You are not the Evil Space Wizard dealing in absolutes. Use a Savings Plan where you can dictate your base spend. Chances are that a bit of analysis can tell you which services are spiking as well as the instance types they’re using. Use reservations for those spiking services and let the savings plans take care of the rest. The bottom line is that Savings Plans commit you per hour and Reserved Instances don’t. Strategize from there.  Of course, that may be easier said than done. If you need some help, give us a shout. #### The Complete Guide to CloudFront's Flat-Rate Pricing (And Who It Actually Helps) I've been complaining about CloudFront pricing for the better part of a decade. Not because it was expensive—though it often was—but because nobody could tell you what it would cost until the invoice arrived. Not customers. Not AWS sales. I'm half-convinced not even the CloudFront team knew. In November, AWS quietly (not intentionally quietly, mind you; they just forgot it was re:Invent season and it got lost in the noise) launched flat-rate pricing bundles for CloudFront. I've been sitting on this take for two months because I wanted to see if I was missing something. I wasn't. The Numbers That Matter Per distribution: Plan Monthly Cost Data Transfer Requests Break-even File Size Free $0 100 GB 1M 100 KB Pro $15 50 TB 10M 5 MB Business $200 50 TB 125M 400 KB Premium $1,000 50 TB 500M 100 KB That last column is the one AWS doesn't advertise: the average request size you'd need to max out both limits simultaneously. The Pro plan's 50 TB / 10M requests means you need 5 MB average responses to use what you're paying for. If you're serving software binaries or video chunks, that's realistic. If you're serving API responses or web assets, you'll hit the request ceiling long before you touch the bandwidth. Now the math AWS hopes you won't do. Under pay-as-you-go pricing, 50 TB of data transfer out to internet from North America runs about $0.085 per GB. That's $4,250. The Pro plan offers the same 50 TB for $15. That's not a "discount," that's a 99.6% price reduction for bandwidth-heavy workloads. I double-checked this math three times because it seemed like a typo. Credit where it's due: this is genuinely good. I don't say that often about AWS pricing changes. Usually "simplified pricing" means "we moved the complexity somewhere you won't notice until the bill arrives, and the reworked pricing dimensions mean your bill just skyrocketed." This time? $15 for 50TB is just... a good deal. I'm as surprised as you are. To put it in smaller terms: 10 TB/month on pay-as-you-go runs roughly $765 in egress alone (before WAF, DNS, and logging). The Pro plan covers that plus all the bundled services for $15. At 20 TB/month, traditional pricing approaches $1,565. Still $15. Most AWS accounts that've been around longer than twenty minutes have a graveyard of CloudFront distributions. The one you created for a demo in 2019. The staging environment nobody decommissioned. The "temporary" test setup that has now outlived three engineers and two corporate reorgs. The one named "DO-NOT-DELETE-ASK-JENKINS" that nobody has touched since Jenkins was replaced by GitHub Actions in 2022, but everyone's too afraid to find out what breaks. This pricing only matters for the one or two distributions actually doing work. The rest can stay on pay-as-you-go at effectively zero cost, since they're serving traffic to an audience of exactly nobody. The Cloudflare Elephant Cloudflare Pro costs $25/month (or $20/month with annual commitment) and includes unlimited bandwidth. No caps. No throttling. CloudFront Pro is $15/month with 50TB and 10M requests. On paper, Cloudflare wins. On your actual infrastructure, AWS wins—and they know it. If your origin is S3 or EC2, you currently pay $0 in egress to CloudFront—data transfer within AWS is free. Switch to Cloudflare and you start paying AWS egress charges: roughly $0.09/GB. At 50TB/month, that's $4,500 in egress alone, every month, forever, unless you store static assets in their S3-alike. (I know, there are exceptions; this math is intentionally illustrating extremes.) AWS can afford to offer 50TB for $15 because they know you're not leaving. The real price of CloudFront flat-rate isn't $15/month—it's $15/month plus the AWS ecosystem lock-in you've already accepted. For greenfield projects with no AWS dependency, Cloudflare's unlimited bandwidth (until it isn't, but you'll get a sales call instead of a Surprise Invoice) is hard to argue with. For existing AWS shops, the math gets complicated fast. What's Actually Included Each plan bundles services that would otherwise require separate billing adventures: CloudFront CDN (obviously) AWS WAF with bot management (mandatory—you can't opt out; WebACL must be associated) DDoS protection (blocked attacks never count against your allowance) Route 53 DNS (with caveats—see below) CloudWatch Logs ingestion (but not storage or querying—those still cost money) TLS certificates via ACM CloudFront Functions (serverless edge compute) S3 storage credits (5GB Free / 50GB Pro / 1TB Business / 5TB Premium—offsets any S3 Standard storage in your account) Route 53 gotchas: Hosted zone must be in the same AWS account as the CloudFront distribution. Cross-account Route 53 setups don't work with flat-rate plans. ALIAS records pointing to CloudFront or other supported AWS services don't count against your DNS query allowance, but CNAME records do. If you exceed DNS query limits, AWS can automatically transition your hosted zone back to pay-as-you-go without asking. DNSSEC KMS costs and health checks bill separately. The WAF value: Pro tier gets 25 WAF rules versus 5 on free. Each rule normally costs $1/month. That's $25/month in WAF value for a $15 plan. If you were going to use WAF anyway, the Pro tier pays for itself in WAF rules alone. The WAF catch: WAF is mandatory on flat-rate plans. AWS, in their infinite wisdom, has decided you need a Web Application Firewall whether you want one or not. Running a simple static site and consider WAF overkill? Too bad. You're getting it anyway (though you need not configure it with any rules). I'm sure this has nothing to do with WAF being a separate revenue stream they'd like you to get comfortable with. The Limits Nobody Talks About Each tier has quotas beyond requests and bandwidth: Limit Free Pro Business Premium Cache behaviors 5 10 50 100 WAF rules 5 25 50 75 Route 53 records 50 100 1,000 5,000 DNS queries (non-ALIAS) 1M 5M 20M 100M Request body inspection 16 KB 16 KB 64 KB 64 KB Cache behaviors are the sleeper constraint. If you have a complex site with different caching rules for /api/*, /static/*, /images/*, and a dozen other path patterns, you might hit the Pro tier's 10-behavior limit before you touch the request ceiling. Business tier jumps to 50, which is why some organizations will end up there even with modest traffic. Tier-Locked Features Not all features are available on all tiers: Feature Availability Geographic restrictions Free and above Rate limiting Free and above Common bot detection Business and Premium only Origin Shield (load reduction + performance) Business and Premium only Custom response headers (CSP, CORS) Business and Premium only Private VPC origins Business and Premium only Automatic origin failover (origin groups) Premium only Mutual TLS (mTLS) Premium only That origin failover restriction is significant. The December 2021 us-east-1 outage is a concrete example where origin groups would have prevented P1 incidents for a lot of companies. If high availability matters, you're looking at Premium ($1,000/month) or staying on pay-as-you-go. What's NOT Included Lambda@Edge doesn't work with flat-rate plans. This is the one that's going to hurt. If your architecture depends on complex edge transformations, request manipulation, or origin selection logic that lives in Lambda@Edge—and you'd be surprised how many architectures accidentally evolved this dependency without anyone noticing until the architect left—you're stuck on pay-as-you-go. No migration path. No "CloudFront Functions are basically the same thing." They're not. CloudFront Functions are a toy by comparison: 2MB memory, 10KB code, JavaScript only, no network access, no AWS SDK. If you're doing anything more sophisticated than header manipulation, Lambda@Edge is why, and flat-rate pricing is not for you. Here's the uncomfortable reality: Lambda@Edge is basically unmaintained at this point. CloudFront Functions is AWS's "way forward," and flat-rate pricing makes that crystal clear. If you're dependent on Lambda@Edge capabilities, AWS isn't building a bridge for you—they're hoping you'll eventually accept the limitations of CloudFront Functions or stay on pay-as-you-go forever. Neither option is great. I suspect this exclusion is load-bearing. Lambda@Edge workloads are unpredictable by nature—that's the whole point of running arbitrary code at the edge. AWS can't flatten pricing on something they can't model. Fair enough. But they could have been clearer about this being a fundamental constraint rather than burying it in a compatibility matrix. I counted 23 features that won't work with flat-rate plans. Most are obscure, but a few will bite: Compute & Logging: Lambda@Edge (covered above—this is the big one) Real-time logs (standard CloudWatch Logs only; no Kinesis streaming) Parquet format logs (extra cost) Distribution Features: Continuous deployment and staging distributions Multi-tenant distributions Anycast IP list configuration Field-level encryption Dedicated IP/SSL (SNI only) Legacy cache settings (must use cache policies) Origin access identity/OAI (must migrate to OAC) IAM server certificates (must use ACM) Security Features: Can't combine with Shield Advanced Can't combine with Firewall Manager WAF Features Excluded: Targeted Bots (common bot detection is included at Business tier, but targeted/advanced bot mitigation is not) CAPTCHA (challenge only) Partner Managed Rules Account Creation Fraud Prevention Account Takeover Protection Rule Groups (must create individual rules) The Shared Resources Trap: If your CloudFront Functions or WAF WebACLs are shared across multiple distributions, you can't subscribe those distributions to flat-rate plans. Each plan requires dedicated, non-shared resources. This means organizations that built reusable edge function libraries or centralized WAF rule sets face a choice: duplicate those resources per distribution (operational overhead) or stay on pay-as-you-go. The One-Domain-Per-Plan Problem The constraint that kills flat-rate for multi-tenant SaaS: one apex (root) domain per plan. If you're running a platform where each customer gets their own subdomain (customer1.yourplatform.com, customer2.yourplatform.com), you're fine—subdomains under the same apex can share a plan. But if you're serving customer-owned domains (customer1.com, customer2.com), each apex domain needs its own plan. Fifty customer domains? Fifty plans. And there's a hard cap: 100 plans maximum per AWS account. Beyond that, you need multiple AWS accounts or a different architecture. For multi-tenant SaaS with custom domains, flat-rate pricing doesn't scale. Stay on pay-as-you-go or rethink your domain strategy. Account-Level Restrictions The limits nobody reads until they hit them: 100 plans maximum per AWS account (across all tiers) 3 Free plans maximum per AWS account AWS Free Tier accounts are ineligible entirely—you need a paid account Exceeding 500M requests OR 50TB/month? You can't self-serve. Contact AWS sales for custom pricing. The Tier Differentiation Problem Look at what actually changes between Business ($200) and Premium ($1,000): more requests, more cache behaviors and WAF rules, origin failover, mTLS, and Origin Shield for load reduction. What doesn't differentiate them is more telling. Geographic restrictions and rate limiting are available on every plan, including Free. Common bot detection starts at Business. So the gap between Business and Premium boils down to high-availability architecture (origin failover) and mTLS. That's it. Nobody choosing between self-service tiers cares about advanced CDN security features. By the time you're dealing with sophisticated bot attacks and need custom WAF rule sets, you're in enterprise territory with negotiated contracts and dedicated TAMs. You're not comparison shopping on a pricing page. AWS got lost in the weeds here. They built differentiation for a customer that doesn't exist at these price points. (I could be wrong about this—maybe there's a thriving market of mid-sized companies who desperately need origin failover but won't spring for enterprise contracts. If so, they're not my clients so I don't know where they're all hiding.) The actual decision points are: Do I fit in Pro's cache behavior limits? (10 behaviors) Do I need more than 10M requests? Do I need origin failover? (Premium only) Do I need mTLS? (Premium only) Everything else is a rounding error. The "No Overages" Fine Print AWS's marketing emphasizes "no overage charges," which is technically true but deserves an asterisk the size of us-east-1. When you exceed your allowances, AWS doesn't bill you more—they (potentially) degrade your performance. From the documentation: "AWS may take appropriate action, which may include reducing your performance (for example, serving your traffic from fewer or more distant edge locations)." The operational nightmare that isn't being mentioned is there's no discernable signal when this happens. "We're being rate-limited by our CDN plan" and "CloudFront is just slow today" look identical from your monitoring. Your on-call engineer at 2 AM isn't going to intuit that the latency spike correlates with hitting a request cap on a pricing plan they didn't know existed. They're going to spend three hours checking origin health, reviewing recent deployments, and questioning their career choices. It gets better. AWS says usage notifications "may be delayed." You'll get emails at 50%, 80%, and 100% of your allowance—eventually. Meanwhile, you might already be throttled. Build your own alerting against the CloudFront metrics. Don't rely on AWS to tell you in time, because they won't. You won't get a surprise bill, so much as you'll get a surprise outage investigation that ends with "oh, we were throttled and nobody told us." That's worse. A bill is something that you can explain to finance. A P1 incident where the root cause is "we forgot about a pricing tier limit" is the kind of thing that ends up in postmortems with the word "embarrassing." There is one bright spot: blocked DDoS attacks and WAF-rejected requests don't count against your allowance. If you get attacked, at least the malicious traffic won't eat your quota. Who Should Switch Tomorrow Obvious wins: Content-heavy distributions under 10M requests/month serving large files (software downloads, video, firmware updates) Organizations that have been meaning to adopt WAF but kept putting it off because the pricing was impossible to predict Teams whose finance departments have threatened violence over variable CDN bills (you know who you are) Existing AWS shops with S3/EC2 origins—the egress trap doesn't apply to you, so the Cloudflare comparison is irrelevant Simple sites with fewer than 10 cache behaviors and no edge compute ambitions Test carefully first: Media streaming (ad beacons and playlist manifest requests add up faster than you'd think) Anything latency-sensitive that might spike past allowances and trigger silent throttling Sites with complex path-based caching (check cache behavior limits) Organizations with shared CloudFront Functions or WAF WebACLs across distributions Stay on pay-as-you-go: Traffic regularly exceeding 500M requests/month (requires AWS sales conversation anyway) Lambda@Edge dependencies Enterprise customers with negotiated rates that already beat these numbers Distributions serving small files with high request counts—do the break-even math first Multi-tenant SaaS with customer-owned domains (one-domain-per-plan kills you) Multi-CDN architectures where you need failover flexibility Greenfield projects where Cloudflare's unlimited bandwidth makes more sense Anyone needing Shield Advanced, CAPTCHA, or Account Takeover Protection The Bigger Picture This isn't primarily a pricing play so much as it is an admission. AWS is conceding that CloudFront's traditional pricing model was too complex for most customers to predict or manage. Under pay-as-you-go, your costs depend on which edge locations serve your traffic. Unless you know exactly where your users are, you can't predict that. Try whiteboarding a global architecture when the answer to "what will CDN cost?" is "depends on the ratio of Australian to European users, which we won't know until we launch." Your unit economic modeling will dress up what's realistically a shrug with a dollar sign. There's a subtler shift here too: who eats the risk. On pay-as-you-go, you do—a traffic spike, a DDoS attack, a viral blog post, and suddenly you're explaining a five-figure invoice to finance. On flat-rate, AWS absorbs that variance. Complexity and fear of massive bill spikes were the two things that made CloudFront pricing a recurring nightmare, and flat-rate kills both. They're competing directly with Cloudflare's "just tell me what it costs" simplicity. Bundling WAF, DNS, and edge compute into a single line item is AWS quietly admitting that their à la carte model created friction that pushed customers toward simpler alternatives. This also repositions CloudFront against the enterprise CDN incumbents. Akamai typically runs 20-40% more expensive than CloudFront for equivalent traffic, with pricing negotiated on a case-by-case basis (good luck finding a pricing page on their website!). Fastly revamped their pricing last November—the old $50/month minimum is now legacy, replaced with a free tier (100 GB, 1M requests) and packages starting at $1,500/month. That's a hundred-to-one gap between CloudFront Pro at $15 and Fastly Basic at $1,500. For the mid-market customer who doesn't want to negotiate enterprise contracts but needs more than Cloudflare's one-size-fits-all approach, CloudFront just became the obvious answer. As AWS principal product manager Cristian Graziano put it: "Plans do not require an annual commitment to get the best available rates." That's AWS acknowledging that their traditional commitment-based discounting was losing deals to simpler alternatives. I've spent years arguing that AWS pricing complexity is a feature, not a bug, because it lets sophisticated buyers optimize in ways flat-rate models don't allow. That's still true for large enterprises with dedicated FinOps teams. For everyone else? This is AWS finally meeting them where they are. The $15 Pro plan in particular feels like a loss leader designed to get smaller customers onto CloudFront and into the AWS ecosystem before they default to Cloudflare because the pricing page was easier to understand. And once you're on CloudFront with S3 origins, that $4,500/month egress penalty makes switching a lot less attractive. Migration Gotchas Historical usage affects eligibility. AWS looks at your recent CloudFront usage when you try to subscribe. If your traffic already exceeds a plan's allowances, you may be forced into a higher tier. You can't game this by subscribing to Pro when you're clearly a Business-tier customer. Disabled distributions still incur charges. Disable a distribution but don't cancel the pricing plan? You keep paying. Cancel the plan explicitly, or you're buying nothing for $15-1,000/month. You can't delete a distribution while subscribed. Cancel the plan first, then delete. Minor operational friction. You can mix pricing models. Keep your experimental distributions on pay-as-you-go (where you pay nearly nothing for low traffic) and move your high-volume production distributions to flat-rate. You don't have to go all-in. Upgrades are immediate; downgrades wait. Upgrade mid-cycle and you get prorated billing and immediate access to higher limits. Downgrade and it takes effect next billing cycle. Changes must propagate first. Wait until distribution configuration changes have propagated to all edge locations before subscribing. Don't subscribe mid-deployment. Unsupported features block subscription. Lambda@Edge, real-time logs, or any other unsupported feature? You must remove them before you can subscribe. The console will block you. What I'd Do I don't have a clean formula for this. Every client engagement I've done involving CloudFront has been its own special snowflake of traffic patterns and architectural constraints. But if I were looking at this for a client tomorrow, here's roughly how I'd approach it. First, audit what you actually have. Pull the list of CloudFront distributions, their traffic patterns over the last 90 days (requests and data transfer), and their configurations. Most accounts have more distributions than anyone remembers. Figure out which ones matter. Then check for blockers. Lambda@Edge? Shared Functions or WebACLs? Real-time logs? Shield Advanced? Any of these disqualify you from flat-rate, full stop. Don't waste time modeling pricing for distributions that can't migrate. Heck, some of my own distributions use some old patterns for setting TTLs and thus aren't eligible without headache. AWS actually made this part easier than I expected: the CloudFront console shows eligibility based on features and usage for each distribution. You can hover over any disabled tier to see exactly why you're ineligible—whether it's a specific feature, too many cache behaviors, or WAF rule limits. It also shows your historical requests and data transfer for the last six months. Credit where due—this saves you from building your own eligibility spreadsheet. Calculate your average response size. This is the number that determines whether you're request-bound or bandwidth-bound. Under 400 KB and you're hitting request limits first. The break-even math shifts significantly depending on which constraint you're actually hitting. Here's the part most people get wrong: model your p95 month, not your average. Flat-rate plans don't care about your typical month, they care about your worst month. If your p95 exceeds a tier's limits, you need the next tier up or you'll get throttled exactly when it matters most—during your traffic spike, during your launch, during the one month your CEO is actually paying attention. Count your cache behaviors. This one's easy to overlook. Over 10? Pro won't work regardless of traffic. Check this before you do any math. Check your origins. Private VPC origins mean Business tier minimum. Cross-account Route 53 means flat-rate won't work at all. These constraints aren't obvious from the pricing page. Build your own alerting. I cannot stress this enough. CloudWatch metrics for requests and bytes transferred, with alarms at 50%, 80%, and 90% of your plan limits. AWS's "may be delayed" notifications are not a monitoring strategy. Start with something non-critical. Migrate one distribution, watch the usage counters for a full billing cycle, validate that your modeling was correct. Then expand. And document your upgrade path before you need it. If you hit limits, can you upgrade mid-cycle? (Yes, prorated.) What's the next tier cost? Who approves that spend? Have this conversation before you're in an incident. The flat-rate plans don't require annual commitments, so the risk of trying them is low. The risk of not evaluating them—and continuing to overpay for predictable, bandwidth-heavy workloads—is higher than most teams realize. Don't mistake this for AWS becoming simple. They've added a fourth pricing dimension to an already complex service. You now get to choose between pay-as-you-go, flat-rate tiers, enterprise agreements, and savings plans—while factoring in the implicit cost of ecosystem lock-in that makes all of this moot anyway. AWS finally made CloudFront pricing predictable. They just made the decision about which pricing model to use more complicated. Somehow, that feels very on-brand. #### The Duckbill Guide to AWS Reserved Instances When you're looking to lower your AWS bill, you're likely to come across AWS Reserved Instances ("RIs"). We're sharing the must-know facts and insights about RIs based on our extensive experience fixing our clients' horrifying AWS bills through our work here at The Duckbill Group. In this article, you'll learn what RIs are, how to pay for them, how they're applied, and interesting edge cases. We've also created a guide to AWS Savings Plans ("SPs"), another common vehicle for committed use discounts. What are AWS Reserved Instances? Reserved Instances are a mechanism AWS provides to commit to using a certain configuration of instances for a certain amount of time in exchange for a discount on the instance cost. One of the things many people new to AWS get confused about is that RIs are not a reservation of capacity, but rather a billing construct (except EC2 zonal reservations, which we explain below). Put simply, you’re not paying for AWS to have the instance available for you; it is entirely possible — though generally rare, in practice — that you could have RIs and be unable to launch the instances that RI calls for.  In fact, the best way to think about RIs is actually by using Google Cloud’s nomenclature: a committed use discount. RIs are a financial construct, not a technical one. Reserved Instances payment options All RIs for all supported services have three payment terms: all upfront, partial upfront, and no upfront. Payment options cannot be modified after you’ve made the purchase, even for no-upfront purchases. Payment options don't impact how Reserved Instances are applied--they only impact the discount you're receiving and when you pay for commitment. All-upfront payment terms are exactly what they sound like: You pay for the entire reservation upfront. This option provides the deepest discounts, but in Duckbill’s opinion, the increased discount is generally not enough to be worthwhile.  Partial-upfront payment terms mean you pay just some of the total cost when you purchase your RIs. There’s very little reason to use this unless you simply need to spend the cash but don’t want to spend as much now as an all-upfront payment. No-upfront payment terms are, again, exactly as they sound. Rather than paying any amount of cash upfront for the purchase, you pay for it monthly. The no-upfront payment option provides the least discount but, in Duckbill’s opinion, should be the default choice for every organization. Choosing no-upfront payment terms over all upfront or partial upfront comes down to how much you like cash in your bank account versus Amazon’s bank account. There are two exceptions to our recommendation to always use no-upfront terms: If you work in an organization that needs to spend its money within a certain budget period, paying upfront might be worthwhile. Paying upfront is a good way to shore up a potential shortfall in meeting your AWS contract's Enterprise Discount Program (“EDP”) or Private Pricing Addendum ("PPA") commitments. Reserved Instances term length All RIs for all supported services have two term lengths available: one year and three years. The term length cannot be modified after you’ve made the purchase. Three-year terms receive the best discount, but as you might imagine, planning your infrastructure for three years out can be difficult for many organizations. While the delta between one-year and three-year terms can be substantial (think 21% vs. 60%), Duckbill recommends defaulting to one-year terms unless you are confident in your workload forecast. What services are Reserved Instances available for? RIs are available for five services: EC2, Elasticache, OpenSearch, RDS, and Redshift (as well as DynamoDB, which has an RI-like structure). Each of these services has its own configuration, and RIs are not swappable among services. In other words, you have to manage RI purchases for each of these services individually. Amazon EC2 Reserved Instances WARNING: If you’re considering purchasing RIs for EC2, consider using Savings Plans instead. They are superior to RIs in every way, offering much more flexibility and an easier-to-understand purchasing model, while offering the exact same discount levels as RIs (though, weirdly, Savings Plans for SUSE Linux generally discount deeper than the equivalent RI for some reason). If you’re insistent on purchasing EC2 RIs, then read on. EC2 RIs have configuration options that make them distinctive from RIs for the other four AWS services: offer class, tenancy, and operating system/platform. Due to all of the configuration options, it’s helpful to think about instance pricing as a SKU, which happens to be how the AWS API provides the data. For example, there's a distinct SKU and price for the combination of:  Standard RI c5.xlarge instance In us-west-2 Running Linux On shared hardware  For a one-year term Paid all upfront Change any of these seven options, and you have a different SKU. Regional and zonal RIs There are regional RIs and zonal RIs. Generally speaking, people are talking about regional RIs when they discuss Reserved Instances. Regional reservations apply to any availability zone in the chosen region. Regional reservations offer what AWS calls “instance flexibility.” With instance size flexibility, an RI may apply to any instances within the same instance family of a smaller size than what the RI configuration is for. For example, if you have one RI for a single db.m6.xlarge and later switch to using a smaller db.m6.large, the RI would cover two of the new, smaller instance types. The reverse is also true: smaller instance reservations can partially cover larger instances, or multiple reservations can combine to cover larger instances entirely. You can see exactly how this would work in the EC2 Reserved Instance normalization chart. Zonal reservations are bound to a specific availability zone (remember, the naming of AZs is not guaranteed to be consistent between AWS accounts in your organization). Zonal reservations do not offer instance flexibility; you must know in advance what size instances you’re going to be using. One little-known benefit to zonal reservations is that they provide a capacity reservation as well, unlike regional reservations.  Offering classes There are two offer classes of EC2 RIs: Standard and Convertible. (Technically, there are also Scheduled Reserved Instances, but they support only a subset of older generation instances, the prices don’t make any sense, and it likely should have been formally deprecated ages ago.) Standard RIs are suitable for static workloads. You cannot change instance family, tenancy, operating system, or payment option. For this inflexibility, Standard RIs offer a larger discount over Convertible RIs. Convertible RIs allow you to modify instance family, tenancy, operating system, and payment option, providing more flexibility as your infrastructure evolves. For this privilege, your discount is smaller. If you’re buying EC2 RIs, you generally want Convertible, not Standard, unless you’re covering highly static workloads. Tenancy Tenancy determines whether you’re running on underlying hardware other customers are or if you have the hardware all to yourself. In the early days of the cloud, people didn’t really trust the isolation guarantees that AWS made, so customers required, for a fee of course, hardware platforms that other customers were guaranteed not to be running atop. In the modern era, these concerns are nowhere near as widespread as they once were. Nowadays, they're generally reserved for commercial software license requirements, such as MacOS EC2 instances.  Operating System This effectively bifurcates into free, open source OS vendors and commercial OS vendors, where commercial OS vendors include RHEL, SUSE, and, of course, Windows. The two options work the same way, but the discount level and pricing shift because the non-FOSS instances have software licensing costs baked into their pricing. As always, purchasing should follow what’s happening in your account; it shouldn’t lead the process. ElastiCache Reserved Cache Nodes ElastiCache Reserved Cache Nodes, as they’re technically called, are available for Memcached, Valkey, and Redis engine options. We’ll refer to them here as RIs, as they’re commonly called. ElastiCache RIs are roughly equivalent to EC2’s Standard RI offering class in terms of flexibility: When you pick a specific instance family and region, you’re stuck with those choices. However, unlike other Reserved Instances, you can change engines between Redis and Valkey and still receive the discount. If you decide to change instance families or move regions, you have to purchase new RIs, and the old RIs are left unused. In 2024, AWS introduced Reserved Instance size flexibility to ElastiCache RIs. Amazon OpenSearch Reserved Instances OpenSearch’s RIs are roughly equivalent to EC2’s Standard RI offering class in terms of flexibility: When you pick a specific instance type and region, you’re stuck with those choices. If you decide to change instance types or move regions, you have to purchase new RIs, and the old RIs are left unused. In other words, there is no instance size flexibility as found with EC2 and RDS. Amazon RDS Reserved Instances Like EC2 RIs, RDS RIs have "instance size flexibility" (exception: Microsoft SQL Server and License Included Oracle RDS offerings). With instance size flexibility, an RI may apply to any instances within the same instance family and database engine of a smaller size than what the RI configuration is for. For example, if you have one RI for a single db.m6g.xlarge and later switch to using a smaller db.m6g.large, the RI would cover two of the new, smaller instance types. The reverse is also true: Smaller instance reservations can partially cover larger instances, or multiple reservations can combine to cover larger instances entirely. This same flexibility also extends to the deployment type. If you have an RI that specifies a multi-AZ deployment and later switch two single-AZ deployments, the RI covers both as if you had purchased two single-AZ deployment RIs. The reverse is also true: If you have two single-AZ deployment RIs and later switch to a single multi-AZ deployment, the RIs are still used at 100%. You can see exactly how this would work in the RDS Reserved Instance normalization chart. Aside from instance size flexibility and deployment type flexibility, RDS RIs are inflexible with respect to region and the database engine. If either of these changes after purchase, the RIs no longer apply. One caveat applies to RDS RIs: No-upfront Reserved Instances are only available for a one-year term. To lock in the higher discount levels for multi-year commitments, you must pay at least some money upfront.  Amazon Redshift Reserved Nodes Redshift’s Reserved Nodes, as they’re technically called, are roughly equivalent to EC2’s Standard RI offering class in terms of flexibility: When you pick a specific instance type and region, you’re stuck with those choices. (We’ll refer to Reserved Nodes here as RIs, as they’re commonly called.) If you decide to change instance types or move regions, you have to purchase new RIs or pay on-demand rates, and the old RIs are left unused. Unique to Redshift, AWS has incentivized the migration to its newer RA3 node types by offering a Reserved Instance migration program in some regions. There are some constraints with this, however: The new cluster must be at least the same size and price when doing the migration, so you can’t switch to the RA3 nodes, downsize the cluster, and still retain your new RIs. Amazon DynamoDB Reserved Capacity DynamoDB Reserved Capacity works similarly to other AWS Reserved Instance offerings, providing a financial construct to save money by pre-committing to a certain level of usage. With DynamoDB Reserved Capacity, you can purchase capacity for a one-year or three-year term, and then pay via the only payment option: partial-upfront in exchange for discounts over on-demand pricing. Like other RI offerings, DynamoDB Reserved Capacity is a billing construct rather than a technical one, meaning it doesn't guarantee capacity availability but instead provides a discount on your usage. It's important to note that Reserved Capacity doesn't work with Global Tables or On-Demand DynamoDB tables, so you'll need to carefully consider your table configurations before making a purchase. As with other RI offerings, it's generally recommended to start with one-year term unless you have a very stable and predictable workload forecast. Note there's also an interesting caveat here: Reserved Capacity doesn't apply to tables with the Standard-Infrequent Access storage class. As a result, if you have a table that continually grows to the point where you may want to convert it at some point, you may wish to avoid purchasing reserved capacity for that table's consumption based upon your projected timeline. A similar caveat exists for DynamoDB's On-Demand billing model; if you suspect you may modify a table to use it, be forewarned when committing to usage. How Reserved Instances are applied A point of significant confusion for folks is that you can’t choose which instances an RI applies to. Many people think about RIs in terms of specific instances, then decide to buy an RI to cover that instance. However, that’s not at all how it actually works — and this mental model leads to some unexpected outcomes. The application of RI-covered workloads is automatic and follows a principle of "deepest discount first." In other words, if you have a handful of instance types running with different post-RI discount levels, then the RIs apply to workloads where the impact is the greatest until the RIs are exhausted.  For example, let’s say you have one c5.9xlarge instance running, and two otherwise-unused Reserved Instances that can apply to it. The first is a one-year no-upfront reserved instance providing 37% off of the on-demand price. The second is a three year no-upfront Reserved Instance that discounts on-demand by 58%. In this scenario, the second RI would apply for a 58% discount, while the first would go unused. It’s best to adjust your mental model and think about your workload in aggregate, rather than by specific instances.  In some organizations that do chargeback to their individual teams, this situation can often lead to intense discussion from those whose compute resources aren’t being covered. On one hand, they’re being held accountable for their spend, and that spend is higher than they intended. On the other hand, the organization is better off this way (after all, money is fungible because it’s all company money). The various resolutions to this particular problem are a topic for another day. Reporting and recommendations on AWS Reserved Instances AWS provides two primary tools to stay on top of your Reserved Instances: the coverage report and the utilization report. The coverage report is a measure of how much of your compute for a given service is covered by RIs. In the absence of further context that's specific to your environment, a reasonable starting goal is 80%. The utilization report is a measure of how much your RIs are being used. A good goal is 100%, though a case can be made for a few hours a day of overage depending upon the delta between the usage peaks and valleys. There are effectively four scenarios you could find yourself in: Low coverage, low utilization: You have no RIs or the wrong RIs. Your spend isn’t being covered by any RIs, even if you have some. Low coverage, high utilization: Not enough RIs. You’re under-committed. Buy more to increase coverage. High coverage, low utilization: Too many RIs. You’re overcommitted. High coverage, high utilization: Everything is great! What is the Reserved Instance Marketplace? The Reserved Instance Marketplace is a mechanism provided and managed by AWS to buy and sell EC2 RIs on a secondary market. This sounds great in theory, but it comes with a lot of caveats in practice. These caveats make the RI Marketplace unreliable in terms of buying what you want or selling RIs for a reasonable return. As a result, the marketplace should be viewed more as an escape hatch to fix a suddenly changed situation, rather than a bulwark of your RI purchasing strategy. More realistically, the RI Marketplace should just be ignored. It’s worth pointing out that "marketplace" evokes a sense of arbitrage, where you could resell sought-after RIs for a good price. However, because the retail rate of RIs is fixed by AWS, there is an upper bound preventing this sort of arbitrage. The best you can hope for is something less than the retail cost, which means you’re always going to lose some amount of money compared to what you paid. There's also a 12% fee charged on all RI sales. Let’s talk about some of the RI Marketplace limitations: Only EC2 Standard RIs can be sold in the RI Marketplace. EC2 Convertible RIs and RIs for all other services can’t be sold. You can’t sell RIs in any region that is disabled by default (as of 2024, the list is: Cape Town, Hong Kong, Hyderabad, Jakarta, Melbourne, Calgary, Zurich, Milan, Spain, Tel Aviv, UAE, Bahrain) as well as GovCloud East, GovCloud West, Beijing, and Ningxia. You must have a U.S. bank account. This means many non-U.S. companies can’t use the RI Marketplace. You cannot sell in the RI Marketplace if your account’s seller of record is in India ("AWS India Private Limited"), even if you have a U.S. bank account. You may only sell $50,000 in RIs or 5,000 RIs for the lifetime of your account, whichever you hit sooner. This specific point matters to large accounts: As per the AWS service terms, you cannot sell an RI on the marketplace if it was acquired at a discount (such as through private pricing discounts).  While it’s listed but before it’s sold, an RI continues to be yours and applies to resources as normal. After it’s sold, any resources that were covered by the RI revert to on-demand rates unless they’re covered by other existing RIs. You can cancel your listing at any time. As a result of these restrictions, it’s been our experience that the primary users of the RI Marketplace are third-party RI management products, which they use as a source of liquidity for the RIs they purchase. Using third-party Reserved Instance management products There exist a slew of third-party products that automate buying and selling RIs, but they have substantial limitations and potential risks for customers. What bugs us the most is that they purport to charge a percentage of the savings they create, but in many cases, they’re simply taking a cut of the savings AWS created and providing no additional savings beyond what would have been possible without them. Our opinion is that you shouldn’t bother using third-party RI management products as you can get the same result on your own for minimal effort. The value they add tends to be psychological far more than it is financial. While that’s not nothing, it significantly complicates what should be an internalized competency of your organization past a certain point of scale. Put more directly, the process of buying and managing reservations isn’t that complicated, and certainly not for the fees these products charge. #### The Duckbill Guide to AWS Savings Plans When you're trying to cut costs on your AWS bill, you're likely to come across AWS Savings Plans ("SPs").  We're sharing what we've learned about Savings Plans based on The Duckbill Group's extensive experience fixing our clients' horrifying AWS bills. In this article, you'll learn what Savings Plans are, how to pay for them, and how they're applied. We've also created a guide to AWS Reserved Instances ("RIs"), another common vehicle for committed use discounts. What are AWS Savings Plans? Launched in 2019, Savings Plans are a mechanism AWS provides to commit to using a certain amount of compute per hour in exchange for a discount on the instance cost. Savings Plans cover EC2, Lambda, and Fargate compute. (Separately, SageMaker has its own program, also called Savings Plans, that functions much the same as Compute Savings Plans.) One of the things many people new to AWS get confused about is that Savings Plans are not a reservation of capacity, but rather a billing construct. In fact, the best way to think about Savings Plans is actually by using Google Cloud’s nomenclature: a committed use discount. SPs are a financial construct, not a technical one. Savings Plans are a fundamentally different model from Reserved Instances. Instead of picking specific instance types or families, you pay for a certain amount of compute per hour. Importantly, you are committing to a certain amount per hour and paying for it regardless of whether you use it. Any amount under the commit is charged at the Savings Plans rate, while any amount over the commitment is charged at the on-demand rate. As you might imagine, determining the right commit number can involve some real work. Thankfully, as we’ll go into later in the article, AWS has several great tools to help. Savings Plans payment options All Savings Plans have three payment terms: all upfront, partial upfront, and no upfront. Payment options cannot be modified after you’ve made the purchase, even for no-upfront purchases. Payment options don't impact how Savings Plans are applied--they only impact the discount you're receiving and when you pay for commitment. All-upfront payment terms are exactly what they sound like: You pay for the entire SP upfront. This option provides the deepest discount, but in Duckbill’s opinion, the increased discount is generally not enough to be worthwhile, with two exceptions: If you work in an organization that needs to spend its money within a certain budget period, paying upfront can be a great way to consume the budget. Paying upfront is a great way to shore up a potential shortfall in meeting your AWS contract's Enterprise Discount Program (“EDP”) or Private Pricing Addendum ("PPA") commitments. Partial-upfront payment terms mean you pay just some of the total cost when you purchase your Savings Plans. There’s very little reason to use this unless you simply need to spend the cash but don’t want to spend as much now as an all-upfront payment. No-upfront payment terms are, again, exactly as they sound. Rather than paying any amount of cash upfront for the purchase, you pay for it monthly. The no-upfront payment option provides the least discount but, in Duckbill’s opinion, should be the default choice for every organization. Choosing no-upfront payment terms over all upfront or partial upfront comes down to how much you like cash in your bank account versus Amazon’s bank account. Savings Plans term length All Savings Plans have two term lengths available: one year and three years. The term length cannot be modified after you’ve made the purchase. Three-year terms receive the best discount, but as you might imagine, planning your infrastructure for three years out can be difficult for many organizations. While the delta between one-year and three-year terms can be substantial, Duckbill recommends defaulting to one-year terms unless you are confident in your workload forecast. How often to buy Savings Plans Savings Plans are designed to stack on each other, so having a few automatically works out. In other words, three active Savings Plans with commitments of $10/hour, $2/hour, and $25/hour will be treated as $37/hour's worth of coverage. We occasionally run across organizations that try to time their Savings Plans purchases with their private pricing contracts or purchase only once a year; these are suboptimal approaches. Instead, you should be buying Savings Plans as often as necessary to maintain coverage and utilization levels at your desired targets. We recommend evaluating on a quarterly basis whether true-ups are needed, though you might find that your growth dictates more frequent analysis. It’s not uncommon for an organization to have dozens or even hundreds of Savings Plans, all with different terms. No Savings Plans resale market Your unused Savings Plans can’t be resold. There's no equivalent to the Reserved Instance Marketplace for Savings Plans. The difference between Compute Savings Plans and EC2 Instance Savings Plans Savings Plans come in two offer types: Compute Savings Plans and EC2 Instance Savings Plans.  Compute Savings Plans cover three services: EC2 on-demand (spot instances very much do not count!), Fargate, and Lambda. All compute for these three combined, in any region, is covered by a Savings Plans commitment, without you having to do anything. This is one of the biggest advantages of Savings Plans over Reserved Instances: extremely low cost of management. EC2 Instance Savings Plans are far more constrained, requiring a choice of region and instance family, though you don’t need to select operating system, tenancy, availability zone, or even instance size. In exchange, you receive a deeper discount. It is Duckbill’s recommendation to default to using Compute Savings Plans and ignore EC2 Instance Savings Plans absent a specific, good reason to simply save on the headache of managing and modeling both. However, there are advanced strategies for mixing the two. Mixing Compute Savings Plans and EC2 Instance Savings Plans In time, as your confidence grows with regard to your workload stability, it can make some sense to mix EC2 Instance Savings Plans with your Compute Savings Plans to cover some baseline level of compute usage that isn't going anywhere anytime soon. If you know the instance family and region aren’t going to be changing, then purchasing EC2 Instance Savings Plans to cover that combination, then layering Compute Savings Plans on top for the remainder, can result in much deeper overall discounts. However, be careful! If there’s a communication breakdown between Engineering and whomever is planning Savings Plans purchasing about the longer-term architectural plan, it’s easy to inadvertently lock yourself into a situation where you’re facing significant financial penalties for evolving your architecture. That’s why we suggest using this approach sparingly. Flexibility has its own value, and ossifying your architecture to chase bigger discount percentages is very often misaligned with your organization’s best interests. How Savings Plans are applied One area that causes confusion for people is that you can’t choose which resources compute Savings Plans apply to. Many people think about Savings Plans in terms of specific instances, then decide to buy a Savings Plans commitment to cover that instance. However, that’s not at all how it actually works — and this mental model leads to some unexpected outcomes. The application of Savings Plans-covered workloads is automatic and follows a principle of "deepest discount first." In other words, if you have a handful of instance types running with different post-Savings Plans discount levels, then the Savings Plans apply to workloads where the impact will be the greatest until the hourly commitment is exhausted.  This does have some implications for your purchasing approach, as some instance types receive such low savings that they would rarely be covered by SPs. For example, instances running Windows with the license-included model receive the worst discounts of all. Because maintaining 100% SP coverage is impossible in any environment with a dynamic workload without incurring significant waste, that means you will always have some amount of compute that isn’t covered by SPs — but, crucially, you don’t get to pick which compute specifically. It’s best to adjust your mental model and think about your workload in aggregate, rather than by specific instances.  In some organizations that do chargeback to their individual teams, this situation can often lead to intense discussion from those whose compute resources aren’t being covered. On one hand, they’re being held accountable for their spend, and that spend is higher than they intended. On the other hand, the organization is better off this way (after all, money is fungible because it’s all company money). The various resolutions to this particular problem are a topic for another day. Combining Reserved Instances and Savings Plans You can combine RIs and SPs, and they apply sensibly in a "deepest discount first" interleaving. This is a common approach for organizations transitioning from EC2 Reserved Instances to Compute Savings Plans: Simply purchase Savings Plans as the EC2 RIs expire. Reporting and recommendations on AWS Savings Plans AWS provides two primary tools to stay on top of your Savings Plans: the coverage report and the utilization report. The coverage report is a measure of how much of your compute is covered by Savings Plans. A good goal is 80%-85%. The utilization report is a measure of how much of your Savings Plans are being used. A good goal is 100%. There are effectively four scenarios you could find yourself in: Low coverage, low utilization: You have no Savings Plans. Low coverage, high utilization: Not enough Savings Plans. You’re under-committed. Buy more to increase coverage. High coverage, low utilization: Too much Savings Plans. You’re overcommitted. High coverage, high utilization: Everything is great! Because Savings Plans are calculated on an hourly basis, highly elastic environments (in which the compute has wild swings hour by hour) might see large fluctuations in coverage and utilization. This is not necessarily a bad thing, but it is worthwhile to do the math. #### The Innovation–Optimization Continuum I often marvel at how, despite the comforting lies we tell ourselves, there’s nothing new under the sun. As an example, take The Duckbill Group’s business of optimizing cloud bills. Our spiritual predecessors are about as old as I am. In 1984, when AT&T (“Ma Bell”) was broken up into the baby bells, complexity rapidly spiraled, and in 1985, a company called Telenalysis started auditing corporate long distance bills. There was huge money there—just ask someone about long distance calling hours back then to get a taste of the complexity. We stand on the shoulders of giants, etc. This of course leads me to GenAI. GenAI Economics: History Repeats In today’s GenAI landscape, a whole bunch of the not-quite-leading models are advertised under the aegis of being economically advantaged. “Sure, it’s no Claude Sonnet, but Amazon Nova is quite competent,” says AWS, subtextually. They’re right! Nova is a fine model! But Claude is better, so I’ve never used Nova for anything beyond a couple of tests. Biasing for frontier models isn’t simply “Corey being weird,” it’s a broad trend. Contrary to what we hear from most companies in 2025, GenAI is still in a place of “will this add value to my workloads?” There aren’t (yet) many scaled-out, steady-state workloads that can be viewed through a lens of long-term planning. Folks are still seeing how and if this emerging technology fits, and approximately none of them is far enough along in that innovation exercise to begin earnestly focusing on improving their cloud economics. The exceptions are few and far between, because “does it work at all?” is where folks are investing their energies, rather than “how do we save money on this thing?”  There’s a Continuum Here’s something we’ve seen play out with remarkable consistency the entire time we’ve been in business. There’s a continuum, with “Innovation” on one end and “Efficiency” on the other, and any given company, project, initiative, or team gets to figure out where on that continuum they want to fall. It isn’t a one-way door; where someone lands on the continuum can be remarkably fluid. But at its heart, these two attitudes are fundamentally opposed. There is a time and a place for both attitudes here. Companies fall all over themselves to self-describe as “innovative,” but innovation for its own sake isn’t some kind of moral imperative. I’m writing this post inside of Google Docs, and keep getting interrupted by AI enhancements. I don’t want this brand of innovation; I want what the product was when it was at its best a few years back. Look at the modern iterations of Slack and Dropbox as two more examples; can anyone seriously claim those products are better at serving the customer needs today than they were in the past? At the same time, I’ve yet to see companies find success using the mindset of “I must build this new thing for the absolute least amount of money possible.” Look at SpaceX; I’ve lost track of how many boosters they’ve smashed into the ocean, the landing pads, and the droneships. Each one of those things costs millions upon millions of dollars. They’d not be where they are now if they hadn’t made that innovation investment. Even so, any engineering manager can attest that extreme innovation in the form of “letting engineers create” tends towards “fearsomely expensive.” Optimization is important, and I don’t really need to explain to you why “setting all the money on fire without regard to how you’ll afford to run the company tomorrow” is a bad thing. Finding Your Balance The pattern I like, and recommend to clients, is having periodic optimization passes through their environments. “Yes, you can go ahead and take chances; get messy; make mistakes!” And then it’s time to periodically clean up after yourself. Otherwise, you find yourself in untenable circumstances of “we have no idea what that cluster is doing, or has been doing since 2012, but it’s huge and we’re scared to touch it.” Maybe it’s a dev environment from someone who’s long since left the company—but maybe it’s your transactions database that somehow never got documented. Spend a few minutes tidying your house every day and you won’t have to spend all weekend cleaning up a disaster area.  The innovation-efficiency continuum isn’t about choosing one extreme or the other permanently. It’s about understanding where your organization needs to be at any given moment. For newer projects or emerging technologies like GenAI, leaning toward innovation makes sense: explore, experiment, and accept that efficiency won't be your primary concern—for now. For mature workloads or during economic tightening, shift toward efficiency without completely abandoning the ability to evolve. I’ve consistently observed that the most successful organizations don’t treat this as an either/or proposition but rather as a cyclical process. They innovate, then optimize, then innovate again. They recognize when it’s time to clean up technical debt and when it’s time to make a mess in the pursuit of something better. It’s important to remember that both innovation and optimization have their place. The trick isn’t choosing one and sticking to it—it’s knowing when to apply each approach, and having the organizational maturity to shift between them as circumstances demand. Innovation is expensive, but choosing efficiency too early is even more expensive. #### The OpenAI Polycule Moves In Together For fourteen years, I had a rotten little chihuahua named Ethel, who was a malevolent scheming weasel. I didn't realize it at the time, but learning to speak "malevolent scheming weasel" was great training for understanding the press releases that AWS puts out semi-regularly. Today, the OpenAI polycule put out a whole swath of press releases. OpenAI itself led the charge with four distinct press releases. Microsoft put out a surprisingly defensive press release that didn't say much. Softbank found $30 billion from somewhere and they announced that. Nvidia didn't announce anything, as they were unable to get to their keyboards due to all the money piling up in their office. And of course, Amazon had their own, which is what I'll unpack for you today. Let's Set the Stage OpenAI announced a $110 billion funding round at a $730 billion pre-money valuation, which for those keeping score at home values them at roughly the GDP of the Netherlands, only with somehow even more windmills at which to tilt. SoftBank and NVIDIA are each putting in $30 billion, while Amazon offered up $50 billion to capture the headline. (Well, $15 billion now and $35 billion later "when certain conditions are met," but we'll get to that.) Additional investors are "expected to join as the round progresses," which is fundraising-speak for "the round isn't closed yet but we wanted the press cycle today." Notably absent from the investor list: Microsoft, who instead published a statement insisting everything is fine with the energy of someone whose partner just moved in with a roommate named Amazon. In return for a pile of cash, Amazon gets three things: First, it becomes the "exclusive third-party cloud distribution provider" for OpenAI Frontier, OpenAI's enterprise agent platform. "Third-party" is the key qualifier -- it carves out Microsoft, which isn't third-party by virtue of its existing relationship. In fact, Microsoft takes pains to mention that Frontier will be hosted in Azure, despite it being hosted in Azure. (Note: this does suggest that without a specific carve-out, use of Frontier will not help you retire your AWS spend commitment, but "different rules" and all that, so who really knows. Check first before you rely on this!) So... will data for a Bedrock service reside in Azure? That's a hell of a caveat if so. The contortions everyone is going through here to respect the letter of an agreement between Microsoft and OpenAI while the relationship has very clearly soured are quite something. Second, OpenAI will build custom models for Amazon's own consumer-facing products, which is Amazon quietly admitting Nova needs reinforcements, presumably after listening to one of the customers who tried to use it as a frontier model. Great, something to shove more ads in front of customers as the customer experience of using Amazon continues to deteriorate. Third, and this is the big deal, Amazon and OpenAI are co-building a "Stateful Runtime Environment" that runs in Bedrock -- a persistent orchestration layer for agents with memory, tool access, identity management, and state that carries across sessions. And the way the companies talk about this is the hidden gem of this entire issue: we see where the contractual lines are drawn with respect to Microsoft and OpenAI. We now see in plain English (albeit with a slight ermine accent) that Microsoft has exclusive rights to stateless API hosting, which is why nobody but Azure hosts any of their models past their GPT-OSS open model. So to get around this, OpenAI and Amazon just invented a new product category called the "Stateful Runtime Environment" that, by definition, falls outside that exclusivity. This deal matters so much to Amazon that Andy Jassy basically trampled AWS CEO Matt Garman out of the way in his mad rush to the spotlight, as noted in today's CNBC appearance. To be fair, the orchestration problem is real, and this could be genuinely useful, but I sincerely doubt Amazon's ability to market it against customer problems that won't come across as, once again, belligerent and out of touch with reality. Microsoft's joint statement hammers the word "stateless" repeatedly, which tells you exactly where the contractual boundary was drawn. Meanwhile, any stateless API calls that the Stateful Runtime makes under the hood still route through Azure -- meaning Microsoft clips a coupon (and transits the data) on every transaction Amazon distributes. Amazon is paying (maybe) $50 (but definitely $15) billion to build a storefront where the back end runs on a competitor's infrastructure. Welcome, Amazon, to the experience that every one of your customers has. As Om Malik pointed out, Amazon is paying roughly 16x what Microsoft paid per percentage point of OpenAI, with none of Microsoft's exclusives. That's not just the cost of being late to AI, but rather the cost of showing up to the auction after OpenAI learned how to run one. As always, the devil is in the details: That $50 Billion Isn't Real The fine print makes it clear that Amazon is in fact giving them $15 billion, with "another $35 billion in the coming months when certain conditions are met." Those conditions have not been disclosed, so I'm going to assume they include Sam Altman serenading both Andy Jassy and Matt Garman on the re:Invent keynote stage. The bold print on the AWS press release says "OpenAI to consume 2 gigawatts of Trainium capacity through AWS infrastructure to support demand for Stateful Runtime Environment, Frontier, and other advanced workloads." I wonder how many terabyte pound-months that works out to. The press release also describes this as enabling enterprises to "consume intelligence on demand," which confirms that whatever brain worm infects AWS's prose stylists has consumed a fair bit of intelligence already, since the rest of us simply say "calling an API." OpenAI has historically been all-in on using NVIDIA GPUs, which makes sense; they work, have overwhelming ecosystem support, and are available in every cloud vendor. (Well, if you're OpenAI. If you're you or me there will be "capacity constraints" most places.) Trainium is untested at scale; the only notable public reference customers have been Anthropic (to whom Amazon has likewise thrown billions of dollars) and Apple (who give public endorsements about as frequently as Siri gives a useful answer on the first try). This suggests that either taking the Trainium capacity means OpenAI can then unlock other contractual benefits, or else they're so desperate for GPU-alikes that they're prepared to run their workloads on anything with a power cord. My sense is that both are likely true. NVIDIA's announcement mentions that they're providing 5GW of capacity to OpenAI as a part of today's announcement. That relegates the Trainium announcement to "hobby project," more or less. Heck, they talk about "Trainium4, which is due in 2027" so suddenly the "forward looking statements" disclaimer at the end that rivals the length of the rest of the announcement goes from "covering their bases" to "absolutely critical." A chunk of this commitment is for silicon which hasn't yet been fabricated. Remember the $38 Billion Commitment? "OpenAI and AWS are expanding their existing $38 billion multi-year agreement by $100 billion over 8 years" quoth the press release, and $138 billion in AWS commitment over a ten year span is... certainly something. Is that a hard commit, or an "up to" number? Is it "use it or lose it," or is it subject to ongoing adjustment and "$100 billion" plays nicely for the cameras because Amazon was tired of being the smallest hyperscaler commitment OpenAI had announced? What happens if Trainium4 underperforms ("wait until you see Trainium5!") and OpenAI wants to shift workloads back to NVIDIA? The press release is frustratingly silent on exit clauses. So Now What? Does the stateful runtime service that doesn't yet technically exist count against your AWS contractual commitments? How will OpenAI's models on Bedrock be priced relative to Anthropic given the Azure passthrough? What does this mean for customers choosing between Azure OpenAI and Bedrock Anthropic? Is the passing mention of "AgentCore integration" a sign that AgentCore is being built upon, or quietly replaced? And how does Anthropic feel about their Bedrock co-tenancy situation now that Amazon's spent six times as much on the new roommate? We're going to find out. #### The Post AWS's Billing Team Could Have Written But Didn't The idea of having multiple AWS accounts in your organization broken out not just by team but also by cost center isn't the terrible idea that some people (including me until a year or two ago!) tend to assume it is. Permit me to explain why. Imagine a mythical organization that has a single AWS account. (For purposes of this exercise, ignore the technical problems of account-wide rate limits, the operational hazards of "oops that was Production," the governance implications of a lack of a discrete audit trail, etc.) Everything's in one place, it gets charged to one bill, one person in Finance pays the bill (presumably, anyway; you'd be amazed how many Disaster Recovery plans overlook "the credit card stops working" as a potential risk factor). All is well. Your finance group is absolutely fine with this. They ask you what percentage of your infrastructure spend is for development work ("oh, about forty percent") and then take it as gospel for years. This is never revisited or questioned again. Years pass. The company scales. Your development spend is now roughly five percent of your infrastructure bill. Somehow this never gets reported through the four organizational layers between the engineer who builds things and the finance person who gets the bill. It's probably not that big of a deal, but one day an engineer reads this blog post and thinks to ask a question of the finance team. "Hey, what are you using the Prod vs Dev numbers for, anyway?" Finance will be thrilled that someone cares enough to ask them about their area. They'll happily give an answer that touches on a bunch of terms of art from the world of finance that bears no relevance to anything engineering ever touches. Internal P&Ls, pitch deck stuff numbers, calculating out the unit economic model… and then they drop in a statement that resembles "oh, and we're claiming a federal R&D tax credit." record scratch freeze frame "You're probably wondering why the CFO just pooped a literal abacus." Suddenly it's no longer just about how Finance thinks about things-- now there's a very real risk that the IRS would like a few words with you, and when they're done the SEC is just down the hall if you're publicly traded. Suddenly the API rate limits aren't really your driving concern anymore. Suddenly the CFO isn't the only one pooping abaci. Instantly "one AWS account" doesn't sound like the best idea anymore. So how do you break up your environment? An initial approach that makes an awful lot of sense is to split your environment a minimum of two accounts-- production, or as finance calls it "COGS (Cost of Goods Sold)", and pre-production, or "R&D (Research and Development)". Those readers following along at home may suddenly realize why a lot of large companies were thrilled to pieces by AWS recently allowing you to prevent RIs from being distributed to or from certain accounts in Organizations. This feels really dumb in terms of dollars and cents, until you factor in the cost of failing an audit-- at which point paying 30% more for a few instances isn't the worst thing in the world. It's tempting to stop here. It feels like you can get by just fine from two accounts. Except by this point, your organization has grown not just in size, but in complexity. Let's say you're on the ball and automatically apply cost allocation tags. "Instance X" gets a tag for whatever service it provides, whichever team owns it, what color tie the CEO wore the day it was spun up, etc. Now finance can happily allocate that instance's cost to various cost centers, and all is well-- except for the EBS volumes, the snapshots of those volumes, the data transfer the instance incurs, the load balancer tied to it, the CloudFront cost, the percentage of the AWS support fee, the Lambda functions that fire off incessantly because for some godforsaken reason we've collectively decided that cron jobs are way too understandable, etc. So you're now in an unpleasant reality where you're tagging ~20% of the cost, with the remainder being allocated to HEY LOOK OVER THERE IT'S A SQUIRREL IMPROPERLY DEPRECIATING AN ASSET! and hoping finance takes the bait so you can escape the conversation. It's not an easy migration process for most shops-- but if you have an AWS account that's just for the Sunrise Service or the Data Team or the Cornflower Blue tie, you can roll everything inside of it up to the cost center without crossing the streams. Mostly. Ultimately, an account per cost center makes an awful lot of sense for finance, except for the part where managing things cross-account still requires non-trivial amounts of work despite the presence of things like aws-vault. There is no API that exists between finance and engineering-- translating requirements back and forth can be exhausting. Finance has no idea what an ELB is, or why EBS is required even if you're storing no data on it-- and engineering doesn't know the best way to amortize an RI purchase, or whether GAAP permits the RI to be depreciated over that span. If you're having trouble determining how to best attribute or reduce your AWS spend, win hearts and minds across the finance / engineering divide, or ultimately just want to chat about the Cloud, we should talk. This is exactly what I do for a living. #### The Secret Ingredient Duckbill Looks for in Employees I hate writing job descriptions. I always feel like I’m going to inadvertently write a job description for a purple squirrel.  I know who we are as a company; that part’s easy to write. But how much do I know about what we’re looking for in a candidate?  Are we basing the job description off the actual work we’re doing or off what we think we need from a candidate? Is that “3 to 5 years experience” really a hard requirement? Are we attracting (and comparing!) candidates based on skill set, professional values, or some combination of the two? When it came time for our Client Services team to hire additional team members, I added a short, three-word description under the “Nice-to-have” section (“Community builder experience”) with the comment “I'm not quite sure how to word this but I think this is an important quality to include.”  Since this description can mean just about anything, someone responded, “Can you explain a bit more about why you think it's valuable for us to be looking for this skillset? What about it makes a Cloud Economist more likely to succeed here?” It took me almost two months to answer this response. I couldn’t quite articulate the thinking behind that phrase, and the question sat unanswered as my brain churned and churned trying to find the words to explain what I meant. That’s when I stumbled upon Simon Sinek. And suddenly, I had the words to explain what I meant. People don’t buy what you do, they buy why you do it In his TED talk “How Great Leaders Inspire Action”, Simon talks about the Golden Circle and the neuroscience behind the importance of knowing why you do something.  In short, there are three concentric circles in the Golden Circle: the why is the purpose that drives everything the organization does, the how is the principles that guide the work, and the what is the services offered by the organization.  Credit: Simon Sinek High-performing organizations, he explains, market themselves based on their why more so than their how or what because, as he puts it, “people don’t buy what you do, they buy why you do it.” Most organizations have a why in the form of company values. But how often do they leverage that information publicly in their marketing and branding? Heck, how often do they leverage that information internally in business decisions like product development and cultural norms?  I’ve worked for countless organizations large and small that plastered their company values all over every wall. But I couldn’t tell you how those values dictated the flow of ideas and work that created their products. That’s what leads to all these hollow-sounding lists of corporate values we’re all oh-so-familiar with. Protecting our clients To me, that’s one of the things that’s so critical about The Duckbill Group: We have a clear why and we talk about it every chance we get. We protect our clients. Our CEO has even written an article about the origin of this company value.  To me, our why is even deeper than that. In a nutshell, our why is to help others succeed and leave them in a better place than when we first met them (a la the Boy Scouts rule). Cloud cost management (our what) is a complex, ever-changing journey, and we want to do right by our clients by guiding them through the process. And we ultimately want people to join The Duckbill Group because they believe as we believe. We want to work with people who care about other people Which brings me back to my three-word suggestion in our job description.  I see a lot of parallels between our why and folks who are active in their professional communities—what I’m calling “community builders” in this context. I’m talking about the people who share their wealth of knowledge to bring people together and strengthen their community.  But even those words don’t feel quite right. They don’t quite capture what I mean. Other single-word descriptions came to mind, but what ultimately kept jumping out at me is this: We want to work with people who care about others. There are many ways this manifests itself, including but definitely not limited to the behaviors I’ve seen in community builders—like sharing knowledge at conferences, writing blog posts or books, and more.  But no matter how it manifests, it’s the underlying desire to help others that’s so important to us at The Duckbill Group. I want that kind of person on my team. Or at the very least, I want to help them help others.  The more we can support each other, the stronger our community becomes. And with a community by our side, we can do anything. Interested in working with The Duckbill Group, helping our clients lower their ridiculous AWS bills, and building a stronger community at the same time? Check out our career opportunities. #### Understanding Data Transfer in AWS The answer to "how much does it cost to move a gigabyte of data in AWS?" is, as it turns out, rather complex. We dove deep into AWS documentation (finding a bug here and there, even!) and hounded an untold number of AWS employees to come up with all the different prices and cost paths. Wowzers... That's complex! Click to take a closer look. #### What I Don't See From AWS Support Today Randall Hunt posted an "emotional rant about AWS support" chronicling what he's seen from them over the past ten years, and invited others to chime in with their experiences. Given that I'm arguably one of the four most sarcastic observers of AWS in the world, I figured I'd take him up on that invitation in a more permanent form than a tweetstorm. My own AWS account is tiny; I don't have a paid support plan for it, so my interactions with AWS support in that context are largely limited to asking for obscene service limit increases for ridiculous reasons. That said, I've been a consultant for a long time-- and have been involved in a number of incidents over the ten years that AWS Support has been around for. It's in that spirit that I'd like to talk about what I've never seen from AWS support. I've never seen AWS support express the exasperation that they must surely feel when confronted with customers who refuse to believe that their issue isn't AWS's fault. I've never seen an AWS support person lose their cool when a customer was melting down and screaming at them in the midst of a giant outage. I've never seen AWS support imply that a customer was stupid-- even when I was speaking with them on a customer's behalf, and I was most definitely being stupid. (And belligerent-- I was authoritatively wrong.) I've never seen AWS support throw an AWS product team under the bus that I at times argue that they richly deserve. I've never seen AWS support pass the buck to a third party-- in fact, just the opposite. I sat on a conference call with a frontline support person, the customer, and the customer's ISP-- and watched as the support rep politely took the ISP's technician to Networking School, without ever blaming the ISP directly. I've never seen AWS tell a customer that they wouldn't help them. And I've never seen AWS support end an interaction with a customer without double-checking that everything was okay. Support is inherently the front line-- they're there to do triage, to diagnose, to repair, and to escalate. They're not a product team; they can't fix any feature shortcomings or bugs that they discover. They get an awful lot of crap from an awful lot of directions-- but they are and remain one of the best things about AWS. Here's to another ten years. #### What the hell is a vCPU-based on-demand service limit? AWS recently sent an email to its customers about vCPU-based on-demand instance limits becoming available. As it came from AWS Marketing (motto: "We're willing to be misunderstood for long periods of time!"), the messaging was confused and unclear.  After the third customer asked me to explain the email to them, it was time to write this blog post.  Before this change, EC2 service limits were based on instance type and instance family. Once upon a time, this was reasonable. It served to keep someone from accidentally billing themselves a phone number due to a misplaced zero or six. This means that, right now, there are a hilarious 261 service limits per region for EC2 alone—all because the EC2 team has no sense of "that's enough" when it comes to launching new instance sizes, families, and hues.  Now, these limits are dramatically simplifying, and this is a good thing. Soon, there will only be five: One for the instances that human beings use: A, C, D, H, I, M, R, T, and Z.One each (four in total) for F, G, P, and X—each of which are ridiculous horseshit instances that virtually nobody should use under any circumstances (read as: these families consist of premium-flavored snake oil for machine learning purposes). These new limits are all tied to number of vCPUs and start with an on-demand baseline of 1152 vCPUs for the nine instances people actually use. That could be 1152 t3.nano instances, 12 r5d.24xlarge instances, or any combination of instances until you reach the limit.  At no point will this change drop limits below your current instance usage.  I can see why AWS is telegraphing this change before it happens with a full salvo of email alerts. I just wish the messaging had been a lot clearer.  #### What To Do When You’re Underwater on Your AWS EDP It’s uncommon, but some companies find themselves in the position of missing their EDP/PPA commitments—sometimes by substantial amounts. A classic scenario: You signed an agreement to spend $150 million over three years, fully expecting to blow past that with all your growth—but now your past spend and projections are showing you’ll miss that by a healthy enough margin to cause concern. The Zen of AWS Cost Management The first thing I tell everyone in this situation: chill, dude. This is not an emergency. It’s not even necessarily a problem. Engineers in particular (speaking as one myself!) have a visceral reaction to paying for something that isn’t getting used. After all, we’re trained to optimize systems and eliminate waste. Paying for nothing fundamentally conflicts with our values.  But consider an alternative perspective: you already decided to spend this money, years ago. It was within the budget and the forecasts supported it. That decision is in the past. You bought an option and didn't exercise it. Your focus now needs to be on the total financial picture—including all the discounts you've already received and will continue to receive—rather than the emotional reaction to “paying for nothing.” Fortunately, you have options. What You Can Do If you’re facing a shortfall, there are a few courses of action you can take to close the gap. Make Some Upfront SP/RI Purchases Both the upfront payments and ongoing usage charges from Savings Plans and Reserved Instances purchases can count against your EDP spend commitment. If you have the cash, stock up on savings by prepaying for compute you know you’ll use. Of course, this option has gotten less attractive post-ZIRP with interest rates on the rise—it may be that the cash is more valuable in your bank account than the savings you’ll get from the upfront SP/RI purchase. Consume More Services Don’t do this. It’s true that one way to avoid paying for something you don’t use is to find a way to use up what you paid for, but this leads to terrible decisions like: Spinning up resources you don’t need Moving workloads to AWS that don’t belong there Building unnecessary projects Overprovisioning existing infrastructure You’re just making the problem worse. You’re wasting resources, wasting engineering time, and setting yourself up for a worse negotiating position in the future—more on that last one in a bit. Move Some of Your Vendors into Marketplace With some restrictions, purchases you make on the AWS Marketplace count toward retiring your EDP spend commitment. For most agreements you’re been able to make a dollar-for-dollar match to retire up to 25% of your commitment. Professional services are excluded, and, earlier this year, it became a condition that the service you purchase must itself run entirely on AWS. Even so, it’s still possible to meaningfully chip away at your balance. Renegotiate You can go back to Amazon to renegotiate your agreement in some circumstances. If they're up for it, here’s what that looks like: AWS will offer to create a new multi-year agreement for you, starting now This new contract (typically 3–5 years) supersedes your remaining commitment period New commitment levels are based on your current actual spend (your “run rate”) Discount percentages will be lower than your original contract for two reasons: They'd never admit to it, but there's an implicit penalty for missing your original commitments Lower spending/usage tiers qualify for smaller discounts Do Nothing Believe it or not, this sometimes is a legitimate intentional strategy based on some simple arithmetic. Suppose you purchased season tickets for your favorite baseball team at a 50% discount off the individual game prices. Even if you miss five games out of 20, it still leaves you with a net saving compared to buying 15 individual tickets. You don’t need to go to all the games to come out ahead. Cloud contracts work similarly. You can still come out ahead when paying shortfall fees thanks to this bit of your agreement: shortfall fees for service discounts are calculated at the discount rate. Instead of baseball tickets: DTAZ is $0.01/GB, but you commit to 500 GB at $0.005/GB, a 50% discount off retail You consume only 350 GB Your shortfall fee is 150GB × $0.005/GB = $0.75 All told, you’ve spent $2.50 (350 × $0.005 + $0.75) instead of $3.50, an effective discount of 29%. Yeah, it's not the 50% discount you signed up for, but it's still not bad. Look at the entire commitment timeline and assess your complete outcome, not just the problem periods. If you’re in year 2 and missing the commitment, but project growth in years 3–5, the total value over the contract’s life might still be positive. In fact, we have some clients who intentionally make aggressive commitments knowing they'll miss them sometimes but the discount will more than make up for it. By committing to a higher usage than they actually expect, they can lock in deeper discounts on the actual usage. (Caveat: this is advanced mode–don't even think about it unless you’e spending $100m+/yr with major growth ahead!) Your Decision Tree Here’s a simple decision tree we encourage our clients to follow: 10–15% Underwater: Do Nothing Since your effective discount rate will be cheaper than the retail rate—even if you left a bit on the table—this is actually a win. Seriously, unless you have a crystal ball to do perfect forecasting, this is probably the best cost outcome. Focus on other stuff. 20–30% Underwater: Do Nothing (Probably) If you go hat-in-hand to AWS for this level of overcommitment, the results will depend on how the tea leaves were read that morning. You might be able to kick off a re-negotiation, but you also could be told to pound sand. Honestly, the shortfall is likely not enough to make renegotiation worth it. Suck it up, re-evaluate the inputs that led to your initial decision, and chalk up the moderate loss to the price of learning. 50% Underwater: Renegotiate If you’re this far underwater, this is where I’d recommend renegotiating your contract. You’re now paying too much in shortfall fees. The first step might feel counterintuitive: go deeper underwater by cutting costs further. It’s time to get aggressive about optimizing costs. Aggressively optimizing costs before renegotiation lowers your baseline for the new contract. AWS will still offer worse discount terms, but your new lower baseline means you’ll come out ahead in total dollars saved.  Suppose your original commitment was $50 million/year and you’re spending only $25 million. A renegotiation would put your new baseline at $25 million/year plus annual growth. But since the best savings is the money you don’t spend, it’s worth your time to reduce your spend to $15 million through optimization efforts, first, to set an even lower baseline. It’s true that your new discounts will be way less interesting than your previous ones, but you’re spending less money overall. Don’t Go It Alone If you wouldn’t give yourself a haircut, you shouldn’t take this on by yourself. The world’s best procurement managers, CFOs, and finance teams rely on AWS experts to model their forecasts, understand how their architecture impacts their bill, and distinguish good contracts from bad—and so can you. Take a breath, acknowledge that the universe continues to expand, and get in touch. #### What's the Difference Between an EDP and a PPA? What's the Difference Between an EDP and a PPA? Tl;dr Nothing, they’re functionally the same thing. We do a lot of AWS contract negotiation at The Duckbill Group, and confusion about discount types seems to pop up during every engagement.  The difference between an Enterprise Discount Program (“EDP”) vs. a Private Pricing Addendum (“PPA”) isn't self-evident. In fact, even other sources online trying to answer the question still get it wrong. So let’s correct the misunderstanding and confusion. The history of EDPs and PPAs Before explaining the difference between the EDPs and PPAs, it’s worth making a distinction about discount types. There are three core commercials in an AWS contract: cross-service discounts, service-specific discounts, and credits. Once upon a time, the EDP was the only contractual vehicle to get a cross-service discount, while the PPA provided service-specific discounts, as an addendum to the EDP contract.  For the past few years, the EDP and PPA programs have become functionally the same thing for customers. Some smaller contracts will still say EDP on them while larger contracts are under the PPA structure. For customer purposes, this is just a leaky abstraction: Non-useful internal details are being exposed to the customer and creating confusion.  What's in AWS contracts today Every contract you see these days says “Private Pricing” or “PPA” on it. What that means is that it’s time to update your own language: “We have an EDP discount of 12%.” → “We have a cross-service discount of 12%.” “Which PPAs are we negotiating for?” → “Which service-specific discounts are we negotiating for?" Rest assured that if your previous contract said EDP and your new one says PPA, you haven’t missed anything. It’s just a new name and nothing more. #### Why Benchmarks Miss The Mark for AWS Spend A question we get asked regularly at The Duckbill Group is whether we can share benchmarks around what other organizations spend on AWS. The question takes several forms, such as: “What is a typical industry cost per transaction?” “How much should I be spending on $service?” “What’s the right amount of idle CPU across a fleet of systems?” “What do other companies spend on prod vs. non-prod?” These are well-intentioned yet ultimately misguided questions. The first issue is that external benchmarks tell you what the mediocre companies do, not what the best of them do. If your goal is to be a competitive leader, you'll need to blaze your own trail. We don't provide benchmarks because of the second issue: They just aren’t that useful and don’t make sense without context. After all, your company has a unique product design and competitive business strategy, and your spend should reflect that. The idea of sharing a "typical industry cost per transaction" is farcical, because there's no such thing as "typical" once you reach the level of individual businesses. Benchmarks ignore product design Let’s take, for example, a company whose product spends a huge amount of time crunching numbers of some sort. Assume it has done optimizations to really pack that compute density. Its typical idle CPU is going to be very low — which makes sense because it's running a CPU-bound workload and it's already made efforts to improve the density of its compute. Taking that idle CPU benchmark to a company whose product is memory-bound doesn’t make any sense. Even choosing instance types that are heavier in memory capacity will still result in high memory utilization and low CPU. Many workloads are not strictly CPU-bound; plenty are constrained by memory, IO, or network. But even taking into account all of these, a benchmark from some other company still doesn’t work for yours because the workloads aren’t the same. That company builds different products than you and in different ways. Even if you were direct competitors with a similar product, you made decisions on trade-offs that it made differently. This holds true even if we were to look for averages across multiple companies of similar products–-there’s just too many different choices made to result in anything meaningful for you. The same goes for decisions around how much you spend on certain services — your product design dictates that, and you won’t find meaningful answers by looking outside. Benchmarks ignore competitive decisions While we’re on the topic of trade-offs and decision-making, let’s talk about prod vs. non-prod spend. Non-prod spend encompasses dev environments, QA, and other environments that aren’t production. The distinction matters for finance purposes: Production gets categorized under cost of goods sold (COGS) while non-prod spend is under research and development (R&D). These are two very different buckets for the finance team. Asking what another organization spends on R&D to make a decision about whether you’re spending the right amount is just as misguided as the resource constraint example above. These are competitive decisions! Do you really want your competitors dictating how much you should spend on R&D? I’ve known many companies that go all-in on R&D spend for years at a time. I’ve known companies that spend years at the other end of the spectrum, simply trying to maintain what they’ve got without innovating further. Some companies go back and forth. I’m not one to judge whether any of them were right or wrong — that’s for their executives and the market at large to decide. Certainly, relying on external industry benchmarks isn’t a healthy approach. Who cares whether your cost per transaction is a penny more than your direct competitor? Maybe you’ve got a better product! Who cares that you’re spending 5% of revenue on R&D? Maybe your product is in such a mature position that increasing it wouldn’t move the needle. Maybe 5% is exactly the right number for your company. These are inherently internal decisions guided by strategy. Don’t let your competitors dictate where and how you invest. Compete against yourself, not others We’ve helped a number of organizations develop KPIs to track how they’re doing. A great many of those KPIs are highly specific to the company, measuring the things that are unique to their particular environment and the things they care about. Among our most top-performing clients, one of the things that stands out is that the best of them are competing against themselves, not against some externally set benchmark.  After all, by the time it becomes an external benchmark, all of your competitors were there already. #### Why Cloud Finance Is Broken and Ineffective When Corey Quinn and I first started The Duckbill Group in 2019, I was expecting we'd be advising organizations on complicated Reserved Instance purchases and the like. As we worked with more and more organizations on their horrifying AWS bills, I came to find that we spent almost all of our time (and still do!) advising Engineering teams on architectural improvements in pursuit of cost optimization. In fact, Reserved Instances and Savings Plans don't even factor into our analysis until the very end of our AWS Cost Optimization engagements. That realization has since become a core thesis for our consulting at The Duckbill Group: Cost management is primarily an engineering problem, not a financial problem. This fundamental misunderstanding leads to organizations building ineffective cloud finance teams. By understanding how this goes wrong, you can build a much more effective cloud finance team at your company. The unexpected reality: Architectural choices drive cost Having seen oh-so-many AWS bills and consulted with dozens upon dozens of organizations, both staggeringly large and super tiny, it's become clear to us that architectural choices are the primary driver of cloud costs. Not those Elastic IPs you don't use anymore. Not those instances you forgot to turn off. Not your developer environments. It's your architecture. Your production environments should generally account for the vast majority of your cost, but within production, it's the choices you make that drive that cost, not unused resources inside production. Through that lens, the real path to optimizing cost begins to appear: Figure out the specific technical drivers of cost inside your architecture, pinpoint them, and then come up with ways to improve the cost of operating them. What I mean by figuring out drivers of cost is more than just saying, “Oh, we use a lot of RDS.” I mean intimately understanding how data flows through the environment. What, exactly, causes costs to fluctuate? If you were to increase the workload of a given subset of your environment, what would happen to costs? If user traffic dropped by 20% tomorrow, what specifically would happen to your costs? Understanding these cost drivers allows you to pinpoint where the cost optimizations are. Any system that charges based on metered usage naturally leads to this situation of needing to intimately understand cost drivers and behaviors. To optimize the costs of that metered-usage system, you have to change how the system is used. And thus, architecture and costs are the same problem. Most of your cost management efforts are ineffective The misunderstanding of the nature of the cost management problem has led, in our experience, to massively ineffective cost management efforts within many organizations. Many of the self-reported "cost-mature" organizations that have come to us tend to have a dedicated person or team for managing their AWS costs. That's great! We recommend that. But when we dig into those day-to-day role responsibilities, we usually find that they're centered on largely ineffective tasks: managing coverage and utilization of Reserved Instances and Savings Plans (RIs/SPs), and chasing down idle resources. Both of those things are obvious tasks to do in all cost management efforts. It's where almost all of the cost optimization SaaS tooling vendors focus their products because the tasks can largely be automated, thus, they're a great opportunity for software-driven solutions. These tasks are useful and valuable in some cases, but they're not the most important work to be done. RIs/SPs should be the last consideration in cost management efforts. When you purchase RIs/SPs, you're fundamentally saying that you agree that what is currently running in the environment is the right stuff to be running, both in terms of number of resources and configuration of them. You can't do that at the start of your efforts. And as for idle resources, yeah, sure, that's a thing but ... it's really not that big of a problem. We had a client many moons ago that was adamant they needed a robust solution to handle the runaway costs in their developer environments due to lots of constantly idle resources. We started discussing various models of solutions, such as automatic deprovisioning after-hours, before thinking to check their total spend on development environments. It was less than 1% of their bill! We advised them to just let the idle resources run and stop worrying about it. The concern of idle resources driving costs is usually more fear than reality. And before you come at me: Yes, there will always be an exception where someone saved a ton of money by turning stuff off. That's great, but it's not the common case. Managing RIs/SPs and idle resources are both worthwhile activities in the right context, but that context isn't generally day-to-day operations. Engineering isn't paying attention the way you think The main trouble I have with how most cost management efforts are structured is that they assume Engineering has made all the right choices to build cost-aware and cost-effective systems. As any capable engineer will attest, this is a very bad assumption. It's not that engineers are willfully wasting money. Your organization employs engineers to build functionality for your customers and create value for them. The quicker they can do that, the sooner the customer gets value. That's exactly what your business wants them doing. Optimizing costs is a later-stage task (read: afterthought) for engineers, because creating customer value always comes first. Sometimes that later stage never arrives simply because your engineers are focused on the neverending backlog of work that creates customer value.  This is a good problem to have. An expensive one, perhaps — but a good one.  That leads me to my next point. You're improperly staffing cost management roles Many organizations hire people with a finance background to staff cost management roles. Please stop doing this — you're making the problem worse. On one hand, you probably have better reports now. But on the other hand, as established earlier, Finance can't solve an Engineering problem. You need an engineer in that role. While you’ve got better reports, you’re no better off in terms of actual understanding and insight into your cloud spend. Finance, given a lack of Engineering knowledge and no context provided by Engineering, is going to base decisions on the assumption that Engineering is doing the right things when it comes to AWS architecture and costs. That's, uhh, not a bet I'd take. Don't get me wrong here: Finance has a crucial role in managing your AWS spend. It's just not in cutting costs, because that's not the best use of their expertise. (Let Finance focus on contract negotiation, unit economics, and forecasting). If you're building an AWS cost management function, a Finance person is not the first hire I'd make. Cost management SaaS tools are only a partial solution One of the more common behaviors we’ve run across is organizations’ propensity to just rub some software on it. They roll out a product such as Cloudability or CloudHealth and believe that their cloud costs are all taken care of. As it turns out, beating people over the head with pretty dashboards is a lot easier than changing the company culture so they care about costs.  One of my favorite things to do is ask how many active users the products have — it’s never more than four or five people, no matter the size of the organization. Software is only part of the solution to cloud finance, and it’s not even the most impactful part. Going down this path risks putting the cloud finance function in a precious position: They have all of the responsibility for managing costs but none of the leverage needed. It results in a constant battle against engineering teams that don’t understand why they have to care about cost management so much, particularly if it’s not part of their compensation or promotion structure. Fixing cloud finance means rethinking cost management Cloud finance is still a new area of work for the industry, so it’s understandable that many companies struggle to build an effective practice of it. The key to fixing broken, ineffective cloud finance teams is realizing that cost management is primarily an engineering problem. To build an effective cloud finance team, your organization should: Rethink what you think you know about cloud spending. Operate from a new assumption: Architecture and costs are the same thing.Work to better understand how your architecture impacts your costs and how your specific cost drivers behave. The majority of your efforts should be on understanding cost drivers instead of RI/SP management and identifying idle resources.Build processes into your engineering release cycles for ongoing cost optimization efforts. A little bit of time spent every week will have much better results than a lot of effort every quarter.Staff your cloud finance efforts with engineering, supported by finance — not the other way around.Acknowledge that your tools aren’t the complete solution. Tools are not a replacement for people; tools augment people. Being more aware of the failure modes I’ve laid out and what to do instead will, hopefully, allow your company to improve how it manages cloud costs. #### You Need a Multi-Cloud Dashboard Because I Want To Sell You One If I called you and told you that your gasoline budget should be on the same dashboard as your movie ticket budget, you'd look at me like I just laid an egg.  Why, then, do so many companies insist you need to view your GCP, Azure, and AWS spend in the same dashboard? Different cloud providers run different workloads for your business (or at least they damn well should!) because multi-cloud for a given workload is invariably the wrong move.  I go into this in some depth elsewhere. But today’s post is less about the poor strategic decision itself than it is the idea of having a dashboard to centralize all of your public cloud spend in the same context.  When you’re using multiple providers, the billing dimensions are different, the confusion levels are similar, and there's no direct comparison that makes sense at a resource level. CPU and RAM options alone vary to a point where it’s very hard to get 1:1 equivalence between instances (or virtual machines) across providers, so you’re rounding. Invariably, you’re rounding badly. Your CFO doesn't care what you're spending on container orchestration. They care how big the checks you’re writing to AWS and Azure are, how much of those checks are for development versus production, and what those numbers are going to look like 18 months from now.  The comparisons that can be exposed and shown via API aren't the things that are relevant to a business context—and business context is something that cloud visibility dashboards are completely lacking.  To wit, everyone’s got a metric for 500 errors. But nobody’s got a metric for how happy their users are. It turns out there’s no API for business insight.  My perhaps overly cynical take is that the companies selling these multi-cloud solutions have a vested interest in customers making suboptimal cloud decisions.  If you go all in on a single cloud provider, the right dashboard answer is either "the one they provide for your use" or "picking up the phone and yelling at them until the dashboard they provide for your use meets your needs." It's not cutting large checks (or worse, a percentage of your cloud bill!) to third-party companies that are selling a solution for a problem you almost certainly don't have.  What’s worse is that as native dashboards continue to improve, third-party vendors become actively incentivised to encourage and promote multi-cloud strategies that actually  harm their customers.  I remain steadfast in my opinion that a big driver behind multi-cloud discussions is a sea of vendors who depend on you making a poor decision. Otherwise, they have nothing to sell you.  ### Pages #### About About Duckbill Duckbill is built for the compute economy. We provide the financial control layer that AI-native companies use to negotiate, manage, and forecast their spend across every cloud, model, neocloud, and inference provider. We're a technology company with deep financial expertise on infrastructure, not a financial firm with a technology bolt-on. Informed by tens of billions in negotiated AI and cloud contracts, our software and expertise work in concert to help organizations manage their largest and fastest-growing areas of strategic spend. We bridge the gap between technical complexity and financial reality: consolidating scattered data across providers, providing expert validation that stands up to board-level scrutiny, and providing you systems to model future scenarios so you can catch opportunities and risks in time. Whether you're sizing a nine-figure commitment or planning next year's budget, we give you the context and confidence to keep your largest and fastest-growing cost under control. In the news Team Brand Assets Download and use the logo assets below. Please do not modify or change the resources. Download assets #### AI Addendum AI Addendum Last Updated: April 1, 2026 This AI Addendum applies to the Product to the extent that it includes AI Systems and/or provides AI Services (as defined below). Capitalized terms used but not defined herein shall have the meanings given to such terms in the terms applicable to Customer’s use of the Product. Definitions. For purposes of this AI Addendum, capitalized terms have the following meanings: “AI Services” means the artificial intelligence or machine learning components of the Product, including the AI System and underlying Model(s).  “AI System” means a system, software, or process using machine learning, statistics, or other artificial intelligence techniques, excluding passive computing infrastructure, that uses computation to generate outputs such as content, predictions, recommendations, or decisions that can influence the environments it interacts with, excluding the underlying Models.  “Customer Inputs” means data, information, or materials submitted by or on behalf of Customer or Users to the Product, but excludes Feedback. “Customer Outputs” means reports, materials, data, and other outputs generated or produced by the Product and made available to Customer that are based on Customer Inputs.  “Model” means a large language, machine learning, or artificial intelligence model.  “Train” or “Trained” means the use of data, information, or materials to create or improve a Model. “Training Data” means Customer Content, Usage Data, and Feedback. AI Services.  The AI Services are part of the Product and subject to the Terms as supplemented by this AI Addendum. Customer may use the AI Services by providing Customer Inputs. The AI Services may generate Customer Outputs in response to Customer Inputs. Provider may copy, display, modify, distribute, and use Customer Inputs to the extent necessary to provide the AI Services as contemplated by this AI Addendum. Customer authorizes Provider to process Customer Inputs for all such purposes.  In addition to Provider’s other rights and obligations under the Terms, Provider may copy, modify, distribute, and use Training Data to Train existing Model(s) in the Product, subject to the following restrictions: Training Data must be aggregated and de-identified before it is used as permitted in this AI Addendum;  Provider may not Train new Model(s) built by Provider after the  Effective Date using Training Data, without the prior written consent of Customer;  All information, systems, services, and data created or collected using the Training Data shall be exclusively stored and used in the Product and shall not be shared with any third parties except as part of the Product; and  All Models that are Trained using Training Data shall either be stored locally by Provider or have a zero data retention agreement with Provider.  Provider may use Training Data to provide, maintain, develop, and improve the AI Services, provided that such usage does not constitute Training except to the extent expressly authorized herein.  Provider must contractually bind all subcontractors used in the development, training, testing, deployment and/or monitoring of the AI Services to terms no less restrictive than those in this AI Addendum. Customer may request that Provider does not Train using Training Data as set forth in this Section 2 by emailing support@duckbillhq.com.   Intellectual Property and Privacy.  As between the parties, Customer (a) retains all right, title, and interest in and to all Customer Inputs, and (b) owns all Customer Outputs. To the extent permitted by Applicable Laws, Provider hereby assigns to Customer all right, title, and interest in and to Customer Outputs, if any. Nothing in this AI Addendum will reduce or limit Provider’s obligations under Applicable Data Protection Laws regarding Personal Data that may be contained in Customer Inputs.  Customer represents and warrants that it, all Users, and anyone submitting Customer Inputs each have and will continue to have all rights necessary to submit such Customer Inputs to the AI Services. Representations and Warranties for the AI Services. With respect to the AI Services and Provider’s obligations under this AI Addendum, Provider represents and warrants that as of the Effective Date and throughout the Term of the Terms: AI Systems shall have been tested and will perform in all material respects according to the specifications of the Terms and the applicable documentation; Provider shall comply with all Applicable Laws governing the development, deployment, and use of its AI Services; and Provider shall employ proactive processes to minimize reasonably foreseeable harms that may arise from Customer’s deployment of the AI Services, including, but not limited to, effective feedback mechanisms, which, at a minimum, facilitate the reporting of any system abuse, unanticipated harms, and/or instances of incorrect or inconsistent outputs of the AI Services.  Disclaimers.  Due to the nature of artificial intelligence and machine learning, information generated by the AI Services may be incorrect or inaccurate. The AI Services are not human and are not a substitute for human oversight. Customer Outputs generated by the AI Services may not be protectable as intellectual property. Customer Outputs may resemble or be duplicative of data, information, and materials created by the AI Services for others. Provider does not provide any representation or warranty that any Customer Outputs (a) do not and will not incorporate or reflect the data, information, prompts, or materials of others, (b) will not violate, misappropriate, or otherwise infringe upon the intellectual property or other proprietary rights of another person or entity, or (c) will not be reproduced in the same or similar way to another user of the AI Services. It is Customer’s responsibility to evaluate whether Outputs are appropriate for Customer’s use case, including where human review is appropriate, before using or sharing Outputs. Customer acknowledges, and must notify its Users, that factual assertions in Outputs should not be relied upon without independently checking their accuracy, as they may be false, incomplete, misleading or not reflective of recent events or information.  Indemnification.  In addition to any other indemnification obligations under the Terms, Provider shall indemnify, defend, and hold Customer, and its officers, directors, employees, and contractors, harmless from and against any claims, suits, losses, liabilities, damages, expenses, costs, attorneys’ fees and court costs arising out of any third-party claim against Customer that alleges that any Customer Output infringes, violates, or misappropriates the third party’s intellectual property rights (including similar or related rights such as moral rights, rights of attribution, and sui generis rights), trade secret, publicity, or privacy rights (“Output Claim”). Notwithstanding the foregoing, Provider will have no liability for any claim to the extent that the Output Claim is based on or arises from: (a) any modification of the Customer Output by Customer; (b) any combination of the Customer Output with any other third party material, content, or information and/or Customer Content; or (c) any Customer Output that is based on a Customer Input provided by Customer or its Users.  In addition to any other indemnification obligations under the Terms, Customer shall indemnify, defend, and hold Provider, and its officers, directors, employees, and contractors, harmless from and against any claims, suits, losses, liabilities, damages, expenses, costs, attorneys’ fees and court costs arising out of any third-party claim against Provider that alleges that (a) the Customer Inputs, when used by Provider according to the terms of the Terms and this AI Addendum, violates, misappropriates, or otherwise infringes upon the intellectual property or other proprietary rights of another person or entity; or (2) results from Customer’s use of the AI Services in violation of the applicable restrictions in the Terms or this AI Addendum.  Order of Precedence. In the event of a conflict between the Terms and this AI Addendum, this AI Addendum takes precedence solely for purposes of the AI Services.  #### Blog URL: https://www.duckbillhq.com/blog/ #### Bluesight Case Study | Bluesight How Bluesight achieved 22x ROI on cloud cost management The healthcare technology company engaged Duckbill for a comprehensive AWS cost assessment during a critical integration period, achieving growth targets while launching three new products at minimal net increase. About Bluesight Bluesight provides critical software solutions for U.S. hospitals, addressing pharmacy operations, compliance, and drug management challenges. Their products support everything from inventory management and drug diversion monitoring to predicting nationwide drug shortages—work that directly impacts patient care and hospital operations. As a private equity-backed company growing through both organic expansion and acquisitions, Bluesight operates entirely in AWS. With multiple product lines and data-intensive workloads, cloud efficiency isn't just about cutting costs—it's about sustainable growth. The challenge Bluesight faced a perfect storm of cost optimization pressures: Recent acquisition complexity. A new acquisition brought disparate tech stacks and data center migrations into the fold, significantly expanding their AWS footprint and operational scope. Time constraints. The business needed to hit aggressive savings targets within 8-9 months to meet board commitments—not the typical timeline for major infrastructure changes. Previously optimized baseline. Bluesight had already implemented a FinOps program, established reserved instances and savings plans, and was running at a healthy 8-10% of revenue as cloud spend. Further optimization required specialist-level expertise. Growth in parallel. While optimizing costs, the engineering team was simultaneously launching three new products with additional data feeds and infrastructure requirements. Why Duckbill Bluesight evaluated multiple approaches, including software-based cost optimization tools. These tools could show where spend was happening, but they couldn't explain why, or what to do about it in the context of Bluesight's specific business constraints, risk tolerance, and timeline. What set Duckbill apart: Flexible engagement models. Duckbill offered options ranging from deep-dive reviews to embedded engineering support, allowing Bluesight to match the engagement to their specific timeline and business constraints. Depth of expertise. Even before the formal engagement, Duckbill reviewed months of AWS invoices and studied Bluesight's entire tech stack. The questions they asked demonstrated specialist-level understanding—not surface-level analysis. He continued: "We didn't have to explain anything about any of the services. They already knew everything about them. It was just a case of, 'Hey, why are you using this, this, and this? Have you considered this instead? Are there any blockers here?'" Executive-ready communication. Duckbill presented findings directly to Bluesight's executive team, including technical advisors and the CEO, building rapport and delivering nuanced recommendations about risk, timeline, and business impact. In order to go deeper, Duckbill Group designed a custom process to analyze these data-intensive tasks. The team looked at the utilization of the compute clusters for batch jobs and set goals that the Fanatics teams could track to better maintain efficiency going forward. These recommendations were tiered by effort and cost-savings so the Fanatics team could prioritize the improvements and tackle them over time.  The first round of changes was easy to prioritize. “When Duckbill Group’s report said we’d save millions of dollars annually everyone could see this would be a big win,” Sheeley said. “With the clarity of the recommendations and the obvious cost savings, the application teams were happy to negotiate priorities. And, it turns out, the changes were relatively easy to make and didn’t take very long to implement.”  Duckbill Group’s report also identified new key performance indicators (KPIs) for this core service, determined their current performance level, and then set new targets for performance based on similar companies. These new benchmarks will help the Fanatics team measure progress over time and maintain cost savings.  The engagement Bluesight worked with Duckbill on a comprehensive cost optimization assessment that examined their entire AWS environment. The engagement focused on areas where Bluesight could achieve meaningful savings within its business timeline, while also identifying longer-term architectural improvements for future consideration. Vijay emphasized the partnership aspect: Key areas of focus included: Storage optimization across backup, snapshot, and disaster recovery strategies Data transfer costs, including inter-region transfers and replication Compute efficiency and workload rightsizing Service-level cost drivers that weren't immediately apparent in standard AWS reporting Throughout the process, Duckbill helped Bluesight understand the trade-offs involved in various optimization approaches, ensuring recommendations aligned with the company's risk tolerance and operational requirements. Results Vijay shared the headline number with Bluesight's board: Exceeded board targets. Bluesight achieved—and surpassed—their aggressive cost savings goals within the required timeline. New products launched at net-zero cost. The savings exceeded the infrastructure costs of three new product launches, effectively making those deployments free from a cloud spend perspective. Strategic roadmap for 2026. Beyond immediate savings, Duckbill identified additional optimization opportunities that Bluesight will execute next year, providing a clear path for continued efficiency gains. The bottom line For engineering leaders, opportunities to directly impact enterprise value are rare. Revenue attribution is complex and indirect. But cloud optimization? That's a straight path to the bottom line. Overview Client: Bluesight, a healthcare technology vendor for U.S. hospitals Number of employees: Enterprise-scale healthcare software company Cloud environment: AWS Situation Following an acquisition that expanded their AWS footprint, Bluesight needed to hit aggressive cost savings targets within 8-9 months, while simultaneously launching three new data-intensive products. Standard optimization approaches wouldn't work—they'd already implemented FinOps practices and captured low-hanging fruit. Solution Duckbill conducted a deep-dive AWS cost assessment, identifying savings opportunities across storage, data transfer, and compute that aligned with Bluesight's timeline and risk tolerance. The engagement included direct executive presentations and a strategic roadmap for future optimization beyond immediate savings targets. Ready to lower your AWS bill? Let's talk #### Careers Careers at Duckbill Join us on a mission to untangle cloud cost management. See open positions Where we are at Duckbill was founded in 2019 as a small boutique services firm focused on a single problem: helping organizations manage their AWS costs. Since then, we’ve grown into one of the most trusted and experienced companies in this space, as evidenced by our customer logos, press coverage (NYT, CNBC, GeekWire, GeekWire again, SiliconAngle, The Information), and longevity. Our work has afforded us a front row seat to the most difficult problems in cloud cost management, and the insight to build exactly the services [link to services] needed to solve them. We are now launching Skyway, a cloud financial data platform to ingest, parse, structure, normalize, and clean consumption and billing data from all cloud providers an organization uses. With an estimated TAM of $5 billion in 2030 for AWS alone and $47.5 billion for all of public cloud by 2030, the opportunity is enormous. Who we are looking for We've built something that enterprise customers genuinely value — they're paying us, renewing, and referring others — with a relatively small team. We have a lot of one-off things that have worked well, and it's now time to build scalable strategies and execute on them. This means two things for people we are hiring: 1) you will have a real impact on where we go, and 2) you don’t need your hand held to get there. We win by being open and straightforward with our customers. Internally, that translates to honest communication, putting customers first, and collaborating without politics or egos getting in the way. We are also fairly irreverent, and we like to have fun when we are not heads down in this cloud economy thing. Benefits & perks Health The company covers healthcare at 100% of the premium. Yes, you really don’t pay anything. Vision The company provides vision insurance and covers 100% of the premium. Dental The company provides dental insurance and covers 100% of the premium. Retirement All employees are eligible for our 401k plan. Paid time off We want you at your best and well-rested, so the company provides a PTO policy of 20 days per year. Sick leave It sucks to work when you’re sick. That’s why the company offers unlimited sick leave to all employees. Open positions #### Commercial Terms of Service Commercial Terms of Service Last Updated: April 1, 2026 These Commercial Terms of Service (“Terms”) are an agreement between L9 Labs, Inc., a Delaware corporation (“Provider”), and the organization, company, or other entity that you represent (“Customer”). These Terms govern Customer’s use of the Product and/or Provider’s provision of the Professional Services, as such terms are defined below (collectively, the “Services”). Additionally, Customer’s use of Provider’s website is governed by the Website Terms of Use (“Terms of Use”), which are incorporated into these Terms by reference. These Terms are effective on the earlier of the date that Customer first electronically consents to a version of these Terms and the date that Customer first accesses any Services (“Effective Date”). Any User accepting these Terms on behalf of an Organization represents and warrants they have the authority to bind such Organization to these Terms. Services under these Terms are not for consumer use.  Definitions. The following terms shall have these prescribed meanings for purposes of these Terms:  “Affiliate” means an entity that, directly or indirectly, controls, is under the control of, or is under common control with a Party, where “control” means having more than fifty percent (50%) of the voting stock or other ownership interest. “Applicable Data Protection Laws” means the Applicable Laws that govern how Provider and/or Skyway may process or use an individual’s personal information, personal data, personally identifiable information, or other similar term. “Applicable Laws” means the laws, rules, regulations, court orders, and other binding requirements of a relevant government authority that apply to or govern Provider or Customer. “Beta Product” means an early or prerelease feature or version of the Product that is identified as beta or similar, or a version of the Product that is not generally available. “Confidential Information” means information in any form disclosed by or on behalf of a Discloser, including before the Effective Date, to a Recipient in connection with these Terms or the Services that (a) the Discloser identifies as “confidential”, “proprietary”, or the like; or (b) should be reasonably understood as confidential or proprietary due to its nature and the circumstances of its disclosure. Confidential Information includes the existence of these Terms and the information in any purchase contract, scope of work, or other ordering document that references these Terms (each, an “Order Form”). Customer’s Confidential Information includes, without limitation, non-public Customer Content and Outputs and Provider’s Confidential Information includes, without limitation, non-public information about, in, and that is a part of the Product. “Customer Content” means all data, information, or materials that Customer or Users submit to the Product or otherwise provide or make available to Provider, or that Provider learns or collects, directly or indirectly, in connection with the Product or Provider’s provision of the Professional Services, but excludes Feedback.  “Deliverables” means documents, spreadsheets, reports, plans, and other materials that are prepared by or on behalf of Provider in the course of performing the Professional Services for Customer and are specifically labeled as “Deliverables” in the applicable Order Form or Provider’s Professional Services Catalog provided to Customer (“Professional Services Catalog”), which is incorporated into these Terms by reference.  “Discloser” means a Party to these Terms when the Party is providing or disclosing Confidential Information to the other Party. “Documentation” means the usage manuals and instructional materials for Skyway and/or the Software that are made available by Provider to Customer. “Embargoed Country” means any country or region to or from where Applicable Laws generally restrict the export or import of goods, services, or money. “Feedback” means suggestions, feedback, or comments about the Product, Professional Services, or related offerings from Customers or Users. “Fees” means the amounts owed by Customer to Provider for the Services, as may be more specifically described in an Order Form. “Force Majeure Event” means an unforeseen event outside a Party’s reasonable control where the affected Party took reasonable measures to avoid or mitigate the impacts of the event on their obligations under these Terms, including, without limitation, unpredicted natural disasters like a major earthquake, war, pandemic, riot, act of terrorism, or public utility or internet failure. “GDPR” means European Union Regulation 2016/679 as implemented by local law in the relevant European Union member nation, and by section 3 of the United Kingdom’s European Union (Withdrawal) Act of 2018 in the United Kingdom. “Indemnifying Party” means a Party to these Terms when the Party is providing protection for a particular Claim under Section 10 (Indemnification) of these Terms. “Internal Use” means access or use solely for Customer’s and its Affiliates’ own internal business purposes. By way of example and not limitation, Internal Use does not include access or use: (a) for the benefit of any person or entity other than Customer or its Affiliates, or (b) in any event, for the development of any product or service. Internal Use is limited to access and use by Customer’s employees and contractors and, in either event, solely on Customer’s behalf and for Customer’s benefit. “OFAC” means the United States Department of Treasury’s Office of Foreign Assets Control. “Outputs” means reports, materials, data, and other outputs generated or produced by the Product and made available to Customer that are based on Customer Content.  “Personal Data” shall have the meaning(s) set forth in the Applicable Data Protection Laws for personal information, personal data, personally identifiable information, or other similar terms. “Product” means Skyway, the Software, and the Documentation, including all proprietary work and notices, analyses, systems, schemas, code, and other content, data, information, and materials therein (including without limitation all intellectual property rights therein and thereto). “Professional Services” means the services provided by Provider to Customer as described in an applicable Order Form and the Professional Services Catalog. “Prohibited Data” means (a) patient, medical, or other protected health information regulated by the Health Insurance Portability and Accountability Act; (b) credit, debit, bank account, or other financial account numbers; (c) social security numbers, driver’s license numbers, or other unique and private government ID numbers; (d) special categories of data as defined in the GDPR; and (e) other similar categories of sensitive information as set forth in the Applicable Data Protection Laws. “Protected Party” means a Party to these Terms when the Party is receiving the benefit of protection for a particular Claim under Section 10 (Indemnification) of these Terms. “Recipient” means a Party to these Terms when the Party receives Confidential Information from the other Party. “Skyway” means the cloud-based financial planning and analysis platform for cloud infrastructure developed and provided by Provider as a SaaS subscription. “Software” means the client-side software or application(s) made available by Provider for Customer to install, download (whether onto a machine or in a browser), or execute as part of the Product. “Usage Data” means data and information about the provision, use, and performance of the Product and related offerings based on Customer’s or a User’s use of the Product. Usage Data does not include Customer Content, Customer Confidential Information, or Outputs.   “User” means any individual who uses the Product on Customer’s behalf or through or associated with Customer’s account. Skyway Subscription. If an Order Form includes a Skyway Subscription, the following applies to the Skyway Subscription in such Order Form:    Skyway Access and Use. During the subscription period set forth in the Order Form, including any renewal periods, if any (“Subscription Period”), Provider grants Customer a non-exclusive, non-transferable (except as expressly provided in Section 12.7 (Assignment)), non-sublicensable license to (a) access and use Skyway for Customer’s Internal Use; and (b) copy and use the included Software and Documentation only as needed to so access and use Skyway in each case.  Responsibility for Users. Use of the Product must comply with all Documentation and access rights provided by Provider. Customer is responsible and liable for all uses of the Product resulting from access provided to or requested by Customer, directly or indirectly, whether such access or use is permitted by or in violation of these Terms. Without limiting the generality of the foregoing, Customer is responsible for all acts and omissions of its Users, and any act or omission by a User that would constitute a breach of these Terms if taken by Customer will be deemed a breach of these Terms by Customer. Customer will promptly notify Provider if it suspects or knows of any fraudulent activity with its accounts, passwords, or credentials related to Skyway or if they become compromised.  Restrictions on Customer. Except as expressly permitted by these Terms, Customer will not (and will not allow anyone else to): (a) reverse engineer, decompile, or attempt to discover any source code or underlying ideas or algorithms of the Product (except to the extent Applicable Laws prohibit this restriction); (b) provide, sell, transfer, sublicense, lend, distribute, rent, or otherwise allow others to access or use the Product; (c) remove any proprietary notices or labels; (d) copy, modify, or create derivative works of the Product; (e) conduct security or vulnerability tests on, interfere with the operation of, cause performance degradation of, or circumvent access restrictions of the Product; (f) access accounts, information, data, or portions of the Product to which Customer does not have explicit authorization; (g) use the Product to develop a competing service or product; (h) use the Product with any activity prohibited by Applicable Laws; (i) use the Product to obtain unauthorized access to anyone else’s networks or equipment; or (j) upload, submit, or otherwise make available to the Product any Customer Content to which Customer and Users do not have the proper rights for such activities.  Reservation of Rights. Except for the limited license to use the Product in Section 2.1 (Skyway Access and Use), Provider retains all right, title, and interest in and to the Product, whether developed before or after the Effective Date. All Feedback received or obtained by Provider shall be owned by Provider and may be used by Provider without any restriction or obligation.  Updates. Provider may provide updates to the Product to provide Customer with the functionalities and information that Customer desires. However, Provider has no obligation to develop or provide any updates or revisions to the Product, unless specifically agreed by the Parties in writing, and Provider reserves the right to alter or adjust performance specifications for the Product as it deems necessary or desirable, provided such alterations or adjustments do not materially diminish the capabilities or services of the Product as of the Effective Date. Customer is not entitled to separate products or new versions of the Product that Provider may from time to time introduce and market generally as a distinct licensed product or service, unless otherwise agreed by the Parties.  Third Party Users. Customer may only permit Users outside of its company or organization (i.e. Users that are not under Customer’s direct control) to access the Product with Provider’s express written consent, which Provider may withhold in its sole discretion or condition on such Users signing a non-disclosure agreement or other written agreement with Provider before such access is granted. Suspension. If Customer (a) has an outstanding, undisputed balance on its account for more than 60 days; (b) materially breaches these Terms; or (c) uses the Product in violation of these Terms or in a way that materially and negatively impacts the Product or others, then Provider may temporarily suspend Customer’s access to the Product with or without notice and without refund. However, Provider will make reasonable efforts to inform Customer before suspending Customer’s account when practical. Provider will reinstate Customer’s access to the Product only if and when Customer resolves all outstanding issues. Data Privacy. Provider shall collect, maintain, use, and process Personal Data in connection with the Product and Provider’s website in compliance with its Data Processing Agreement (“DPA”) and Privacy Policy (“Privacy Policy”), which are incorporated into these Terms by reference. Prohibited Data. Customer will not (and will not allow anyone else to) submit Prohibited Data to the Product unless authorized by an Order Form or in mutual written agreement of the Parties (email will suffice). AI Services. The use of artificial intelligence and/or machine learning components by Provider and Customer in relation to the Product shall be governed by the terms of Provider’s AI Addendum, which is incorporated into these Terms by reference (“AI Addendum”).  Usage Data. Notwithstanding anything to the contrary in these Terms, Provider may collect and compile Usage Data. As between Provider and Customer, all right, title, and interest in Usage Data, and all intellectual property rights therein, belong to and are retained solely by Provider, and Customer agrees that Provider may use Usage Data to the extent and in any manner permitted under Applicable Law, including, without limitation, to compile statistical and performance information, to maintain, improve, enhance, and promote the Product and other Provider products and services, and for other development, diagnostic, and corrective purposes in connection with the Product and other Provider products and services, provided that such use does not publicly identify Customer or Customer’s Confidential Information. Nothing in this section will reduce or limit Provider’s obligations regarding Personal Data that may be contained in Usage Data or Customer Content under Applicable Data Protection Laws. Professional Services. If an Order Form includes Professional Services, the following applies to the Professional Services in such Order Form: Professional Services; Deliverables. During the term set forth in the applicable Order Form, Provider shall provide the Professional Services and corresponding Deliverables to Customer as set forth in the Order Form and the Professional Services Catalog. In the event of a conflict or difference between the terms in the Professional Services Catalog, an Order Form, and/or these Terms, the order of precedence is as follows as to the Professional Services provided under the applicable Order Form: (1) the Order Form, (2) these Terms, and (3) the Professional Services Catalog, except as expressly set forth in Section 5.2.4. Reliance on Customer. The Professional Services are based on Customer Content. Provider is not responsible for any deficiencies in the Professional Services or related losses or damages incurred by Customer arising from or related to any incorrect or incomplete Customer Content.  Customer undertakes that all documents, information, and data necessary for Provider to perform the Professional Services will be made available to Provider in a timely fashion. Customer will make available such employees of its organization as are necessary to assist Provider in providing the Professional Services. In case any of the above conditions are not timely complied with, or if Provider has to interrupt the Professional Services for reasons not attributable to Provider’s gross negligence or willful misconduct, the period of completion set forth in the Order Form shall be automatically extended for such additional time as shall be necessary to perform the Professional Services, and any and all additional costs resulting therefrom shall be the responsibility of Customer.  Client Ownership of Professional Services Deliverables. The Professional Services generally do not constitute “works for hire,” “works made in the course of duty,” or similar terms under laws where the transfer of intellectual property occurs on the performance of services to a payor. Instead, subject to Section 3.4 (Provider IP), Customer acknowledges that only the Deliverables, as specified in the applicable Order Form and/or Professional Services Catalog, are “works made for hire” as that term is used in the Copyright Law of the United States, and Provider agrees that Customer shall and will be the owner of all intellectual property rights in and to such Deliverables following receipt of full payment for the Professional Services resulting in such Deliverables by Provider.  Provider IP. As between Provider and Customer, Provider exclusively owns any and all software (including object and source code), flow charts, algorithms, documentation, information, report templates, know-how, inventions, techniques, models, trademarks, ideas, and any and all other works and materials developed by Provider in connection with performing the Professional Services (including without limitation all intellectual property rights therein and thereto) (collectively, the “Provider IP”), other than the Deliverables, and that title shall remain with Provider. For the avoidance of doubt, the Provider IP does not include any Customer Confidential Information or Customer Content. Upon payment in full of the amounts due for the applicable Professional Services and to the extent any Provider IP is incorporated into the Deliverable(s), Customer shall have a perpetual, non-transferable (except as expressly provided in the Section 12.7 (Assignment)), non-exclusive license to use the Provider IP solely as a part of the Deliverable(s) for Customer’s Internal Use.  Fees; Payment. Fees. As consideration for the obligations of Provider under these Terms and any Order Form, Customer will pay Provider the Fees set forth in the applicable Order Form, according to the terms and invoicing procedures set forth in the Order Form. Unless the Order Form specifies a different currency, all Fees are in U.S. Dollars. Customer shall be responsible for any and all taxes levied on transactions under these Terms other than taxes on Provider’s income. Provider accepts payment via credit card, Automatic Clearing House (ACH), or wire transfers.   Late Payments. If Customer fails to make any payment when due, without limiting Provider’s other rights and remedies, Provider may suspend Customer’s and its Users’ access to any portion or all of the Product and/or provision of the Professional Services until such amounts are paid in full or, at Provider’s option, terminate this Agreement and pursue collection of such overdue amounts, and Customer agrees to be responsible for Provider’s attorneys’ fees and other costs incurred in any such proceedings or efforts.  Term; Termination.  Term. These Terms start on the Effective Date and continue until terminated by either Party or automatically as permitted herein (“Term”). Each Order Form shall remain in effect until the first date on which no Professional Services or Skyway Subscription are being provided under the Order Form.  Termination. Skyway Subscription Termination for Convenience. A Skyway Subscription may only be terminated for convenience at the end of the then-current Subscription Period (or during a Pilot Period, if applicable), as further set forth in the applicable Order Form.  Professional Services Termination for Convenience. The Professional Services set forth in any Order Form may be terminated by either Party upon at least thirty (30) days’ prior written Notice to the other Party. Automatic Termination. Any Professional Services shall automatically terminate when the Completion Criteria is met or, when applicable, when the Retainer Term expires and no additional Retainer Hours are purchased (as such terms are defined in the Professional Services Catalog). A Skyway Subscription shall automatically terminate at the end of the Subscription Period.   Additional Termination Rights. In addition to all other termination rights in these Terms, either Party may terminate these Terms in the following instances, and such termination will automatically also terminate all active Order Forms (regardless of any conflicting terms in such Order Forms): if the other Party fails to cure a material breach of these Terms or an Order Form following 30 days’ written Notice from the non-breaching Party to the breaching Party;  upon written Notice to the other Party if either Party (a) materially breaches these Terms or an Order Form in a manner that cannot be cured; (b) dissolves or stops conducting business without a successor; (c) makes an assignment for the benefit of creditors; or (d) becomes the debtor in insolvency, receivership, or bankruptcy proceedings that continue for more than 60 days; or  upon written Notice to the other Party if a Force Majeure Event prevents the Product from materially operating for 30 or more consecutive days and/or prevents Provider from providing the Professional Services to Customer for 30 or more consecutive days.  Notwithstanding anything to the contrary in these Terms, termination of these Terms does not excuse any of Customer’s payment obligations incurred prior to termination, subject to any refund rights set forth in Section 5.4 below.  Effect of Termination. Upon any expiration or termination of an Order Form or these Terms for any reason: All rights of Customer to use the Product (if any) will immediately terminate;  Upon Customer’s request, Provider will delete Customer Content related to the terminated Order Form(s) and/or Terms within 60 days; and Provider will submit a final bill or invoice for all outstanding Fees accrued before termination, if any, and Customer will pay the invoice according to the terms of these Terms.  Refunds Upon Early Termination. In the event that an Order Form is terminated before completion of all of the ordered Professional Services and/or the current Subscription Period because Provider terminated due to convenience, because Provider breached and did not cure such breach, or because Provider is the party undergoing any of the events in Section 5.2.4.2 of these Terms, then, as applicable to such Order Form, (a) for any incomplete Professional Services billed on a fixed fee basis, the Provider shall refund any amounts paid by Customer for such Professional Services prior to termination of the Order Form, and for any Professional Services billed on a monthly recurring basis or retainer basis the Provider shall refund any amounts paid by Customer for Professional Services not performed prior to termination of the Order Form; and (b) Provider shall refund a pro rata portion of the Skyway Fees related to the remaining portion of the Subscription Period.  In the event that an Order Form is terminated before completion of all of the ordered Professional Services and/or the current Subscription Period because Customer breached and did not cure such breach, or because Customer is the Party undergoing any of the events in Section 5.2.4.2 of these Terms, then no refunds will be given and any amounts owed prior to termination  (including any minimum billing cycles required) shall be due in full. Survival. The following sections will survive expiration or termination of these Terms: Section 1 (Definitions), Section 2.4 (Reservation of Rights), Section 2.10 (AI Services), Section 2.11 (Usage Data), Section 3.4 (Client Ownership of Professional Services Deliverables), Section 3.4 (Provider IP), Section 4.1 (Fees) for Fees accrued or payable before expiration or termination, Section 5.3 (Effect of Termination), this Section 5.5 (Survival), Section 6 (Customer Information), Section 7 (Representations & Warranties), Section 8 (Disclaimers), Section 9 (Limitation of Liability), Section 10 (Indemnification), Section 11 (Confidentiality),  and Section 12 (General Terms). Each Recipient may retain Discloser’s Confidential Information in accordance with its standard backup or record retention policies maintained in the ordinary course of business or as required by Applicable Laws, in which case Section 11 (Confidentiality) will continue to apply to retained Confidential Information. Customer Information.  Customer Content. As between the Parties, Customer owns all Customer Content.  Provider’s Use of Customer Content. Customer acknowledges and agrees that Provider’s access to and use of Customer Content and similar content from Provider’s other customers is an integral and necessary part of Provider’s provision of the Product and Professional Services to Customer and Provider’s other customers. Therefore, Customer grants Provider a non-exclusive, perpetual, irrevocable (other than for Provider’s uncured breach), fully paid-up right and license to access, reproduce, create derivative works, distribute, collect, retain, save, and use the Customer Content solely for the purposes of providing the Product and Professional Services to Customer and providing similar products and services to Provider’s other customers; improving the Product, the Professional Services, and Provider’s other products and services; and to otherwise fulfill its obligations and exercise its rights under these Terms; provided, however, that Customer Content will be anonymized, de-identified, and/or aggregated when used by Provider for any purpose other than to provide the Product and/or Professional Services to Customer or otherwise fulfilling Provider’s obligations or exercising Provider’s rights under these Terms. Anonymized, de-identified, and/or aggregated Customer Content combined with other content shall be derivative works created and owned exclusively by Provider.  Representations & Warranties. Mutual. Each Party represents and warrants that: (a) it has the legal power and authority to enter into these Terms; (b) it is duly organized, validly existing, and in good standing under the Applicable Laws of the jurisdiction of its origin; and (c) it will comply with all Applicable Laws in performing its obligations or exercising its rights in these Terms.  From Customer. Customer represents and warrants that it, all Users, and anyone submitting Customer Content to the Product and/or otherwise to Provider, such as for the Professional Services, each have and will continue to have all rights necessary to submit or make available such Customer Content and to allow the use of Customer Content as described in these Terms, the applicable Order Form, and/or by Provider at the time of the request. From Provider. Provider represents and warrants that: (a) it will not materially reduce the general functionality of Skyway during the Subscription Period; and (b) Provider shall have all necessary rights, title, and interest in and to the Product, Outputs, and/or the Deliverables sufficient for Provider to grant the intellectual property rights provided in these Terms. Provider Product Warranty Remedy. If Provider breaches the warranty in Section 7.3(a) (Representations & Warranties from Provider), Customer must give Provider Notice (with enough detail for Provider to understand or replicate the issue) within 45 days of discovering the issue. Within 45 days of receiving sufficient details of the warranty issue, Provider will attempt to restore the general functionality of Skyway. If Provider cannot resolve the issue, Customer may terminate the applicable Order Form and Provider will pay to Customer a prorated refund of prepaid Skyway Fees for the remainder of the Subscription Period. Provider’s restoration obligation, and Customer’s termination right, are Customer’s only remedies if Provider does not meet the warranty in Section 7.3(a) (Representations & Warranties from Provider). Disclaimers.  Not Legal Advice; No Guarantees. CUSTOMER UNDERSTANDS AND AGREES THAT THE INFORMATION, CONTENT, DATA, RESULTS, AND/OR MATERIALS OBTAINED FROM OR PROVIDED BY THE PRODUCT OR PROFESSIONAL SERVICES, INCLUDING, WITHOUT LIMITATION, OUTPUTS, (A) ARE NOT LEGAL ADVICE AND MAY NOT BE RELIED ON FOR LEGAL PURPOSES OR REGULATORY COMPLIANCE; (B) ARE FOR INFORMATIONAL PURPOSES ONLY AND SHOULD NOT BE RELIED ON FOR PHYSICAL ENGINEERING, ARCHITECTURAL, CONSTRUCTION, OR OTHER TANGIBLE (AS OPPOSED TO DIGITAL) TECHNICAL PURPOSES; AND (C) DO NOT GUARANTEE AN OUTCOME OR RESULT, INCLUDING BUT NOT LIMITED TO, REDUCED COSTS, RELEASE OF A PRODUCT, OR CUSTOMER SATISFACTION. FURTHER, PROVIDER MAKES NO GUARANTEES THAT THE PRODUCT WILL ALWAYS BE SAFE, SECURE, OR ERROR-FREE, OR THAT IT WILL FUNCTION WITHOUT DISRUPTIONS, DELAYS, OR IMPERFECTIONS. CUSTOMER IS SOLELY RESPONSIBLE FOR ALL LEGAL, FINANCIAL, AND REGULATORY COMPLIANCE AND DECISIONMAKING RELATED TO ITS BUSINESS ACTIVITIES AND FOR VERIFYING ALL INFORMATION RECEIVED FROM THE PRODUCT AND PROFESSIONAL SERVICES, INCLUDING, WITHOUT LIMITATION, OUTPUTS, AND PROVIDER TAKES ON NO RESPONSIBILITY OR LIABILITY RELATED TO ANY OF THE FOREGOING AS A RESULT OF PROVIDING THE PRODUCT OR PROFESSIONAL SERVICES.  Responsibilities of Customer. Customer is responsible for the accuracy and content of Customer Content. While Provider may assist Customer in negotiations with third parties, all decisions and obligations related to the negotiations and outcomes thereof are solely the responsibility of Customer, not Provider, including, without limitation, reviewing all legal, financial, and business terms of any contract with Customer’s legal and financial advisors and complying with all contractual obligations. Provider has no obligations to any third parties other than Customer as a result of these Terms.   Disclaimer of Warranties. ALL SERVICES AND WORK PRODUCT OF ANY SORT HEREUNDER, INCLUDING, WITHOUT LIMITATION, OUTPUTS AND DELIVERABLES, SHALL BE PROVIDED OR DELIVERED ON AN “AS-IS” BASIS. EXCEPT FOR THE WARRANTIES IN SECTION 7 (REPRESENTATIONS & WARRANTIES), ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, AND ANY CONDITIONS, REPRESENTATIONS, AND WARRANTIES ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE, ARE HEREBY EXCLUDED TO THE EXTENT ALLOWED BY LAW. THE WARRANTIES IN SECTION 7 (REPRESENTATIONS & WARRANTIES) DO NOT APPLY TO ANY MISUSE OR UNAUTHORIZED MODIFICATION OF THE PRODUCT OR OUTPUTS, NOR TO ANY PRODUCT OR SERVICE PROVIDED BY ANYONE OTHER THAN PROVIDER. THESE DISCLAIMERS APPLY TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAWS. Limitation of Liability.  Liability Cap. Except as provided in Section 9.3 (Exceptions), each Party’s total cumulative liability for all claims arising out of or relating to these Terms will not be more than the total amount of all Fees paid to Provider under these Terms in the 12-month period immediately preceding a claim.  Damages Waiver. Except as provided in Section 9.3 (Exceptions), under no circumstances will either Party be liable to the other for any incidental, indirect, special, consequential, or punitive damages, regardless of the nature of the claim, including, without limitation, lost profits, costs of delay, any failure of delivery, business interruption, costs of lost or damaged data or documentation, contractual obligations to third parties, or liabilities to third parties arising from any source, even if the Party from whom such damages are sought has been advised of the possibility of such damages. Exceptions. The liability cap in Section 9.1 does not apply to Section 10 (Indemnification), any breach of Section 11 (Confidentiality), or any claim arising from a Party’s gross negligence or willful misconduct. Nothing in these Terms will limit, exclude, or restrict a Party’s liability to the extent prohibited by Applicable Laws. Indemnification.  Protection by Provider. Provider will indemnify, defend, and hold harmless Customer from and against all losses, liabilities, damages, claims, and expenses, including reasonable attorneys’ fees and court costs (each, a “Claim”, and collectively, “Claims”), that arise from any third party (so not Customer, Customer’s Affiliates, or any User) action, proceeding, or claim that the Product or any Deliverables, when used by Customer according to the terms of these Terms, including the applicable Order Form, violates, misappropriates, or otherwise infringes upon any third party’s intellectual property or other proprietary rights. Protection by Customer. Customer will indemnify, defend, and hold harmless Provider, and its officers, directors, shareholders, employees, contractors, representatives, and agents, from and against all Claims that arise from any third party (so not Provider or Provider’s Affiliates) action, proceeding, or claim that (a) the Customer Content, when used according to these Terms or any Order Form, violates, misappropriates, or otherwise infringes upon any third party’s intellectual property or other proprietary rights, or (b) results from Customer’s or any of its Users’ breach or alleged breach of these Terms, including any Order Form, or use of the Product, including, without limitation, Users who are not part of Customer’s organization or company. Procedure. The Indemnifying Party’s obligations in this Section 10 are contingent upon the Protected Party: (a) promptly notifying the Indemnifying Party of each Claim for which it seeks protection; (b) providing reasonable assistance to the Indemnifying Party at the Indemnifying Party’s expense; and (c) giving the Indemnifying Party sole control over the defense and settlement of each Claim. A Protected Party may participate in a Claim for which it seeks protection with its own attorneys only at its own expense. The Indemnifying Party may not agree to any settlement of a Claim that contains an admission of fault or otherwise materially and adversely impacts the Protected Party without the prior written consent of the Protected Party.  Changes to Product. If required by settlement or court order, or if deemed reasonably necessary in response to a Claim under Section 10.1 (Protection by Provider), Provider may: (a) obtain the right for Customer to continue using the Product; (b) replace or modify the affected component of the Product without materially reducing the general functionality of the Product; or (c) if neither (a) nor (b) are reasonable, terminate the affected Order Form and issue a pro-rated refund of prepaid Fees for the remainder of the Subscription Period. Exclusions. Provider’s obligations as an Indemnifying Party will not apply to any Claims that result from: (a) modifications to the Product made by Customer or Outputs; (b) unauthorized use of the Product, including use in violation of these Terms; (c) the combination of the Services or Outputs with technology or content not provided by Provider; (d) Customer Content; (e) use of the Services or Outputs in a manner that Customer knows or reasonably should know violates or infringes the rights of others; or (f) use of an old version of the Product or failure to use updates to the Product provided by Provider where a newer release or update would avoid the Claim. Customer’s obligations as an Indemnifying Party will not apply to Claims that result from Provider’s unauthorized use of the Customer Content, including use in violation of these Terms. Exclusive Remedy. This Section 10 (Indemnification), together with any termination rights, describes each Protected Party’s exclusive remedy and each Indemnifying Party’s entire liability for a Claim covered by this Section 10 (Indemnification). Confidentiality. Non-Use and Non-Disclosure. Except as otherwise authorized in these Terms or as needed to fulfill its obligations or exercise its rights under these Terms , Recipient will not (a) use Discloser’s Confidential Information; or (b) intentionally disclose Discloser’s Confidential Information to anyone else. In addition, Recipient will protect Discloser’s Confidential Information using at least the same protections Recipient uses for its own similar information but no less than a reasonable standard of care. Exclusions. Confidential Information does not include information that (a) Recipient knew without any obligation of confidentiality before disclosure by Discloser; (b) is or becomes publicly known and generally available through no fault of Recipient; (c) Recipient receives under no obligation of confidentiality from someone else who is authorized to make the disclosure; or (d) Recipient independently developed without use of or reference to Discloser’s Confidential Information. Required Disclosures. Recipient may disclose Discloser’s Confidential Information to the extent required by Applicable Laws if, unless prohibited by Applicable Laws, Recipient provides Discloser reasonable advance Notice of the required disclosure and reasonably cooperates, at Discloser’s expense, with Discloser’s efforts to obtain confidential treatment for the Confidential Information.  Permitted Disclosures. Recipient may disclose Discloser’s Confidential Information to Users, employees, advisors, contractors, and representatives who each have a need to know the Confidential Information, but only if the person or entity is bound by confidentiality obligations at least as protective as those in this Section 11 (Confidentiality) and Recipient remains responsible for everyone’s compliance with the terms of these Terms, including, without limitation, this Section 11 (Confidentiality). General Terms.  Entire Agreement. These Terms (including the Website Terms of Use, AI Addendum, DPA, Privacy Policy, Order Forms, and other documents or terms that are incorporated by reference into these Terms) constitute the Parties’ entire understanding as to the Services’ provision and use, and supersede all prior negotiations, understandings, and agreements, between the Parties concerning the Product and/or Professional Services, to the extent ordered by Customer with an Order Form subject to these Terms. Notwithstanding the foregoing, in the event that Provider and Customer negotiate and execute a Master Agreement or other similar agreement concerning the Product and/or Professional Services (“Negotiated Agreement”) that is effective during the term of these Terms, such Negotiated Agreement will override and supersede these Terms as to the Product and/or Professional Services provided under such Negotiated Agreement, unless expressly agreed by Provider in writing. No terms or conditions in any Customer documentation or online vendor portal will apply to the Services unless expressly agreed to in a legally binding written agreement signed by an authorized Provider representative, regardless of what such terms may say. For any instances other than as set forth in Section 3.1 above, in the event of a conflict between these Terms and an Order Form, the terms of the Order Form shall control as to that particular Order Form only, except as expressly set forth in the DPA and Section 5.2.4 of these Terms.  Modifications, Severability, and Waiver. Provider may update these Terms  at any time by posting revised terms at https://www.duckbillhq.com/terms/commercial-terms/, to be effective 30 days after the updates are posted by Provider or Customer otherwise receives written Notice, except that updates made in response to changes to law or regulation take effect immediately upon posting or Notice. Changes will not apply retroactively. No other amendment to or modification of these Terms is effective unless it is in writing and signed by both Parties. If any term of these Term sis determined to be invalid or unenforceable by a relevant court or governing body, the remaining terms of these Terms will remain in full force and effect. The failure of a Party to enforce a term or to exercise an option or right in these Terms will not constitute a waiver by that Party of the term, option, or right. Governing Law and Chosen Courts. California law will govern all interpretations and disputes about these Terms, without regard to its conflict of laws provisions. The Parties will bring any legal suit, action, or proceeding about these Terms in the courts in the City and County of San Francisco and each Party irrevocably submits to the exclusive jurisdiction of the courts in the City and County of San Francisco. In any action or proceeding related to these Terms, the prevailing party will be entitled to recover costs and attorneys’ fees.  Injunctive Relief. Despite Section 12.4 (Governing Law and Chosen Courts), a breach of Section 11 (Confidentiality) or the violation of a Party’s intellectual property rights may cause irreparable harm for which monetary damages cannot adequately compensate. As a result, upon the actual or threatened breach of Section 11 (Confidentiality) or violation of a Party’s intellectual property rights, the non-breaching or non-violating Party may seek appropriate equitable relief, including an injunction, in any court of competent jurisdiction without the need to post a bond and without limiting its other rights or remedies. Non-Exhaustive Remedies. Except where these Terms provides for an exclusive remedy, seeking or exercising a remedy does not limit the other rights or remedies available to a Party. Assignment. Neither Party may transfer, license, or assign any rights or obligations under these Terms without the prior written consent of the other Party. However, either Party may assign these Terms upon Notice to the other Party if such Party undergoes a merger, change of control, reorganization, or sale of all or substantially all its equity, business, or assets to which these Terms relate. Any attempted but non-permitted assignment is void. These Terms will be binding upon and inure to the benefit of the Parties and their permitted successors and assigns. Publicity Rights. Provided that Customer gives its prior written consent, Provider may identify Customer and use Customer’s name and logo in marketing to identify Customer as a user of Provider’s products and services, provider Customer may opt out by emailing support@duckbillhq.com with the request.  Notices. All notices, demands, waivers, and other communications under these Terms (each, a “Notice”) must be in writing. Except for notices related to demands to arbitrate or where equitable relief is sought, any Notices provided under these Terms may be delivered electronically to the email address provided to Provider if to Customer; and to legal@duckbillhq.com if to Provider. Notice is effective only: (a) upon receipt by the receiving Party, and (b) if the Party giving the Notice has complied with all requirements of this Section 12.9.  Beta Products or Pilot Periods. If Provider gives Customer access to a Beta Product or the Product during a trial period (“Pilot Period”), the Beta Product, or the Product accessed during the Pilot Period, is provided “AS IS” and Sections 7.3 (Representations & Warranty From Provider) and 10.1 (Protections by Provider) do not apply to any Beta Products or Pilot Periods. Customer acknowledges that Beta Products are experimental in nature and may be modified or removed at Provider’s discretion with or without notice. Independent Contractors. The Parties are independent contractors, not agents, partners, or joint venturers. Neither Party is authorized to bind the other to any liability or obligation.  No Third-Party Beneficiary. There are no third-party beneficiaries of these Terms. Force Majeure. Neither Party will be liable for a delay or failure to perform its obligations of these Terms if caused by a Force Majeure Event. However, this section does not excuse Customer’s obligations to pay Fees.  Export Controls. Customer may not remove or export from the United States or allow the export or re-export of the Product or any related technology or materials in violation of any restrictions, laws, or regulations of the United States Department of Commerce, OFAC, or any other United States or foreign agency or authority. Customer represents and warrants that it is not (a) a resident or national of an Embargoed Country; (b) an entity organized under the laws of an Embargoed Country; (c) designated on any list of prohibited, restricted, or sanctioned parties maintained by the U.S. government or agencies or other applicable governments or agencies, including OFAC’s Specially Designated Nationals and Blocked Persons List and the UN Security Council Consolidated List; or (d) 50% or more owned by any party designated on any of the above lists. Provider may terminate these Terms immediately without notice or liability to comply, as determined in Provider’s sole discretion, with applicable export controls and sanctions laws and regulations.  Anti-Bribery. Neither Party will take any action that would be a violation of any Applicable Laws that prohibit the offering, giving, promising to offer or give, or receiving, directly or indirectly, money or anything of value to any third party to assist Provider or Customer in retaining or obtaining business. Examples of these kinds of laws include the U.S. Foreign Corrupt Practices Act and the UK Bribery Act 2010. Titles and Interpretation. Section titles are for convenience and reference only. All uses of “including” and similar phrases are non-exhaustive and without limitation. The United Nations Convention for the International Sale of Goods and the Uniform Computer Information Transaction Act do not apply to these Terms. #### Contact Us Let's talk compute finance Thorny compute problem? Questions about your cloud or AI spend? We're always willing to discuss what challenges you're facing and see if we can help. If we can't, we'll point you to someone who can. Prefer the phone? Give us a call: +1 (833) 297-2455 #### Contract Negotiations - AI model labs Services | Contract Negotiation - AI MODEL LABS Take control of your AI contracts Model and inference consumption is climbing fast and how you buy quietly decides what you pay. Whether it's frontier models from an AI lab or open models from a dozen different inference providers, Duckbill maps your options against the commitments you're already carrying to help you negotiate from a position of strength. Contact us The information is stacked against you Providers know exactly how their pricing works and what they can offer. Most buyers don't. Duckbill evens the table so you can commit without second-guessing your deal. List prices aren't your prices Public per-token rates say little about what you should pay at volume. Our proprietary pricing data and financial models show you what a strong agreement actually looks like before you sign. The market moves faster than you can track New models launch and old ones are deprecated constantly, and pricing moves with them. We negotiate against where the market is right now, not where it was last quarter or last month. The smartest deal starts with where you buy Choose the right channel before you negotiate the rate The same model can be used directly from a lab or accessed through a hyperscaler. Each path prices differently, carries different terms, and impacts with commitments you may already hold. If you already have a large cloud commitment, routing model spend through that provider can let you draw down existing dollars instead of spending net-new. We weigh these options with you first, so you're not leaving money or leverage on the table, then we negotiate the deal itself. Understand what actually drives your bill Token consumption, model mix, and usage patterns move your inference costs in ways that aren't obvious. We translate those technical drivers into financial terms, so you can see what's shaping your spend today and how it changes as you grow. Get a second set of eyes before you commit The details in these agreements are easy to miss. Our team validates your cost estimates and consumption forecasts against real market pricing, giving your leadership, board, and investors more confidence in your projections. Specialized contracts call for specialized help This isn't a standard software deal Per-token pricing, prepaid credits, committed-spend tiers, provisioned throughput, and model deprecation terms each carry their own risks. Negotiating them well takes an understanding of both the economics and the engineering behind them. Data and software, working together Our solutions give you a live view of the market and your own consumption and turns that into a negotiation strategy. Know whether your rates are competitive, which channel serves you best, and where there's room to push in negotiations. We build on the expertise you already have You know your roadmap and your workloads better than anyone. We know how model providers price and structure their deals, and where the market is heading. We work alongside your team, start to finish, to land you the best terms possible. Negotiate your next AI contract with Duckbill in your corner Whether you're choosing between buying direct or through a hyperscaler, signing your first committed-spend agreement, or renegotiating as your usage scales, we'll help you approach it with the market context and confidence to get it right. Contact us #### Contract Negotiations - Hyperscalers Services | Contract Negotiation - Hyperscalers Get the best cloud contract for your business Cloud contracts are complex, layered, and your single biggest expense line item after payroll. Working with experienced specialists to tailor a specific solution is your best bet for getting the contract you need. Contact us Don't go at it alone Negotiating with cloud providers can feel like fighting an uphill battle. Even the best procurement managers, CFOs, and finance teams need expert negotiators to procure cloud success. Don't leave anything on the table When cloud providers hold all the cards, it’s like negotiating in the dark. Duckbill will strengthen your side of the deal, so you have full confidence in your negotiation strategy. Don't settle for poor benchmarks Internal guesswork, generalist industry benchmarking, and peer advice will still leave you wondering if you have the right terms. Let Duckbill's specialized financial models guide your negotiation. Worry less, win more with experts in your corner Get ahead of leadership questions Across industries, leaders are asking harder questions about cloud spend. Duckbill can help you give the right answers and demonstrate authority on your cloud contract. We'll equip you to provide effective upward reporting, point to precise benchmarks, and show clear business wins. See what’s behind your cloud contract Our Cloud Economists will help you grasp how every aspect of your infrastructure impacts your hyperscaler contracts. We’ll put technical insights into context so you can understand which factors influence your bill today, and how they will influence it tomorrow. Benefit from a second set of eyes In cloud contract negotiation, every detail counts. Our experts have years of experience negotiating with AWS, Microsoft Azure, and Google Cloud and can validate your cost estimates and budget forecasts, giving your leadership, board, and investors more confidence in your commitments. There is no substitute for cloud expertise Generalists can’t do the work of specialists Cloud contracts are a beast of their own. Generalists can’t specialize in every kind of technology negotiation, and lack the engineering expertise to have leverage with cloud providers. You need a technically adept team that has deep, first-hand experience with cloud contracts. Experience is the difference maker We've seen just about every cloud environment and type of contract out there. Having negotiated billions in cloud contracts, we have the reputation and the know-how to ensure you secure the best deal for your business. We are experts at leveraging your expertise Your procurement team knows every detail of your business. Duckbill knows cloud providers inside out. At every step of the process, until your contract is signed, we'll work in tandem to ensure you get the best contract terms possible. Our singular guarantee We’re confident we can negotiate the best contract for your business. If you feel that we haven’t helped your team achieve your goals, we will refund you up to 100% of your fee. Contact us #### Cookie Policy URL: https://www.duckbillhq.com/cookie-policy/ #### Customers Our customers We help strategic finance, engineering, and procurement leaders take control of compute, their largest and fastest-growing spend, across every cloud, model, and inference provider. Ready to get ahead of your compute spend? Your largest, fastest-growing cost shouldn't run on spreadsheets and guesswork. Join the companies making it predictable. Contact us #### Data Processing Agreement Data Processing Agreement Last Updated: April 1, 2026 This Data Processing Agreement (“DPA”) is incorporated into and forms part of the Commercial Terms of Service, Master Agreement, or other agreement between Customer and Provider that references this DPA and governs Customer’s use of the Product and/or Professional Services (the “Agreement”), and applies to Provider’s processing of Customer Information. Capitalized terms used but not defined in this DPA will have the meaning set forth in the Agreement. Provider may amend this DPA from time to time on reasonable notice to Customer to the extent such changes are required due to changes in Privacy Laws. Customer and Provider may be referred to herein individually as a “Party” and collectively as the “Parties”.  1. DATA PROCESSING AND PROTECTION. Compliance with Law. Provider will comply with all Privacy Laws applicable to Provider relating to the privacy, confidentiality, security, or Processing of Personal Information in connection with the providing the Product and/or Professional Services (the “Services”). Limitations on Use. Provider will Process Personal Information (i) on Customer’s behalf, in the context of its business relationship with Customer and in accordance with Customer’s instructions, and (ii) as required by Privacy Laws or any other legal obligation to which Provider is subject, provided that Provider will inform Customer (unless prohibited by law) of the applicable legal requirement before any such processing. Specifically, the scope, classification and details of Processing (the “Business Purpose”) are described in Schedule A below (Description of Transfer).  The duration of the Processing will be the same as the duration of the Master Agreement, except as otherwise agreed to in this DPA. The details provided in Schedule A are deemed to satisfy any requirement to provide such details under any Privacy Laws.   CCPA. Provider acknowledges and agrees that the obligations in this DPA apply with respect to Personal Information subject to CCPA. The terms “commercial purpose,” “personal information,” “service provider,” “sell,” and “share,” have the meanings set out in CCPA. Provider acknowledges that it is a service provider and agrees and certifies that it shall not: (a) sell or share Personal Information; (b) retain, use, or disclose Personal Information for any purpose, including a commercial purpose, other than the Business Purpose; (c) retain, use, or disclose Personal Information outside of the direct business relationship between Provider and Customer; or (d) combine Personal Information with personal information that Provider receives from or on behalf of another person, or collects from its own interactions with the Individual.  Confidentiality. Provider will ensure that Provider Personnel who will be provided access to, or will otherwise Process, Personal Information are subject to a written confidentiality agreement or are under an appropriate statutory obligation of confidentiality. Information Security Program. Provider will implement, maintain, and, where necessary, update a written information security program that contains appropriate administrative, technical and physical safeguards to ensure the integrity and resilience of Personal Information and protect Personal Information against anticipated threats or hazards to its security, confidentiality or integrity (such as unauthorized access, collection, use, copying, modification, or disposal; unauthorized, unlawful or accidental loss, destruction, acquisition or damage; or any other unauthorized form of Processing) (“Information Security Program”).  Subprocessors. Customer hereby provides general authorization to Provider’s use of subprocessors to process Customer Information. A list of subprocessors currently engaged by Provider is available at https://trust.duckbillhq.com/subprocessors. Provider will enter into written agreements with each subprocessor containing reasonable provisions relating to the implementation of technical and organizational measures in compliance with Privacy Laws. Provider will remain liable for acts and omissions of its subprocessors in connection with its obligations under the Master Agreement.  Requests or Complaints from Individuals. To the extent Customer is unable to independently access the relevant Personal Information within Skyway, Provider will, taking into account the nature of the Processing, provide reasonable cooperation to assist Customer to respond to any requests or complaints from Individuals or applicable data protection authorities relating to the Processing of Personal Information under the Master Agreement. If any such request is made to Provider directly, Provider will not respond unless expressly authorized to do so by Customer, unless Provider is legally compelled to do so. If Provider is legally compelled to respond to such a request, Provider will promptly notify Customer and provide it with a copy of the request unless it is legally prohibited from doing so.  Regulatory Investigations. Upon notice to Provider, Provider will assist and support Customer in the event of an investigation by any law enforcement body or regulator, including a data protection or similar authority, if and to the extent that such investigation relates to Personal Information handled by Provider on behalf of Customer in accordance with this DPA. Such assistance will be at Customer’s sole expense, except where investigation was required due to Provider’s acts or omissions, in which case such assistance will be at Provider’s sole expense.  Data Breach.  Provider will notify Customer without undue delay (and in any event within seventy-two (72) hours) of any known breach of security leading to the accidental, unauthorized or unlawful destruction, loss, alteration, disclosure of, or access to, Customer Information stored or otherwise Processed by Provider in connection with the Master Agreement (a “Data Breach”). Provider will also provide reasonable assistance to Customer in Customer’s compliance with its Data Breach-related obligations, including without limitation by: (a) taking steps to mitigate the effects of the Data Breach and reduce the risk to Individuals whose Personal Information was involved (such steps to be determined by Provider in its sole discretion); and (b) providing Customer with the following information, to the extent known: (i) the nature of the Data Breach, including, where possible, how the Data Breach occurred, the categories and approximate number of Individuals concerned, and the categories and approximate number of records containing Customer Information concerned; (ii) the likely consequences of the Data Breach; and (iii) the measures Provider has taken or proposes to take to address the Data Breach, including where appropriate measures to mitigate its possible adverse effects. Where, and in so far as, it is not possible to provide all information at the same time, the initial notification will contain the information then available and further information will, as it becomes available, subsequently be provided without undue delay. For the avoidance of doubt, “Data Breach” does not include unsuccessful attempts or activities that do not result in the accidental, unauthorized or unlawful destruction, loss, alteration, unauthorized disclosure of, or access to, Customer Information, including, but not limited to, unsuccessful log-in attempts, pings, port scans, denial of service attacks, and other network attacks on firewalls or networked systems. The parties agree that notice under this section is not an admission of fault or liability by the notifying party..  Return or Disposal. Provider will, as appropriate and as directed by Customer, regularly dispose of Personal Information that is maintained by Provider but that is no longer necessary to perform its obligations under the Master Agreement or applicable laws. Upon Customer’s request or as otherwise required by law, Provider will immediately cease handling Personal Information and will return such Personal Information in a manner and format reasonably requested by Customer or, if specifically directed by Customer, will destroy, any or all Personal Information in Provider’s possession, power or control, except as otherwise required by law applicable to Provider. If Provider has such a legal obligation to retain Personal Information beyond the period otherwise specified by this Section, Provider will notify Customer in writing of that obligation, to the extent permitted by applicable law, and will return or destroy the Personal Information in accordance with this Section as soon as possible after that legally required retention period has ended. Upon request, Provider will provide a written certification that Personal Information has been returned or securely destroyed in accordance with this DPA. Assistance. Taking into account the nature of the processing and the information available to Provider, Provider will provide reasonable assistance to Customer in complying with Customer’s obligations under applicable Privacy Laws which address obligations with regard to security, breach notifications, data protection impact assessments, and prior consultation. Any such assistance is subject to Provider’s written agreement and may be subject to additional fees. In addition, Provider will inform Customer if Provider believes that any instructions of Customer regarding the Processing of Personal Information would violate applicable law. Adverse Changes. Provider will notify Customer promptly if Provider: (i) has reason to believe that it is unable to comply with any of its obligations under this DPA and cannot cure this inability to comply within a reasonable time frame; or (ii) becomes aware of any circumstances or change in applicable law that is likely to prevent it from fulfilling its obligations under this DPA.  In the event that this DPA, or any actions to be taken or contemplated to be taken in performance of this DPA, do not or would not satisfy either party’s obligations under the laws applicable to each Party, the Parties will negotiate in good faith upon an appropriate amendment to this DPA. DATA TRANSFERS.  Restricted Transfers of Personal Information Subject to GDPR or Adopting Countries. Except as otherwise set forth in this paragraph, the SCCs will apply to (i) any Transfer of Personal Information that is subject to the EU General Data Protection Regulation ((EU) 2016/679) (“GDPR”), or the laws of a country outside the EEA in which the competent authority has approved the use of the SCCs (each, an “Adopting Country”) or otherwise requires a legal basis for the Transfer of Personal Information; and (ii) any onward Transfer of such Personal Information to Provider located outside of the EEA or the UK (or if such Provider will access EU data from outside of the EEA or the UK). Where the Transfer relates to Personal Information of an Adopting Country, the Parties agree: All references in the SCCs to “EU,” “Union” or “Member State” will be interpreted as references to the Adopting Country; All references to EU law will be interpreted as references to the relevant provisions of the Adopting Country’s data protection law; For the purpose of Clause 17 of the SCCs, the SCCs will be governed by the law of the Adopting Country for transfer of Personal Information subject to the data protection laws of the Adopting Country. For the purpose of Clause 18 of the SCCs, any dispute arising from the SCCs will be resolved by the courts of the Adopting Country. For the purpose of Annex I.C of the SCCs, the competent data protection authority is the data protection authority of the Adopting Country. The SCCs, attached hereto as Exhibit A, are incorporated into and form part of this DPA. Restricted transfers from the United Kingdom under the SCCs: In case of any transfers of Personal Information from the United Kingdom subject to the data protection laws of the United Kingdom, the UK Addendum to the SCCs attached as Annex II to the SCCs shall apply. Restricted Transfers from Australia. If Customer, or any relevant Customer affiliate, is located in Australia and Transfers Personal Information to Provider or an Authorized Subprocessor that is located outside Australia, or Customer otherwise notifies Provider that this Section applies, the Provider and any Authorized Subprocessors must comply with the Privacy Act 1988 (Cth), including the Australian Privacy Principles, when dealing with Personal Information or otherwise providing the Services pursuant to the Master Agreement. For avoidance of doubt, Provider must continue to comply with its general obligations under Sections 1-4 of this DPA, in addition to this Section 2, where applicable. MISCELLANEOUS. The obligations of Provider under this DPA will continue for as long as Provider continues to have access to, is in possession or control of, or acquires Personal Information, even if all agreements between Provider and Customer have expired or have been terminated. The Parties agree that this DPA may be amended only by written agreement between the Parties. To the extent there is any conflict between Sections 1 to 3 of this DPA and the terms of any applicable Standard Contractual Clauses (“SCC’s”) , the terms of the SCC’s will prevail. To the extent the terms of the DPA conflict with any Agreement between the Parties with regard to the Processing of Personal Information, the terms of the DPA will prevail. This DPA may be executed in several counterparts (including delivery via facsimile or electronic mail), each of which will be deemed to be an original but all of which together will constitute one and the same instrument.  DEFINITIONS. Capitalized terms used but not defined in this DPA will have the meanings set forth in the Agreement.  “CCPA” means the California Consumer Privacy Act of 2018, as may be amended and superseded from time to time, including by the California Privacy Rights Act of 2020, and any regulations promulgated thereunder “Customer Information” means information or data provided by Customer in any form, and data used, generated or stored in connection with Customer’s use of the Services, including Personal Data.  “Individual” means any individual about whom Personal Information may be Processed under this DPA. “Personal Information” or “Personal Data” means any Customer Information received under this DPA that identifies, directly or indirectly, an individual or relates to an identifiable individual. “Privacy Laws” means all applicable international, federal, state, provincial and local laws, rules, regulations, directives and governmental requirements currently in effect and as they become effective relating in any way to the privacy, confidentiality or security or Processing of Personal Information including, without limitation, the CCPA, the General Data Protection Regulation (2016/679), the European Union Directives governing electronic commerce (Directive 2002/58/EC), and data retention (Directive 2006/24/EC); the UK General Data Protection Regulation; the Privacy Act 1988 (Cth), the Canadian Personal Information Protection and Electronic Documents Act (PIPEDA), Canada’s anti-spam legislation or “CASL”); the Controlling the Assault of Non-Solicited Pornography and Marketing Act (CAN-SPAM); information security breach notification laws (such as Cal. Civ. Code §§ 1798.29, 1798.82 - 1798.84); laws imposing minimum information security requirements (such as Cal. Civ. Code § 1798.81.5 and 201 Mass. Code Reg. 17.00); laws requiring the secure disposal of records containing certain Personal Information (such as N.Y. Gen. Bus. Law § 399-H), and all similar international, federal, provincial, state and local requirements “Process” or “Processing” means any operation or set of operations performed upon any information or data, whether or not by automatic means, including the collection, recording, organization, structuring, alteration, access, disclosure, copying, transfer, storage, deletion, retention, combination, restriction, adaptation, retrieval, consultation, destruction, disposal, sale, sharing, augmentation or other use of Personal Information, whether by automated means or otherwise. “Provider Personnel” means any Provider’s employee, contractor, subcontractor or agent to whom Provider authorizes to access or Process Customer Information.  “Transfer” means the access by, transfer or delivery to or disclosure of Personal Information to a person, entity or system located in a country or jurisdiction other than the country or jurisdiction from which the Personal Information originated. Schedule A Details of the Processing Activities Nature and Purpose of Processing: Provider will process Customer Information as necessary to provide the Services under the Agreement, for the purposes specified in the Agreement and this DPA, and in accordance with Customer’s instructions as set forth in this DPA. Duration of Processing: Provider will process Customer’s Personal Data as long as required (i) to provide the Services to Customer under the Agreement; (ii) for Provider’s legitimate business needs; or (iii) by applicable law or regulation. Customer Content and Usage Data will be processed and stored as set forth in the Agreement and this DPA.  Categories of Data Subjects: Customer employees and other users authorized by Customer to use the Services.  Categories of Personal Data: Provider processes Personal Data contained in Customer Content, Usage Data, and any Personal Data provided by Customer (including any Personal Data Customer collects from its end users and processes through its use of the Services) or collected by Provider in order to provide the Services or as otherwise set forth in the Agreement or this DPA. Categories of Personal Data include name, location, email address, and unique identifiers such as passwords. Sensitive Data or Special Categories of Data: None. Duration of Data Retention: Within thirty (30) days of the date of termination or expiration of the Agreement, Provider will (a) return a copy of all Customer Information in its control or possession or provide a self-service functionality allowing Customer to do the same,  if requested to do so by Customer within that period, and (b) delete all copies of Customer Information processed by Provider, except to the extent (i) applicable Privacy Laws or other applicable legal or regulatory requirements requires storage of the Customer Information, (ii) retention of the Customer Information by Provider is necessary to resolve a dispute between the parties, or (iii) retention of the Customer Information is necessary to combat harmful use of the Services. Exhibit A ​​EU CONTROLLER TO PROCESSOR STANDARD CONTRACTUAL CLAUSES SECTION I  Clause 1Purpose and scope (a)         The purpose of these standard contractual clauses is to ensure compliance with the requirements of Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation) for the transfer of personal data to a third country. (b)        The Parties: (i)         the natural or legal person(s), public authority/ies, agency/ies or other body/ies (hereinafter “entity/ies”) transferring the personal data, as listed in Annex I.A. (hereinafter each “data exporter”), and (ii)        the entity/ies in a third country receiving the personal data from the data exporter, directly or indirectly via another entity also Party to these Clauses, as listed in Annex I.A. (hereinafter each “data importer”) have agreed to these standard contractual clauses (hereinafter: “Clauses”). (c)         These Clauses apply with respect to the transfer of personal data as specified in Annex I.B. (d)        The Appendix to these Clauses containing the Annexes referred to therein forms an integral part of these Clauses. Clause 2Effect and invariability of the Clauses (a)         These Clauses set out appropriate safeguards, including enforceable data subject rights and effective legal remedies, pursuant to Article 46(1) and Article 46 (2)(c) of Regulation (EU) 2016/679 and, with respect to data transfers from controllers to processors and/or processors to processors, standard contractual clauses pursuant to Article 28(7) of Regulation (EU) 2016/679, provided they are not modified, except to select the appropriate Module(s) or to add or update information in the Appendix. This does not prevent the Parties from including the standard contractual clauses laid down in these Clauses in a wider contract and/or to add other clauses or additional safeguards, provided that they do not contradict, directly or indirectly, these Clauses or prejudice the fundamental rights or freedoms of data subjects. (b)        These Clauses are without prejudice to obligations to which the data exporter is subject by virtue of Regulation (EU) 2016/679. Clause 3Third-party beneficiaries (a)         Data subjects may invoke and enforce these Clauses, as third-party beneficiaries, against the data exporter and/or data importer, with the following exceptions: (i)         Clause 1, Clause 2, Clause 3, Clause 6, Clause 7; (ii)        Clause 8 - Clause 8.1(b), 8.9(a), (c), (d) and (e); (iii)       Clause 9 - Clause 9(a), (c), (d) and (e); (iv)       Clause 12 - Clause 12(a), (d) and (f); (v)        Clause 13; (vi)       Clause 15.1(c), (d) and (e); (vii)      Clause 16(e); (viii)    Clause 18 - Clause 18(a) and (b). (b)        Paragraph (a) is without prejudice to rights of data subjects under Regulation (EU) 2016/679. Clause 4Interpretation (a)         Where these Clauses use terms that are defined in Regulation (EU) 2016/679, those terms shall have the same meaning as in that Regulation. (b)        These Clauses shall be read and interpreted in the light of the provisions of Regulation (EU) 2016/679. (c)         These Clauses shall not be interpreted in a way that conflicts with rights and obligations provided for in Regulation (EU) 2016/679. Clause 5Hierarchy In the event of a contradiction between these Clauses and the provisions of related agreements between the Parties, existing at the time these Clauses are agreed or entered into thereafter, these Clauses shall prevail. Clause 6Description of the transfer(s) The details of the transfer(s), and in particular the categories of personal data that are transferred and the purpose(s) for which they are transferred, are specified in Annex I.B. Clause 7Docking clause (a)         An entity that is not a Party to these Clauses may, with the agreement of the Parties, accede to these Clauses at any time, either as a data exporter or as a data importer, by completing the Appendix and signing Annex I.A. (b)        Once it has completed the Appendix and signed Annex I.A, the acceding entity shall become a Party to these Clauses and have the rights and obligations of a data exporter or data importer in accordance with its designation in Annex I.A. (c)         The acceding entity shall have no rights or obligations arising under these Clauses from the period prior to becoming a Party. SECTION II– OBLIGATIONS OF THE PARTIES Clause 8Data protection safeguards The data exporter warrants that it has used reasonable efforts to determine that the data importer is able, through the implementation of appropriate technical and organisational measures, to satisfy its obligations under these Clauses. 8.1        Instructions (a)         The data importer shall process the personal data only on documented instructions from the data exporter. The data exporter may give such instructions throughout the duration of the contract. (b)        The data importer shall immediately inform the data exporter if it is unable to follow those instructions. 8.2        Purpose limitation The data importer shall process the personal data only for the specific purpose(s) of the transfer, as set out in Annex I.B, unless on further instructions from the data exporter. 8.3        Transparency On request, the data exporter shall make a copy of these Clauses, including the Appendix as completed by the Parties, available to the data subject free of charge. To the extent necessary to protect business secrets or other confidential information, the data exporter may redact part of the text of the Appendix to these Clauses prior to sharing a copy, but shall provide a meaningful summary where the data subject would otherwise not be able to understand the its content or exercise his/her rights. On request, the Parties shall provide the data subject with the reasons for the redactions, to the extent possible without revealing the redacted information. This Clause is without prejudice to the obligations of the data exporter under Articles 13 and 14 of Regulation (EU) 2016/679. 8.4        Accuracy If the data importer becomes aware that the personal data it has received is inaccurate, or has become outdated, it shall inform the data exporter without undue delay. In this case, the data importer shall cooperate with the data exporter to erase or rectify the data. 8.5        Duration of processing and erasure or return of data Processing by the data importer shall only take place for the duration specified in Annex I.B. After the end of the provision of the processing services, the data importer shall, at the choice of the data exporter, delete all personal data processed on behalf of the data exporter and certify to the data exporter that it has done so, or return to the data exporter all personal data processed on its behalf and delete existing copies. Until the data is deleted or returned, the data importer shall continue to ensure compliance with these Clauses. In case of local laws applicable to the data importer that prohibit return or deletion of the personal data, the data importer warrants that it will continue to ensure compliance with these Clauses and will only process it to the extent and for as long as required under that local law. This is without prejudice to Clause 14, in particular the requirement for the data importer under Clause 14(e) to notify the data exporter throughout the duration of the contract if it has reason to believe that it is or has become subject to laws or practices not in line with the requirements under Clause 14(a). 8.6        Security of processing (a)         The data importer and, during transmission, also the data exporter shall implement appropriate technical and organisational measures to ensure the security of the data, including protection against a breach of security leading to accidental or unlawful destruction, loss, alteration, unauthorised disclosure or access to that data (hereinafter “personal data breach”). In assessing the appropriate level of security, the Parties shall take due account of the state of the art, the costs of implementation, the nature, scope, context and purpose(s) of processing and the risks involved in the processing for the data subjects. The Parties shall in particular consider having recourse to encryption or pseudonymisation, including during transmission, where the purpose of processing can be fulfilled in that manner. In case of pseudonymisation, the additional information for attributing the personal data to a specific data subject shall, where possible, remain under the exclusive control of the data exporter. The data importer shall carry out regular checks to ensure that its measures continue to provide an appropriate level of security. (b)        The data importer shall grant access to the personal data to members of its personnel only to the extent strictly necessary for the implementation, management and monitoring of the contract. It shall ensure that persons authorised to process the personal data have committed themselves to confidentiality or are under an appropriate statutory obligation of confidentiality. (c)         In the event of a personal data breach concerning personal data processed by the data importer under these Clauses, the data importer shall take appropriate measures to address the breach, including measures to mitigate its adverse effects. The data importer shall also notify the data exporter without undue delay after having become aware of the breach. Such notification shall contain the details of a contact point where more information can be obtained, a description of the nature of the breach (including, where possible, categories and approximate number of data subjects and personal data records concerned), its likely consequences and the measures taken or proposed to address the breach including, where appropriate, measures to mitigate its possible adverse effects. Where, and in so far as, it is not possible to provide all information at the same time, the initial notification shall contain the information then available and further information shall, as it becomes available, subsequently be provided without undue delay. (d)        The data importer shall cooperate with and assist the data exporter to enable the data exporter to comply with its obligations under Regulation (EU) 2016/679, in particular to notify the competent supervisory authority and the affected data subjects, taking into account the nature of processing and the information available to the data importer. 8.7        Sensitive data Where the transfer involves personal data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, genetic data, or biometric data for the purpose of uniquely identifying a natural person, data concerning health or a person’s sex life or sexual orientation, or data relating to criminal convictions and offences (hereinafter “sensitive data”), the data importer shall apply the specific restrictions and/or additional safeguards described in Annex I.B. 8.8        Onward transfers The data importer shall only disclose the personal data to a third party on documented instructions from the data exporter. In addition, the data may only be disclosed to a third party located outside the European Union (in the same country as the data importer or in another third country, hereinafter “onward transfer”) if the third party is or agrees to be bound by these Clauses, under the appropriate Module, or if: (i)         the onward transfer is to a country benefitting from an adequacy decision pursuant to Article 45 of Regulation (EU) 2016/679 that covers the onward transfer; (ii)        the third party otherwise ensures appropriate safeguards pursuant to Articles 46 or 47 Regulation of (EU) 2016/679 with respect to the processing in question; (iii)       the onward transfer is necessary for the establishment, exercise or defence of legal claims in the context of specific administrative, regulatory or judicial proceedings; or (iv)       the onward transfer is necessary in order to protect the vital interests of the data subject or of another natural person. Any onward transfer is subject to compliance by the data importer with all the other safeguards under these Clauses, in particular purpose limitation. 8.9        Documentation and compliance (a)         The data importer shall promptly and adequately deal with enquiries from the data exporter that relate to the processing under these Clauses. (b)        The Parties shall be able to demonstrate compliance with these Clauses. In particular, the data importer shall keep appropriate documentation on the processing activities carried out on behalf of the data exporter. (c)         The data importer shall make available to the data exporter all information necessary to demonstrate compliance with the obligations set out in these Clauses and at the data exporter’s request, allow for and contribute to audits of the processing activities covered by these Clauses, at reasonable intervals or if there are indications of non- compliance. In deciding on a review or audit, the data exporter may take into account relevant certifications held by the data importer. (d)        The data exporter may choose to conduct the audit by itself or mandate an independent auditor. Audits may include inspections at the premises or physical facilities of the data importer and shall, where appropriate, be carried out with reasonable notice. (e)         The Parties shall make the information referred to in paragraphs (b) and (c), including the results of any audits, available to the competent supervisory authority on request. Clause 9Use of sub-processors (a)         The data importer has the data exporter’s general authorisation for the engagement of sub-processor(s) from an agreed list. The data importer shall specifically inform the data exporter in writing of any intended changes to that list through the addition or replacement of sub- processors at least thirty (30) days in advance, thereby giving the data exporter sufficient time to be able to object to such changes prior to the engagement of the sub-processor(s). The data importer shall provide the data exporter with the information necessary to enable the data exporter to exercise its right to object. (b)        Where the data importer engages a sub-processor to carry out specific processing activities (on behalf of the data exporter), it shall do so by way of a written contract that provides for, in substance, the same data protection obligations as those binding the data importer under these Clauses, including in terms of third-party beneficiary rights for data subjects. The Parties agree that, by complying with this Clause, the data importer fulfils its obligations under Clause 8.8. The data importer shall ensure that the sub-processor complies with the obligations to which the data importer is subject pursuant to these Clauses. (c)         The data importer shall provide, at the data exporter’s request, a copy of such a sub- processor agreement and any subsequent amendments to the data exporter. To the extent necessary to protect business secrets or other confidential information, including personal data, the data importer may redact the text of the agreement prior to sharing a copy. (d)        The data importer shall remain fully responsible to the data exporter for the performance of the sub-processor’s obligations under its contract with the data importer. The data importer shall notify the data exporter of any failure by the sub- processor to fulfil its obligations under that contract. (e)         The data importer shall agree a third-party beneficiary clause with the sub-processor whereby - in the event the data importer has factually disappeared, ceased to exist in law or has become insolvent - the data exporter shall have the right to terminate the sub-processor contract and to instruct the sub-processor to erase or return the personal data. Clause 10Data subject rights (a)         The data importer shall promptly notify the data exporter of any request it has received from a data subject. It shall not respond to that request itself unless it has been authorised to do so by the data exporter. (b)        The data importer shall assist the data exporter in fulfilling its obligations to respond to data subjects’ requests for the exercise of their rights under Regulation (EU) 2016/679. In this regard, the Parties may agree to the appropriate technical and organisational measures, taking into account the nature of the processing, by which the assistance shall be provided, as well as the scope and the extent of the assistance required. (c)         In fulfilling its obligations under paragraphs (a) and (b), the data importer shall comply with the instructions from the data exporter. Clause 11Redress (a)         The data importer shall inform data subjects in a transparent and easily accessible format, through individual notice or on its website, of a contact point authorised to handle complaints. It shall deal promptly with any complaints it receives from a data subject. (b)        In case of a dispute between a data subject and one of the Parties as regards compliance with these Clauses, that Party shall use its best efforts to resolve the issue amicably in a timely fashion. The Parties shall keep each other informed about such disputes and, where appropriate, cooperate in resolving them. (c)         Where the data subject invokes a third-party beneficiary right pursuant to Clause 3, the data importer shall accept the decision of the data subject to: (i)         lodge a complaint with the supervisory authority in the Member State of his/her habitual residence or place of work, or the competent supervisory authority pursuant to Clause 13; (ii)        refer the dispute to the competent courts within the meaning of Clause 18. (d)        The Parties accept that the data subject may be represented by a not-for-profit body, organisation or association under the conditions set out in Article 80(1) of Regulation (EU) 2016/679. (e)         The data importer shall abide by a decision that is binding under the applicable EU or Member State law. (f)         The data importer agrees that the choice made by the data subject will not prejudice his/her substantive and procedural rights to seek remedies in accordance with applicable laws. Clause 12Liability (a)         Each Party shall be liable to the other Party/ies for any damages it causes the other Party/ies by any breach of these Clauses. (b)        The data importer shall be liable to the data subject, and the data subject shall be entitled to receive compensation, for any material or non-material damages the data importer or its sub-processor causes the data subject by breaching the third-party beneficiary rights under these Clauses. (c)         Notwithstanding paragraph (b), the data exporter shall be liable to the data subject, and the data subject shall be entitled to receive compensation, for any material or non-material damages the data exporter or the data importer (or its sub-processor) causes the data subject by breaching the third-party beneficiary rights under these Clauses. This is without prejudice to the liability of the data exporter and, where the data exporter is a processor acting on behalf of a controller, to the liability of the controller under Regulation (EU) 2016/679 or Regulation (EU) 2018/1725, as applicable. (d)        The Parties agree that if the data exporter is held liable under paragraph (c) for damages caused by the data importer (or its sub-processor), it shall be entitled to claim back from the data importer that part of the compensation corresponding to the data importer’s responsibility for the damage. (e)         Where more than one Party is responsible for any damage caused to the data subject as a result of a breach of these Clauses, all responsible Parties shall be jointly and severally liable and the data subject is entitled to bring an action in court against any of these Parties. (f)         The Parties agree that if one Party is held liable under paragraph (e), it shall be entitled to claim back from the other Party/ies that part of the compensation corresponding to its / their responsibility for the damage. (g)        The data importer may not invoke the conduct of a sub-processor to avoid its own liability. Clause 13Supervision (a)         The supervisory authority with responsibility for ensuring compliance by the data exporter with Regulation (EU) 2016/679 as regards the data transfer, as indicated in Annex I.C, shall act as competent supervisory authority. (b)        The data importer agrees to submit itself to the jurisdiction of and cooperate with the competent supervisory authority in any procedures aimed at ensuring compliance with these Clauses. In particular, the data importer agrees to respond to enquiries, submit to audits and comply with the measures adopted by the supervisory authority, including remedial and compensatory measures. It shall provide the supervisory authority with written confirmation that the necessary actions have been taken. SECTION III – LOCAL LAWS AND OBLIGATIONS IN CASE OF ACCESS BY PUBLIC AUTHORITIES Clause 14Local laws and practices affecting compliance with the Clauses (a)         The Parties warrant that they have no reason to believe that the laws and practices in the third country of destination applicable to the processing of the personal data by the data importer, including any requirements to disclose personal data or measures authorising access by public authorities, prevent the data importer from fulfilling its obligations under these Clauses. This is based on the understanding that laws and practices that respect the essence of the fundamental rights and freedoms and do not exceed what is necessary and proportionate in a democratic society to safeguard one of the objectives listed in Article 23(1) of Regulation (EU) 2016/679, are not in contradiction with these Clauses. (b)        The Parties declare that in providing the warranty in paragraph (a), they have taken due account in particular of the following elements: (i)         the specific circumstances of the transfer, including the length of the processing chain, the number of actors involved and the transmission channels used; intended onward transfers; the type of recipient; the purpose of processing; the categories and format of the transferred personal data; the economic sector in which the transfer occurs; the storage location of the data transferred; (ii)        the laws and practices of the third country of destination– including those requiring the disclosure of data to public authorities or authorising access by such authorities – relevant in light of the specific circumstances of the transfer, and the applicable limitations and safeguards; (iii)       any relevant contractual, technical or organisational safeguards put in place to supplement the safeguards under these Clauses, including measures applied during transmission and to the processing of the personal data in the country of destination. (c)         The data importer warrants that, in carrying out the assessment under paragraph (b), it has made its best efforts to provide the data exporter with relevant information and agrees that it will continue to cooperate with the data exporter in ensuring compliance with these Clauses. (d)        The Parties agree to document the assessment under paragraph (b) and make it available to the competent supervisory authority on request. (e)         The data importer agrees to notify the data exporter promptly if, after having agreed to these Clauses and for the duration of the contract, it has reason to believe that it is or has become subject to laws or practices not in line with the requirements under paragraph (a), including following a change in the laws of the third country or a measure (such as a disclosure request) indicating an application of such laws in practice that is not in line with the requirements in paragraph (a). (f)         Following a notification pursuant to paragraph (e), or if the data exporter otherwise has reason to believe that the data importer can no longer fulfil its obligations under these Clauses, the data exporter shall promptly identify appropriate measures (e.g. technical or organisational measures to ensure security and confidentiality) to be adopted by the data exporter and/or data importer to address the situation. The data exporter shall suspend the data transfer if it considers that no appropriate safeguards for such transfer can be ensured, or if instructed by the competent supervisory authority to do so. In this case, the data exporter shall be entitled to terminate the contract, insofar as it concerns the processing of personal data under these Clauses. If the contract involves more than two Parties, the data exporter may exercise this right to termination only with respect to the relevant Party, unless the Parties have agreed otherwise. Where the contract is terminated pursuant to this Clause, Clause 16(d) and (e) shall apply. Clause 15Obligations of the data importer in case of access by public authorities 15.1     Notification (a)         The data importer agrees to notify the data exporter and, where possible, the data subject promptly (if necessary with the help of the data exporter) if it: (i)         receives a legally binding request from a public authority, including judicial authorities, under the laws of the country of destination for the disclosure of personal data transferred pursuant to these Clauses; such notification shall include information about the personal data requested, the requesting authority, the legal basis for the request and the response provided; or (ii)        becomes aware of any direct access by public authorities to personal data transferred pursuant to these Clauses in accordance with the laws of the country of destination; such notification shall include all information available to the importer. (b)        If the data importer is prohibited from notifying the data exporter and/or the data subject under the laws of the country of destination, the data importer agrees to use its best efforts to obtain a waiver of the prohibition, with a view to communicating as much information as possible, as soon as possible. The data importer agrees to document its best efforts in order to be able to demonstrate them on request of the data exporter. (c)         Where permissible under the laws of the country of destination, the data importer agrees to provide the data exporter, at regular intervals for the duration of the contract, with as much relevant information as possible on the requests received (in particular, number of requests, type of data requested, requesting authority/ies, whether requests have been challenged and the outcome of such challenges, etc.). (d)        The data importer agrees to preserve the information pursuant to paragraphs (a) to (c) for the duration of the contract and make it available to the competent supervisory authority on request. (e)         Paragraphs (a) to (c) are without prejudice to the obligation of the data importer pursuant to Clause 14(e) and Clause 16 to inform the data exporter promptly where it is unable to comply with these Clauses. 15.2     Review of legality and data minimisation (a)         The data importer agrees to review the legality of the request for disclosure, in particular whether it remains within the powers granted to the requesting public authority, and to challenge the request if, after careful assessment, it concludes that there are reasonable grounds to consider that the request is unlawful under the laws of the country of destination, applicable obligations under international law and principles of international comity. The data importer shall, under the same conditions, pursue possibilities of appeal. When challenging a request, the data importer shall seek interim measures with a view to suspending the effects of the request until the competent judicial authority has decided on its merits. It shall not disclose the personal data requested until required to do so under the applicable procedural rules. These requirements are without prejudice to the obligations of the data importer under Clause 14(e). (b)        The data importer agrees to document its legal assessment and any challenge to the request for disclosure and, to the extent permissible under the laws of the country of destination, make the documentation available to the data exporter. It shall also make it available to the competent supervisory authority on request. (c)         The data importer agrees to provide the minimum amount of information permissible when responding to a request for disclosure, based on a reasonable interpretation of the request. SECTION IV – FINAL PROVISIONS Clause 16Non-compliance with the Clauses and termination (a)         The data importer shall promptly inform the data exporter if it is unable to comply with these Clauses, for whatever reason. (b)        In the event that the data importer is in breach of these Clauses or unable to comply with these Clauses, the data exporter shall suspend the transfer of personal data to the data importer until compliance is again ensured or the contract is terminated. This is without prejudice to Clause 14(f). (c)         The data exporter shall be entitled to terminate the contract, insofar as it concerns the processing of personal data under these Clauses, where: (i)         the data exporter has suspended the transfer of personal data to the data importer pursuant to paragraph (b) and compliance with these Clauses is not restored within a reasonable time and in any event within one month of suspension; (ii)        the data importer is in substantial or persistent breach of these Clauses; or (iii)       the data importer fails to comply with a binding decision of a competent court or supervisory authority regarding its obligations under these Clauses. In these cases, it shall inform the competent supervisory authority of such non-compliance. Where the contract involves more than two Parties, the data exporter may exercise this right to termination only with respect to the relevant Party, unless the Parties have agreed otherwise. (d)        Personal data that has been transferred prior to the termination of the contract pursuant to paragraph (c) shall at the choice of the data exporter immediately be returned to the data exporter or deleted in its entirety. The same shall apply to any copies of the data. The data importer shall certify the deletion of the data to the data exporter. Until the data is deleted or returned, the data importer shall continue to ensure compliance with these Clauses. In case of local laws applicable to the data importer that prohibit the return or deletion of the transferred personal data, the data importer warrants that it will continue to ensure compliance with these Clauses and will only process the data to the extent and for as long as required under that local law. (e)         Either Party may revoke its agreement to be bound by these Clauses where (i) the European Commission adopts a decision pursuant to Article 45(3) of Regulation (EU) 2016/679 that covers the transfer of personal data to which these Clauses apply; or (ii) Regulation (EU) 2016/679 becomes part of the legal framework of the country to which the personal data is transferred. This is without prejudice to other obligations applying to the processing in question under Regulation (EU) 2016/679. Clause 17Governing law These Clauses shall be governed by the law of one of the EU Member States, provided such law allows for third-party beneficiary rights. The Parties agree that this shall be the law of Ireland. Clause 18Choice of forum and jurisdiction (a)         Any dispute arising from these Clauses shall be resolved by the courts of an EU Member State. (b)        The Parties agree that those shall be the courts of Ireland. (c)         A data subject may also bring legal proceedings against the data exporter and/or data importer before the courts of the Member State in which he/she has his/her habitual residence. (d)        The Parties agree to submit themselves to the jurisdiction of such courts. ANNEX I A.    LIST OF PARTIES Data exporter(s): Customer, as defined and described in the Agreement Activities relevant to the data transferred under these Clauses: Provision of the Services as described in the Agreement By agreeing to the DPA, Customer also agrees to be bound by the UK Addendum to the Standard Contractual Clauses as applicable Role (controller/processor): Controller Data importer(s): 1.Name: L9 Labs Inc. Address: 548 Market Street #79031, San Francisco, CA 94104 Activities relevant to the data transferred under these Clauses: Provision of Services as described in the Master Agreement By agreeing to the DPA, Customer also agrees to be bound by the UK Addendum to the Standard Contractual Clauses as applicable Role (controller/processor): Processor   B.     DESCRIPTION OF TRANSFER Any Customer Data processed by Company in connection with the Services that constitutes Personal Data including name, business contact information, date of birth, gender, identity document numbers, occupation, IP address, and user ID. Data subjects As detailed in Schedule A of the DPA Categories of data As detailed in Schedule A of the DPA Special categories of data As detailed in Schedule A of the DPA Processing operations As detailed in Schedule A of the DPA Purpose(s) of the data transfer and further processing As detailed in Schedule A of the DPA Frequency of the transfer As detailed in Schedule A of the DPA Duration of the Processing As detailed in Schedule A of the DPA The period for which the personal data will be retained, or, if that is not possible, the criteria used to determine that period As detailed in Schedule A of the DPA For transfers to (sub-) processors, also specify subject matter, nature and duration of the processing As detailed in Schedule A of the DPA C.    COMPETENT SUPERVISORY AUTHORITY Identify the competent supervisory authority/ies in accordance with Clause 13 Irish Data Protection Commission ANNEX II: UK Addendum to the Standard Contractual Clauses Date of this Addendum 1.         This Addendum is effective from the date of the Agreement. Background 2.   The Information Commissioner considers this Addendum provides appropriate safeguards for the purposes of transfers of personal data to a third country or an international organisation in reliance on Articles 46 of the UK GDPR and, with respect to data transfers from controllers to processors and/or processors to processors. Interpretation of this Addendum 3.      Where this Addendum uses terms that are defined in the Annex those terms shall have the same meaning as in the Annex. In addition, the following terms have the following meanings:  This Addendum This Addendum to the Clauses The Annex The Standard Contractual Clauses set out in the Annex of Commission Implementing Decision (EU) 2021/914 of 4 June 2021 UK Data Protection Laws All laws relating to data protection, the processing of personal data, privacy and/or electronic communications in force from time to time in the UK, including the UK GDPR  GDPR and the Data Protection Act 2018. UK GDPR The United Kingdom General Data Protection Regulation, as it forms part of the law of England and Wales, Scotland and Northern Ireland by virtue of section 3 of the European Union (Withdrawal) Act 2018. UK The United Kingdom of Great Britain and Northern Ireland   4.      This Addendum shall be read and interpreted in the light of the provisions of UK Data Protection Laws, and so that it fulfils the intention for it to provide the appropriate safeguards as required by Article 46 GDPR. 5.      This Addendum shall not be interpreted in a way that conflicts with rights and obligations provided for in UK Data Protection Laws. 6.      Any references to legislation (or specific provisions of legislation) means that legislation (or specific provision) as it may change over time. This includes where that legislation (or specific provision) has been consolidated, re- enacted and/or replaced after this Addendum has been entered into. Hierarchy 7.      In the event of a conflict or inconsistency between this Addendum and the provisions of the Clauses or other related agreements between the Parties, existing at the time this Addendum is agreed or entered into thereafter, the provisions which provide the most protection to data subjects shall prevail. Incorporation of the Clauses 8.      This Addendum incorporates the Clauses which are deemed to be amended to the extent necessary so they operate: (i)     for transfers made by the data exporter to the data importer, to the extent that UK Data Protection Laws apply to the data exporter’s processing when making that transfer; and (ii)   to provide appropriate safeguards for the transfers in accordance with Articles 46 of the UK GDPR Laws. 9.      The amendments required by Section 7 above, include (without limitation): (i)     References to the “Clauses” means this Addendum as it incorporates the Clauses (ii)    Clause 6 Description of the transfer(s) is replaced with: “The details of the transfers(s) and in particular the categories of personal data that are transferred and the purpose(s) for which they are transferred) are those specified in Annex I.B where UK Data Protection Laws apply to the data exporter’s processing when making that transfer.” (iii)  References to “Regulation (EU) 2016/679” or “that Regulation” are replaced by “UK Data Protection Laws” and references to specific Article(s) of “Regulation (EU) 2016/679” are replaced with the equivalent Article or Section of UK Data Protection Laws. (iv)   References to Regulation (EU) 2018/1725 are removed. (v)    References to the “Union”, “EU” and “EU Member State” are all replaced with the “UK” (vi)  Clause 13(a) and Part C of Annex II are not used; the “competent supervisory authority” is the Information Commissioner; (vii) Clause 17 is replaced to state “These Clauses are governed by the laws of England and Wales”. (viii) Clause 18 is replaced to state: “Any dispute arising from these Clauses shall be resolved by the courts of England and Wales. A data subject may also bring legal proceedings against the data exporter and/or data importer before the courts of any country in the UK. The Parties agree to submit themselves to the jurisdiction of such courts.” (ix)   The footnotes to the Clauses do not form part of the Addendum. Amendments to this Addendum 10.   The Parties may agree to change Clause 17 and/or 18 to refer to the laws and/or courts of Scotland or Northern Ireland. 11.   The Parties may amend this Addendum provided it maintains the appropriate safeguards required by Art 46 UK GDPR for the relevant transfer by incorporating the Clauses and making changes to them in accordance with Section 7 above. Executing this Addendum 12.   The Parties may enter into the Addendum (incorporating the Clauses) in any way that makes them legally binding on the Parties and allows data subjects to enforce their rights as set out in the Clauses. This includes (but is not limited to): (i)     By adding this Addendum to the Clauses and including in the following above the signatures in Annex 1A: “By signing we agree to be bound by the UK Addendum to the EU Commission Standard Contractual Clauses dated:” and add the date (where all transfers are under the Addendum) “By signing we also agree to be bound by the UK Addendum to the EU Commission Standard Contractual Clauses dated” and add the date (where there are transfers both under the Clauses and under the Addendum) (or words to the same effect) and executing the Clauses; or (ii)   By amending the Clauses in accordance with this Addendum, and executing those amended Clauses. #### Engineering Leadership Use Cases | Engineering Leadership Ship features, not spreadsheets Your focus is shipping but the business needs infrastructure and model spend that never takes margins by surprise. Every hour spent answering finance and procurement's questions about token usage or GPU commitments is an hour not spent on your backlog. Duckbill helps you make cost-aware architecture and model choices while shipping at full velocity. Make informed architecture and model decisions Your architecture and model choices drive your compute costs far more than idle resources ever will. Which model you route to, how you cache, where inference runs, and what you commit to shape your unit economics as much as your roadmap does. Duckbill gives you the data to make financially sound, future-proof decisions: proper demand planning, visibility into where volume discounts and committed rates kick in, and clarity on how each commitment affects your options. Navigate AI and cloud contracts like a pro Per-token tiers, prepaid credits, provisioned throughput, savings plans, reserved capacity: every provider structures its deals differently, and the terms are deliberately complex. When you're deciding whether to renegotiate now or ride out the term, or how much throughput to commit ahead of the next launch, you need concrete answers. Duckbill lets you analyze unified data across providers, ask better questions, and make confident decisions. Bridge the gap with finance and procurement Compute cost is everyone's business now, and the questions cut both ways. From "do we really need this much capacity" to "why did inference spend double this month," Duckbill helps you put deeply technical detail into financial context so you can give finance and procurement the answers they need, in the language they speak. This information turns shared ownership of compute cost from a source of friction into something that just works. You ship code. We'll sweat the contracts. Cost is a core engineering concern, but modeling, attribution, and contract negotiation are specialized financial skills your team shouldn't have to build in-house. Duckbill handles the forecasting, attribution, and contract analysis that pulls engineers away from feature work. You build the product, we do the rest. Related case studies Ship at full velocity. Commit with full confidence. Your job is building systems that scale sustainably while shipping features faster. Let us get the rest under control. Contact us #### Epsilon Case Study | EPSILON Epsilon unlocks savings with Duckbill’s expertise Epsilon is a leader in global adtech and martech company working with the world’s top enterprises. To help streamline their AWS architecture and reduce their AWS costs, Epsilon turned to Duckbill. The savings Duckbill found exceeded Epsilon’s estimates by more than six times, and the initial findings Duckbill found paid for the consultation costs within weeks. Now Epsilon plans to work with Duckbill as they move further into serverless architecture.  Overview 20MM customer database  100M emails sent annually  15T in transactions  Saved 6x initial Duckbill projection  Managing security, scale, and spend on AWS Epsilon helps enterprises understand their customers on a granular level so they can message those customers more effectively. Their extensive platform lets companies centralize their data, automate and track marketing campaigns, and develop deep analytical insights on how those campaigns are running so companies can tailor them to their business goals.  Scale, security, and performance are paramount to Epsilon. With Fortune 500 companies on their client roster, they have to be able to rely on the cloud services that power their platform.  But, scale and performance can come at a high cost. Epsilon’s wide range of SaaS products, managed services, and programs that power their customer’s marketing efforts all rely on AWS. To ensure their customers have a consistent experience using their platform, Epsilon invests heavily in AWS performance.  Epsilon wanted to crunch the numbers, review their architecture, and make sure they were getting the most out of what they were spending on AWS. That’s when they turned to Duckbill.  The power of the outlier Jason LeBaron was familiar with Duckbill before Epsilon started working with Duckbill. As VP of Solutions Architecture at Epsilon, he had seen Duckbill content floating around his corner of the internet.  “The work Duckbill produced was meaningful and relevant to the community on the security, operational stability, and cost front,” said LeBaron.  But, being interested in AWS cost optimization content, and being interested in working with the company behind that content are two different things. Epsilon had previously worked with a third party AWS expert that promised to cut their costs. “The other providers asked for high time commitments, or asked to embed additional agents and data collection tools to collect telemetry and the reports we were seeing were all the same. They all said optimize CloudTrail, NAT Gateways — all these common bits of advice. We already knew this. Where was the value in telling us stuff we already know?”“It left a bad taste in our mouth, so we were hesitant to do it again. But, Duckbill’s content and the reviews we saw from the community made Duckbill an outlier,” said LeBaron.  Part of what drew LeBaron to Duckbill was the fact that Epsilon would be getting advice from real, expert engineers who specialize specifically in AWS cost optimization. There was no layer of abstraction between analyst and expert. In traditional third party cost optimization engagements, a less-than-technical analyst might work directly with a client and then consult engineers for their insight. With Duckbill, there was no intermediary and no high-level abstraction. Duckbill would work directly with Epsilon and provide tangible, actionable insights.  “We saw a significantly different perspective than other providers when engaging with The Duckbill Group,” said LeBaron   Impact Revenue planning that once took 30 minutes now takes 5 seconds Answers in real-time — even mid-call with the CEO Forecasts stay current without manual rebuilds Spotting cost centers within AWS Epsilon had their own processes for finding anomalies in their AWS usage. They examine cost through a narrow, client-focused lens and a broader, company focused lens.  Looking through the client-focused lens, Epsilon runs mirrored deployments of similar use cases. With their customer’s data on hand, they know that Client A runs a similar use case to Client B. So, the two customers’ cost per customer transaction should be comparable. If it’s not, Epsilon can flag that there might be an anomaly, and can go investigate to see why one use case costs more. Then, they’ll tune that anomalous use case accordingly to get it back within an expected price point.  Looking through a business-focused lens, Epsilon monitors their AWS spend across their various product suites in the cloud and finds patterns in how they bring data in, store data, and move data. By observing the trends in their behavior, they can see where different APIs, or other costly services might be driving up their AWS bill and ask, “Are we being effective?” or “Why is the spend this high?”  Observing these granular, and thousand-foot patterns help Epsilon set their own internal benchmarks for AWS spend. Duckbill helped them find areas that they can reduce that spend. .  Duckbill’s observations and impact Duckbill didn’t just map out how Epsilon was using AWS, they mapped out how Epsilon could achieve key business objectives by streamlining their AWS usage. Duckbill engineers helped advise Epsilon on how they could store, move, and share data between systems, and pointed out which levers Epsilon could pull to save on AWS cost.  One of the first things that Duckbill found when examining Epsilon’s AWS usage with a fine-tooth comb was data transfer costs between two different regions of AWS Athena. Duckbill identified the issue, brought it up in an initial meeting with Epsilon, found the stakeholder whose product was related to that Athena service, and Epsilon was able to make a change.  “Right away, we were able to stop the flood of dollars going out the door related to that Athena service. We started getting that and other items extracted right away and we were able to take advantage of those savings,” said LeBaron. “Within the first one or two things that Duckbill found we more than paid for the cost of the engagement out the door and that was in a couple of weeks at most.”  Duckbill validated what Epsilon was already doing well, and examined new, high profile initiatives Epsilon had planned. For example, while AWS offers managed databases like RDS, Epsilon thought that creating their own database might be the right business move, but they wanted a second, expert opinion. Duckbill agreed, helping Epsilon launch that initiative with more confidence.  After several discovery meetings with Duckbill, a few project-focused meetings, and a thorough analysis of Epsilon’s AWS bill and architecture, Duckbill pointed out to Epsilon exactly where they could unlock vehicles to reduce cost reduction. These findings were shared across DevOps, IT, and engineering. With the cross-functional teams in the loop, it would be easier to make the changes that could help Epsilon save on AWS costs.  “We weren’t asked to install anything or sift through unnecessary details about the solutions,” said LeBaron. “There was a very low level of effort for our stakeholder community to come in and engage with Duckbill and it still brought very meaningful results on the backside.”  Those results speak for themselves. Duckbill initially estimated their findings would save Epsilon $3.5 million annually. Epsilon is already on track to hit that goal, and 12-15 individual areas Duckbill targeted for cost savings have already exceeded their respective savings estimates.  Now, Epsilon can build with more budget, and more confidence. Instead of wondering how a change to their AWS infrastructure will affect their bill, they know in advance what the cost will be. Seeing engineering actions and how they’re coupled with AWS spend is a tremendous advantage for Epsilon.  As Epsilon moves further into new architecture and transitions from EC2 to adopt more serverless infrastructure like Lambda, Jason LeBaron is keeping Duckbill’s number closeby.   Overview Client: Epsilon, a global adtech and martech leader Number of employees: Over 9,000 globally Situation This enterprise-scale marketing technology company needed to optimize their AWS infrastructure costs while maintaining security, scale, and performance for Fortune 500 clients managing billions of customer interactions. Solution Duckbill conducted a comprehensive AWS cost analysis, identifying data transfer inefficiencies and architectural optimization opportunities. Within weeks, initial findings paid for the engagement cost, with annual savings exceeding Epsilon's estimates by more than 6x ($3.5M+), enabling confident investment in serverless architecture migration. Ready to lower your AWS bill? Yes, Please! #### Events Events Join us for in-depth discussions on cloud cost challenges, emerging trends, and strategies that actually work. Upcoming events Past events Don't miss the next good thing Get updates on the San Francisco FinOps Meetup and other cloud cost events worth your time. Subscribe on Lu.ma #### Fanatics Case Study | Fanatics Fanatics streamlined cloud utilization, saving millions annually The sports merchandise e-commerce company engaged Duckbill to perform a major analysis of its business-critical data processing and analytics environment, uncovering millions in savings without impacting business operations. Impact Revenue planning that once took 30 minutes now takes 5 seconds Answers in real-time — even mid-call with the CEO Forecasts stay current without manual rebuilds About Fanatics Fanatics, the global leader for licensed sports merchandise, is a new breed of retailer – part tech company, part logistics expert, part e-commerce company, and part manufacturer – that combines exclusive deals with leagues and teams with on-demand production. Its agile supply chain increases the speed-to-market of high-quality fan merchandise distributed globally across all retail channels. One of the strategies that make Fanatics so successful is its ability to predict and respond to demand for sports merchandise for a specific team or player. For example, if an unknown rookie player makes an incredible play and gains popularity among fans, Fanatics can track in real-time the new demand for that player’s products and instantly offer a jersey sporting the newcomer’s name. By tracking signals via social media, sports news, and searches on its network of hundreds of online stores, when fans come shopping Fanatics is ready to deliver.  A renewed focus on cloud operational costs For Fanatics, driving new revenue is a top priority. As the company planned for the year ahead, even with strong growth forecasted, the technology organization knew that improving efficiency in the cloud was key to improving margins.  Like most e-commerce companies, the engineering teams are primarily focused on shipping new features and delivering a great customer experience, and the platform engineering team’s primary job is to support them in achieving a common business goal.  For Johnny Sheeley, Fanatics Director of Cloud Engineering, improving the efficiency of the cloud infrastructure was a top priority. The first point of investigation: The utilization of the large number of data-processing instances that powered the smart downstream systems. Daily, thousands of ephemeral instances were spun up for batch-processing data from many different partners, as well as from Fanatics’ own internal services. This added large quantities to the data lake.  Using a cloud cost management tool, Fanatics monitors usage stats, but the tool doesn’t offer guidance on how to manage individual resources to high-level business KPIs. Moreover, the engineering team didn’t want to veer away from product development priorities.  “We aren’t going to cut the velocity of our new feature delivery and that’s where our teams are focused,” said Sheeley. “So we decided to bring in the cloud experts at Duckbill who could give us a fresh perspective on our cloud architecture and help us streamline our cloud infrastructure costs.”  Impact Revenue planning that once took 30 minutes now takes 5 seconds Answers in real-time — even mid-call with the CEO Forecasts stay current without manual rebuilds Custom analysis identifies cost savings in automated processes Duckbill analyzed the recorded usage data on hundreds of thousands of data-processing instances. The team designed a strategy that would dramatically reduce Fanatics’ AWS spend: reduce the number of machines used in overnight batch processes, as well as streamline each machine’s resource allocation based on expected workload. To get there, the specifics for this action plan started with sophisticated data analysis and a deep understanding of Fanatics’ unique data usage model. While the cloud cost management tool could quantify the number of cloud instances, it took the special data analytics skills and custom analysis from Duckbill’s engineers to design a solution. The standard procedure for this type of large-scale batch data processing is to create ephemeral instances, run the jobs, and then terminate the instances, all automatically. With thousands of short-lived instances to analyze there isn’t a “dashboard view” of utilization that points to a workable solution. In order to go deeper, Duckbill designed a custom process to analyze these data-intensive tasks. The team looked at the utilization of the compute clusters for batch jobs and set goals that the Fanatics teams could track to better maintain efficiency going forward. These recommendations were tiered by effort and cost-savings so the Fanatics team could prioritize the improvements and tackle them over time.  The first round of changes was easy to prioritize. “When Duckbill’s report said we’d save millions of dollars annually everyone could see this would be a big win,” Sheeley said. “With the clarity of the recommendations and the obvious cost savings, the application teams were happy to negotiate priorities. And, it turns out, the changes were relatively easy to make and didn’t take very long to implement.”  Duckbill’s report also identified new key performance indicators (KPIs) for this core service, determined their current performance level, and then set new targets for performance based on similar companies. These new benchmarks will help the Fanatics team measure progress over time and maintain cost savings.  Bringing predictability to cloud spend “Before the cloud, you had to justify your investments in hardware and software purchases and explain why the capacity was necessary,” Sheeley remembers. “The whole reason people adopt AWS is to avoid that, right? But it’s way too easy to get hooked on AWS and all of a sudden, your spend is out of control.  “One of the greatest benefits we got from our engagement with Duckbill is we now have lifecycle management for every massive data bucket. And we now have a predictable spend.” Sheeley isn’t finished working with Duckbill yet. Next, the teams will tackle architecture and performance improvements. “This first round was focused primarily on cost savings and some aspects of performance, and next we want to tackle more architectural decisions and upstream issues.” Sheeley found Duckbill easy to work with, and especially appreciated the straightforward approach to reporting recommendations. “They explained the approach that they would take, the things that they needed to be successful, and from there, they interviewed our internal teams to gather context and data from our environments,” Sheeley said. “I know the analysis they did was complex but what was great is the report was not. This helped a lot because I could say ‘Read this and we can save two million dollars.’ People could easily understand the impact.” Overview Client: Fanatics, a global sports merchandise retailer Number of employees: Over 7,500 globally Situation This fast-growing e-commerce retailer needed to improve the efficiency of its large-scale data processing platform, improving its cost basis and increasing profit margins. Solution This fast-growing e-commerce retailer needed to improve the efficiency of its large-scale data processing platform, improving its cost basis and increasing profit margins. Ready to lower your AWS bill? Yes, Please! #### Finance Use Cases | Finance Your biggest, fastest-growing cost shouldn't be a surprise Compute is now one of the largest line items on your P&L and one of the hardest to predict. Spend is split across hyperscalers, model labs, inference providers, and neoclouds, each with its own pricing model, commitments, and rate of change. Spreadsheets weren't built to forecast a cost that reprices this often, or to commit tens of millions against usage that shifts month to month. Duckbill bridges the gap between compute complexity and financial rigor, giving you the context, accuracy, and credibility you need. Forecast a moving target with confidence Model economics, usage, and provider pricing all change faster than an annual budget cycle can absorb. Duckbill gives you rolling and long-range forecasts across every provider, built on your actual consumption and contract terms so you can plan for what's coming instead of explaining what already happened. And because you know your roadmap best, our data and recommendations are yours to shape, not a black box you have to trust blindly. Commit to the right number, not a guess PPAs, prepaid credits, reserved capacity, and committed-spend tiers are the largest commitment decisions you make against future usage you can't fully see. Duckbill models those commitments against your real burn-down and projected demand, so you can size them correctly and know your exposure before you sign, not after. End data quality and reconciliation chaos Nothing undermines trust faster than numbers that don't match across tools and providers. Duckbill consolidates contract terms, consumption, and cost drivers into one consistent view across every cloud, model, and inference provider so you spend less time reconciling discrepancies and more time driving strategic decisions. Walk into board meetings with defensible numbers Stop hedging your forecasts or leaning solely on engineering estimates. Duckbill equips you with data and analysis that withstand executive and investor review, positioning you as the authoritative voice on the company's compute spend, size, its trajectory, and what you're doing to manage it. Know whether your deals are actually competitive Your current tools tell you what you spent, not whether you should have paid it. Duckbill benchmarks your rates and terms against real market pricing and tells you where to push. Know which commitments to revisit, which discounts you're owed, and when a renegotiation is worth it. Not just visibility, but a recommendation you can take to leadership. Drive the biggest bets, not just the budget You're already in the room when the big commitments get made. Duckbill arms you to drive them: model multi-year scenarios, pressure-test provider strategies, and put a rigorous financial frame around every major compute decision before it's signed. On the fastest-growing line of the P&L, you're not fighting for a seat; you're setting the agenda. Related case studies Stop second-guessing your compute spend Finance teams shouldn't be left on the outside looking in. Get the clarity, context, and predictability your role demands across every provider that matters. Contact us #### FinOps Use Cases | Finops Take enterprise FinOps from spreadsheet wrestling to strategic advisory FinOps at enterprise scale faces a hard challenge: managing cloud spend that impacts millions in budget decisions while navigating inherent technical complexity of cloud pricing models, all while building organizational credibility in a discipline that's still evolving. Duckbill was purpose-built to untangle this exact knot — we just got a bit of a head start. Bring certainty, build credibility Leadership questions about cloud spend don't allow for hedging — you need precise answers backed by solid analysis. Duckbill will ensure you walk into those conversations with comprehensive visibility into cost drivers, contract implications, and business context, so you can respond with authority and confidence. Pioneer proactivity in your cloud cost management Stop playing defense with cloud costs. Duckbill enables you to model future scenarios, identify emerging optimization opportunities, and provide strategic guidance on multi-year cloud strategy. Instead of reacting to billing surprises, position your organization ahead of changes in the cloud landscape. Know what you are measuring When unit economics are unclear, attribution chaos can undermine your data analysis and interpretation. Our experts put technical insights into context so you can understand the factors that influence your cloud bill. Instead of explaining data discrepancies, you'll have consistent metrics that everyone can trust, and that you can defend with confidence. End the manual tracking nightmares Stop stitching together data from multiple spreadsheets and inadequate tools. Our platform gives you clarity through context by consolidating contract terms, consumption data, and cost drivers into one transparent view — so you never again have to wonder what you're missing in your analysis. Stop winging your contract decisions When leadership asks "Should we renegotiate now or next year?" or "Can we get a better deal?" you need confident answers. Duckbill streamlines tracking commitments and untangling discount structures, enabling you to forecast contract performance, and model and budget cloud costs confidently. Demonstrate measurable value in a hard-to-measure role We don’t just empower you with accurate forecasting, budget adherence, and the ability to translate analysis into organizational action. We enable you to become the authority people turn to for your enterprise’s cloud spend, so you can focus on clear value, not explanations of complex processes. Related case studies Get top cloud expertise in your corner Put that nagging question “Could I be doing more?” to rest. Contact us #### FinOps Academy Join the FinOps Academy Stop guessing. Start building a FinOps practice that actually works. We've spent years in the trenches with hundreds of AWS customers. This 15-day email series answers the questions every team asks us—the ones you're probably asking right now. Real problems, real solutions, zero fluff. Start your 15-day FinOps crash course Sign up and we'll send you the playbook we wish existed when we started. Plus first access to everything else we're building. What you'll get Daily insights from actual customer engagements, not vendor marketing The uncomfortable truths about what works (and what wastes everyone's time) Frameworks you can implement Monday morning Start learning #### FinOps Practice Assessments Services | FinOps Practice Assessments Are you getting the most out of your FinOps program? Building a FinOps team from the ground up is challenging. So is assessing if your team’s FinOps strategy is working. Duckbill can pave your way to a more mature FinOps function. Let's talk Build certainty into your cloud cost Unpredictability plagues even the best FinOps efforts. We are experienced engineers and finance experts who can cut through cloud cost complexity and align the people in your organization so you can control costs, forecast spend, and take back the reins of your cloud spend. Tap into multi-cloud expertise Benefit from the knowledge of Duckbill experts who have worked with some of the largest and most complex cloud environments in the world. By leveraging best practices honed across diverse industries and infrastructures, we bring insights and solutions tailored to your unique challenges. Leverage our focused experience Duckbill has extensive experience with organizations that have what we call "high spend dispersion": environments where cloud spend is widely distributed across multiple teams, departments, or agencies. Our proprietary methodology combined with latest industry standards ensures robust cost allocation, detailed reporting, and clear accountability, enabling you to scale governance that balances central oversight with distributed responsibility. Certifications & partnerships The Duckbill Group maintains strong connections with the FinOps community through active memberships in key industry organizations: FinOps Foundation As a FinOps Foundation member, we contribute to and stay current with evolving industry standards and best practices. Technology Business Management (TBM) Council Our membership ensures our methodologies align with established IT financial management frameworks. AWS Partner Network Our AWS Partner status gives us access to specialized training, resources, and technical support to enhance our service delivery. Many of our staff have certifications via FinOps Foundation and AWS at varying levels, ensuring ongoing deep technical expertise in cloud architecture, optimization, and financial management. Level up your FinOps practice today There’s a big difference between writing a playbook, and running a playbook. Duckbill knows how to help businesses run FinOps playbooks that produce results. Contact us #### Home Your AI contracts need more than a spreadsheet. Negotiate, manage, and forecast your compute across every cloud, neocloud, and inference provider with confidence. Schedule a demo AI spend is your fastest-growing cost and the hardest one to manage.We change that. Understand what you're spending Track all your consumption and commitments across every provider. Forecast with confidence Scenario-model your AI and cloud usage against real pricing, and know exactly when a commitment or renegotiation makes sense. Negotiate with strength  The commitment tracking, invoice validation, and benchmark data you need to negotiate with confidence. Compute is complex. Managing it doesn't have to be. The only compute financial planning platform that is contract-aware. Skyway accounts for the specifics of your contract alongside your actual consumption, enabling you to make commitment decisions with confidence. Manage every technology provider that matters Evolve from outdated, siloed spreadsheets to always-on data feeds for your model labs, inference APIs, neoclouds, coding assistants, and hyperscalers. Stop wondering if your contract is competitive We maintain the market's pricing data, benchmark your deal against it, and hand you your next move. Catch the discrepancies before they cost you Misapplied discounts, billing errors, credits that never landed. We check every invoice against your contract and hand you the evidence to recover what you're owed. What sets us apart Built for companies signing the largest commitments in tech history We've negotiated tens of billions in cloud and AI contracts, where your biggest, most complex commitments live. All our customers keep saying, "I've never seen anything like this" The largest commitments in tech history have changed the game and new approaches are needed. Model your infrastructure contracts and consumption together, across every provider that matters. We provide context on public rates and visibility into the private reality Anyone can scrape a price list. We pair public pricing with proprietary benchmark data, then tie it to your actual consumption and contracts to decipher what your rates mean for you. We have the platform and the expertise Purpose-built software handles data complexity while seasoned practitioners make critical judgment calls — the only combination that solves contract chaos. Duckbill Blog Expert takes, deep dives, and pro tips on cloud cost The biggest commitments in tech history are still managed in spreadsheets. Yours shouldn't be. See your compute position #### Honeycomb Case Study | HONEYCOMB Honeycomb discovers a hidden AWS pricing edge case Honeycomb is one of the most innovative companies in the observability space, having taken the lead in defining the space and setting the bar for what observability means in software. Their unique requirements for durability and resiliency led to some interesting challenges when it came to managing data transfer costs. Enter Charity Majors, CTO & Cofounder: Our AWS bill was really high — weirdly high. I tried to lower it myself. (I’ve been running stuff with AWS for years and have managed the billing before.) A big percentage of the bill was driven by cross-AZ data transfer, which seemed really strange because we weren’t doing anything way out of the ordinary. We figured we’d see if Duckbill could do something about it. They know more about how AWS billing works than anyone else in the world, including all of AWS. It’s a lot of experience that you can only have if you’re focusing on this one problem for a long time. My hesitation was the fact that it was very expensive to hire Duckbill, but they said if they didn’t save us at least that much in three months, we wouldn’t be charged. That’s a pretty easy decision. Mike and Corey were our expert guides, spiritual leaders, and life coaches for three months of intense soul searching. We walked through the architecture together and looking for alternate ways to structure it. We’ve been architecting our app with all of the best practices in mind for failovers and resiliency, but it’s a trade off between cost and reliability. We had gone all in on the reliability side and didn’t realize there were going to be so many hidden costs associated with that. Working with Duckbill, we discovered a hidden edge-case pricing inflection for cross-AZ data transfers and how we could re-architect everything to strike a different balance. Not content with just finding the issue, they arranged meetings with AWS on our behalf to ensure our concerns reached the right level (which was disturbingly high!) as well as to deliver both short-term and long-term mitigations.  Due to the high cost and complexity of AWS, everybody’s got stuff working in their systems they don’t realize they’re getting charged for. Nobody’s putting time into understanding this problem like Duckbill does. There is no downside to having them work on your AWS bill to look for ways to save you a lot of money. It’s really free, because they won’t charge more than they find.  Overview Client: Honeycomb.io, an observability platform leader Number of employees: Approximately 200-500 Situation This innovative observability company faced unusually high AWS costs driven by cross-AZ data transfer charges. Despite the CTO's AWS expertise and following reliability best practices, hidden architectural costs were impacting their operating budget. Solution Duckbill conducted a three-month deep-dive analysis, uncovering a hidden AWS pricing edge case for cross-AZ data transfers. Through architectural re-evaluation and direct AWS escalation, Duckbill helped Honeycomb reclaim 10% of their entire operating budget within weeks, balancing reliability with cost efficiency. Ready to lower your AWS bill? Yes, Please! #### Hornblower Cruises & Events URL: https://www.duckbillhq.com/customers/hornblower-cruises-events/ #### Hypergrowth Security Company Case Study | hypergrowth security company Hypergrowth Security Company Saves $63 Million with Duckbill A leading cybersecurity platform provider facing rapid 100% annual growth needed to negotiate a $300M AWS contract spanning both customer and partner relationships. By partnering with Duckbill, they secured $63M in effective discounts—a 21% improvement over AWS’s initial offer. The comprehensive negotiation strategy included cross-service discounting, service-specific optimizations, and strategic program incentives that the company wouldn’t have discovered independently. From simple vendor to strategic partner Most companies start using AWS in the context of a relatively straightforward vendor relationship—you use their services and pay the list price for the resources you use. But with a sustained 100% annual growth rate, this cybersecurity company correctly sensed an opportunity to negotiate a much more favorable deal with its upcoming contract renewal. While the leader tasked with the negotiation had navigated AWS contracts before, this was a different, multi-headed beast. The new contract would establish a complex relationship with AWS spanning their two distinct roles: Customer: The traditional role, purchasing AWS services for internal operations, as they had been doing Partner: Selling their product through AWS Marketplace and co-selling with AWS sales teams  With AWS being so integral to the client’s product, they knew they needed additional expertise to get the best possible deal.  Bring in the experts The client recognized the value of bringing in experts. “As a company, you’ll never get enough at-bats to see multiple negotiations. So you should bring somebody in who has gone through it dozens of times, all the various iterations, not to mention all the contacts within the AWS teams.” A colleague who had previously worked with Duckbill at another company made a strong recommendation. “This guy said we just need to call Duckbill. To him, it was an absolute no-brainer. The cost-benefit is a slam dunk. I just knew it was going to be good, and we’re going to get all kinds of valuable insight.” Deep dive analysis For a company in a hypergrowth stage, predicting AWS usage three years out is difficult. Overcommit, and you’ll owe shortfall fees. Undercommit, and you’ll leave available discounts on the table. You have to accurately predict a moving target.  Duckbill conducted an extensive usage analysis of all services. “Duckbill analyzed the hell out of it,” the client explained.  “We got all that analysis back as detailed spreadsheets where we could adjust all the variables.” With no stone left unturned, the analysis gave the client the confidence to move forward. Multi-layered etrategy for maximum savings Duckbill advised the client on a multi-layered negotiation approach to find all possible savings. The strategy employed both cross-service discounting along service-specific discounting with individual AWS high-usage services.    As the client explained, “We had very high spend on some different services that we leveraged. That’s how it became so detailed—we negotiated each of those different things separately.” Beyond the negotiated discounts, Duckbill identified additional credits and incentives tied to specific AWS programs like migrations and new service adoption, further reducing the overall bill on top of the discounts. “We wouldn’t have known about those without Duckbill,” the client said. On the flip side, not every service warrants negotiation effort, and knowing where to focus saves everyone time and energy. “Duckbill could say, ‘Hey, we’ve done this before. It’s a waste of your time to negotiate on such and such component, because AWS won’t budge in this case.’ Okay, great. That just saved us 10 hours.” The client team led the negotiations, with Duckbill providing strategic support. Rather than treating AWS as an adversary, Duckbill coached the team on partnership-oriented discussions that aligned goals and created win-win outcomes. When AWS manages your entire infrastructure, they’re essentially your partner. Collaborative approaches work far better than adversarial ones. No money left on the table The client’s choice to bring in Duckbill paid off. The engagement delivered $63M in effective discounts off the initial offer on their $300M contract, representing an improvement of 21%. “The single biggest highlight was the confidence that we didn’t leave money on the table,” the client emphasized. The value extended beyond immediate cost savings into strategic insights and opportunities the company wouldn’t have discovered otherwise. “Three or four little things Duckbill spotted easily paid for the cost of bringing in a third party. I can’t see us finding any of those smaller details if we were running the negotiation on our own,” he said. Duckbill’s value was clear. “At no point were we talking about what Duckbill charged… it was so far in the rearview mirror. We could stay focused on the project and not on the negotiation price.” “I would still call Duckbill” When asked about recommending Duckbill, the client was unequivocal: “A hundred percent. Even now, having experienced an AWS negotiation, if you were to throw another few hundred million dollar deal in front of me, I would still call Duckbill,” he said. Overview Client: Leading cybersecurity platform provider AWS agreement size: $300M 3-year EDP/PPA Situation The VP of Partnerships at a fast-growing cybersecurity company needed to negotiate a new AWS contract worth $300M before any discounts. The company had a reasonably complex AWS relationship, acting as both an AWS customer and partner. Solution By turning to Duckbill, the client improved the AWS contract by 21% compared to AWS’s initial offer. Duckbill coached the company through a comprehensive negotiation strategy that included: Qualitative and quantitative benchmark data A comprehensive deal strategy that established negotiation priorities, aligned internal stakeholders, and maximized leverage throughout the process Detailed usage modeling and risk assessment for multi-year commitments Expert guidance on pursuing a meaningful cross-service discount along with service-specific discounting for high-usage individual services like Amazon’s Simple Queue Service (SQS) Strategic identification of additional credits and program incentive opportunities Ready to lower your AWS bill? Yes, Please! #### Instana Case Study | INSTANA Instana optimized its cloud architecture and cut 25% off its cloud hosting bill This data-intensive software company partnered with The Duckbill Group and identified optimizations in its hybrid environment that contributed to a 25% reduction in its cloud hosting bill in less than a month.. Impact Client: Instana, Application Performance Monitoring (APM) and Observability for microservices Number of employees: 165 Customer growth YoY: 1.5x ARR growth YoY: 200% About Instana  Instana offers an Application Performance Monitoring (APM) and Observability solution that can manage cloud-native microservice applications. Customers use Instana to manage applications and ensure top performance and availability of services throughout their lifecycles. With Applications growing more complex, monitoring them has become more business-critical, but at the same time, more difficult and complex to do well. To meet customer demands for up-time and optimum performance, Instana helps Dev+Ops teams by automatically discovering each component of the Application infrastructure and continuously monitoring the performance of applications and services continuously. A mandate to streamline operational costs Instana’s solution captures a vast amount of application data while its customer base is also growing quickly. The company is collecting more and more data from its customers’ environment and, with this, Instana watched its infrastructure and operational costs rise as quickly as revenue. Fabian Lange, Instana founder and VP of Engineering, took charge of reining in these operational costs — fast.  “We needed to optimize our margins to drive both innovation and growth,” he said. “We wanted to optimize as many items as we could, as quickly as we could—like in a few weeks.” The plan: Identify ways to reduce cloud hosting costs Lange first tackled the ever-growing Amazon Web Services (AWS) bill. Initially, Lange’s engineering team investigated the company’s AWS cost-optimization issues on their own. They were happy with the progress they were making, but Lange didn’t want this work to remain his team’s top priority. He wanted them to focus on what they do best—product innovation and new feature delivery to customers. Still, he was concerned that an outside firm, one that didn’t know Instana’s infrastructure as deeply as his team does, would struggle to find any meaningful optimizations and slow the process down. “Every day we waited to optimize, we burned a lot of money,” Lange said. Deep expertise in cloud architecture and hosting accelerates time to savings Lange’s team suggested he reach out to The Duckbill Group.  “We knew Duckbill’s reputation as the industry experts,” Lange said, “and with their satisfaction guarantee, if they weren’t able to help us, we wouldn’t lose any money.” Instana hired Duckbill for a fast-paced engagement where everyone immediately got to work on change implementation. The Instana team, with offices all over the world, onboarded Duckbill in a few meetings. The team explained their product architecture, which parts of it could and could not change, and briefed Duckbill on the work they had already completed themselves up to that point.  For Duckbill, understanding product architecture is key to making useful, money-saving recommendations. “We learned in our first meetings that Instana had hard constraints on the product database,” explained Corey Quinn, Chief Cloud Economist at Duckbill Group. “The application runs in a variety of cloud providers and data centers, which isn’t unusual. So we looked at the components that surround that system and tailored our recommendations accordingly.” At the start of work, the Duckbill team set up a Slack channel so the internationally distributing team at Instana (mainly US and Germany) could communicate effectively with the Duckbill team and bounce ideas off each other. When Duckbill suggested an optimization change, Lange’s team could implement it right away, immediately saving the company money. Know when to say when: The value in validating cloud architecture design The Duckbill team understood Instana wanted to reduce costs where possible, but also knew that the priority of any engineering team is to deliver value to customers in the form of new features and a stable product. “One of the most important things we offer to our clients is not just advice on how to save money on cloud hosting, but also tell them when to stop hunting for things to cut,” said Duckbill CEO Mike Julian. “There is an upper limit to how much you can save in cloud hosting costs. Engineering teams should spend their time focusing on delivering the right features, which can be worth a hundred times more in revenue than any single cost-savings measure.” Lange appreciated the architectural validation Duckbill offered—and also the recommendations for change, especially how they were delivered. “We had very open communication, very direct,” Lange said. “It was really more like having an extended team working together. It never felt like, ‘Oh, you are the external consultancy and you’ll lecture us at the end of the two weeks about what we did wrong,’” he said. “Duckbill was very solution-oriented. Everybody liked working with them.” Related Reducing AWS EBS Volume Cost — Lessons from an Instana SRE Overview Client: Instana, Application Performance Monitoring (APM) and Observability platform Number of employees: Approximately 165 Situation This fast-growing APM software company needed to optimize cloud hosting costs to support rapid growth (200% ARR YoY), increase margins, and maintain focus on product innovation rather than infrastructure cost management.s. Solution Duckbill Group recommended and validated architecture changes in Instana's hybrid cloud environment that immediately contributed to a 25% reduction in cloud hosting costs within weeks, allowing engineering teams to refocus on feature delivery and innovation. Ready to lower your AWS bill? Yes, Please! #### Lendable Case Study | LENDABLE Lendable scales their technical system as data grows Lendable provides access to credit for growing African companies, having disbursed more than $30mm USD since inception. As every overhead is a major concern for Lendable, we helped optimize their AWS bill with an eye toward their future growth plans. From quick wins to long-term cost strategy Our AWS bill is a significant cost line item for us. As the scale of our data grows, we need to make sure we’re scaling our technical systems along with it and that we’re doing it in a cost effective way. Duckbill came recommended to us and it seemed like a good opportunity to get some expert advice on the issue. Our tech team is spread between Austin, New York, and Nairobi. Working with Duckbill provided a great opportunity for our tech team from across the world to interact with a wide set of technical expertise. Duckbill gave us recommendations on specific changes that we can make to control our AWS costs, identifying 20-27% in savings. Even before they delivered the final report, they pointed out some low-hanging fruit we could address in the meantime. It helped to have someone external say “Hey, are you actually using X service? It doesn’t look like that’s getting hit, but you are paying for it.” We’ve gone through and turned some of those services off to reduce costs. They gave us tips on tools to increase security as well as tools to use for logging and monitoring our infrastructure. They also advised on how to evaluate the options of going with an AWS service versus an external service versus rolling our own solutions, offering tips on what they see other companies doing and how we might think about that.  Real-time answers for resource-constrained teams Having access to Duckbill to ask questions about AWS infrastructure saved us time by helping us avoid running down dead ends. With a small technical team and limited resources to support our data pipeline, anytime that we can save time, that’s the biggest benefit for us because it’s all about opportunity costs. So being able to short-circuit certain ideas on things to implement and say, “Nope, that’s not worth spending any time on,” or “yes, you can do that and here’s how” helps us a lot. They were very easy to communicate with. We had access to them directly in our Slack to send them quick messages asking questions about our AWS infrastructure. For our calls, they were extremely accommodating of the pretty broad set of times zones we’re spread out across. They took meetings at bizarre times of the morning for them on the West Coast, so we really appreciated their flexibility. I certainly would recommend that others work with Duckbill. They are experts in AWS and there’s no question that they are able to help reduce costs. Overview Client: Lendable, a fintech providing credit access to African companies Situation This growing fintech company with a distributed team across three continents (Austin, New York, Nairobi) needed to optimize AWS costs as their data scaled, while avoiding dead ends and maximizing their small technical team's time and resources. Solution Duckbill identified 20-27% in AWS savings through immediate quick wins and long-term architectural guidance. Direct Slack access and flexible cross-timezone support helped Lendable's resource-constrained team short-circuit inefficient approaches and build cost-effective, scalable infrastructure for future growth. Ready to lower your AWS bill? Yes, Please! #### Office hours Duckbill office hours You have questions. We have coffee. Meet our team and get personalized advice on your most pressing AI and cloud cost challenges. Office hours: Thursdays 10 - 10:30 am PST / 1 - 1:30 pm EST Join us on Thursdays AI & Cloud Contract Negotiation Office hours We can help with your compute contracts. Join our weekly session for personalized advice and expert recommendations on hyperscalers, model providers, and beyond. Our specialists can advise on: Strategy Forecasting Contract management And more You don’t have to go it alone—and, in fact, you shouldn’t. Let’s talk. Register now #### Platform Contract clarity and actionable intelligence for your AI and cloud costs Because spreadsheets only get you so far Complex spend and usage data shouldn’t live in siloed spreadsheets and tools that break under scale. Skyway helps you negotiate, manage, and forecast your AI and cloud contracts, giving you the intelligence to manage your commitments proactively and never leave a dollar on the table. Schedule a demo Finally, know what's happening and why Skyway is the only compute financial management platform that is contract-aware. It models your contracts and your consumption together, so every number comes with context and your next move. Track every commitment, everywhere One registry for everything you've signed: hyperscaler commitments, prepaid credits, committed-spend tiers, and reserved capacity. See burn-down and projected end-of-term position for every agreement Catch under- and over-consumption risk while you can still act on it Stay ahead of renewals with runway to negotiate, not react Know whether your deal is competitive A rate card isn’t a benchmark. Skyway pairs the market's public rates with proprietary benchmark data, then ties both to your contracts and consumption. Benchmark your effective rates against retail and the private market See which discounts you qualify for and should be requesting Get your next move, not just your numbers Value that grows as you scale Treat commitments like the financial positions they are A compute commitment isn't a set-and-forget purchase; it's a position that moves as your usage, model mix, and the market change. Skyway tracks how every agreement performs against reality, so you can adjust, renegotiate, or expand at exactly the right moment. Bring rigor to your biggest bets Replace ad-hoc dealmaking and gut-feel commitments with a repeatable, defensible process, the same discipline you'd demand for any other position of this size on your balance sheet. Recover the money you already earned Every invoice from every provider gets checked against your contract and your usage. Skyway flags misapplied discounts, billing errors, and credits that never landed, and hands you the evidence to dispute them. #### Privacy Policy Privacy Policy Last updated: April 1, 2026 At L9 Labs, Inc. dba Duckbill ("Duckbill", “we”, or “our”), we take your privacy seriously. By using or accessing our Services (defined below) in any manner, you acknowledge that you accept the practices and policies outlined in this Privacy Policy, and you understand that we will collect, use, and share your information as described in this Privacy Policy. What this Privacy Policy Covers This Privacy Policy explains how we collect, use, disclose, and process your personal data when you use our website and interact with Skyway or other Duckbill products as a consumer for personal use (“Services") or when Duckbill operates and provides our commercial customers and their end users with access to our commercial products, such as a Skyway enterprise plan (“Commercial Services”). “Personal Data” means any information that identifies or relates to a particular individual and also includes information referred to as “personally identifiable information” or “personal information” under applicable data privacy laws, rules or regulations.  For clarity, this Privacy Policy does not apply where Duckbill processes personal data on behalf of commercial customers using our Commercial Services – for example, your employer has provisioned you with a Skyway account. In those cases, the commercial customer - such as your employer - is the controller of your personal data, and you can review their policies for more information about how they handle your personal data. This Privacy Policy also does not cover the practices of companies we don’t own or control or people we don’t manage. Personal Data Categories of Personal Data We Collect This list details the categories of Personal Data that we collect and have collected over the past 12 months: Profile or Contact Data - such as first and last name, email address, and unique identifiers such as passwords that may contain Personal Data Payment Data - such as credit card number, bank account number, and billing address and phone number Device/IP Data - such as IP address, device ID, domain server, type of device/operating system/browser used to access the Services  Web Analytics - such as web page interactions, referring webpage/source through which you accessed the Services, non-identifiable request IDs, statistics associated with the interaction between your device or browser and the Services Other Identifying Information that You Voluntarily Choose to Provide - such as Personal Data that you voluntarily choose to send us in emails, chats, or other communications  Categories of Sources of Personal Data We collect Personal Data about you from the following categories of sources: From you: When you provide such information directly to us. When you create an account or use our interactive tools and Services. When you voluntarily provide information in free-form text boxes through the Services or through responses to surveys or questionnaires. When you send us an email or otherwise contact us. When you use the Services and such information is collected automatically. Through Cookies (as defined and described further below). If you use a location-enabled browser, we may receive information about your location, as applicable. From third parties: We may use analytics providers to analyze how you interact and engage with the Services, or third parties may help us provide you with customer support. Our Commercial or Business Purposes for Collecting Personal Data Providing, Customizing and Improving the Services - such as:  Creating and managing your account or other user profiles. Processing orders or other transactions; billing. Providing you with the products, services or information you request. Meeting or fulfilling the reason you provided the information to us. Providing support and assistance for the Services. Improving the Services, including testing, research, internal analytics, and product development. Personalizing the Services, website content, and communications based on your preferences. Conducting fraud protection, security and debugging. Carrying out other business purposes stated when collecting your Personal Data or as otherwise set forth in applicable data privacy laws. Marketing the Services - such as:  Marketing and selling the Services to you. Providing you with updates about the Services and special offers. Corresponding with You - such as:  Responding to correspondence that we receive from you, contacting you when necessary or requested, and sending you information about Duckbill or the Services. Sending emails and other communications according to your preferences or that display content that we think will interest you. Meeting Legal Requirements and Enforcing Legal Terms - such as:  Fulfilling our legal obligations under applicable law, regulation, court order or other legal process, such as preventing, detecting and investigating security incidents and potentially illegal or prohibited activities. Protecting the rights, property, or safety of you, Duckbill, or another party. Enforcing any agreements with you. Responding to claims that any posting or other content violates third-party rights. Resolving disputes. Business Transfers -  If we undergo a merger, acquisition, bankruptcy or other transaction in which a third party assumes control of our business (in whole or in part); provided, however, should one of these events occur, we will make reasonable efforts to notify you before your information becomes subject to different privacy and security policies and practices.  We will not collect additional categories of Personal Data or use the Personal Data we collected for materially different, unrelated, or incompatible purposes without providing you notice. How We Share Your Personal Data We disclose your Personal Data to the categories of parties listed in this section.  Service Providers These parties help us provide the Services or perform business functions on our behalf, such as: Hosting, technology, and communication providers. Security and fraud prevention consultants. Analytics providers, including those listed in our Trust Center. Support and customer service vendors. Product fulfillment and delivery providers. Payment processors. Our payment processing partner collects your voluntarily-provided payment card information necessary to process your payment. Please see Stripe’s and Intuit’s terms of service and privacy policy for information on its use and storage of your Personal Data. Business Partners These parties partner with us in offering various services, such as: Businesses that you have a relationship with. Companies that we partner with to offer joint promotional offers or opportunities. Parties You Authorize, Access or Authenticate - such as:  Third parties you access through the services. Social media services. Other users. Data that is Not Personal Data We may create aggregated, de-identified or anonymized data from the Personal Data we collect, including by removing information that makes the data personally identifiable to a particular user. We may use such aggregated, de-identified or anonymized data and share it with third parties for any lawful business purposes, including, without limitation, to analyze, build, and improve the Services and promote our business, provided that we will not share such data in a manner that could identify you. California Resident Rights Under California Civil Code Sections 1798.83-1798.84, California residents are entitled to contact us to prevent disclosure of Personal Data to third parties for such third parties’ direct marketing purposes; in order to submit such a request, please contact us at support@duckbillhq.com. Tracking Tools and Opt-Out The Services use cookies and similar technologies such as pixel tags, web beacons, clear GIFs and JavaScript (collectively, “Cookies”) to enable our servers to recognize your web browser, tell us how and when you visit and use our Services, analyze trends, learn about our user base and operate and improve our Services. Cookies are small pieces of data– usually text files – placed on your computer, tablet, phone or similar device when you use that device to access our Services. We may also supplement the information we collect from you with information received from third parties, including third parties that have placed their own Cookies on your device(s). Please note that because of our use of Cookies, the Services do not support “Do Not Track” requests sent from a browser at this time. We use the following types of Cookies: Essential Cookies. Essential Cookies are required for providing you with features or services that you have requested. For example, certain Cookies enable you to log into secure areas of our Services. Disabling these Cookies may make certain features and services unavailable. Functional Cookies. Functional Cookies are used to record your choices and settings regarding our Services, maintain your preferences over time and recognize you when you return to our Services. These Cookies help us to personalize our content for you, greet you by name and remember your preferences (for example, your choice of language or region). Performance/Analytical Cookies. Performance/Analytical Cookies allow us to understand how visitors use our Services. They do this by collecting information about the number of visitors to the Services, what pages visitors view on our Services and how long visitors are viewing pages on the Services. Performance/Analytical Cookies also help us measure the performance of our advertising campaigns in order to help us improve our campaigns and the Services’ content for those who engage with our advertising. You can decide whether or not to accept Cookies through your internet browser’s settings. Most browsers have an option for turning off the Cookie feature, which will prevent your browser from accepting new Cookies, as well as (depending on the sophistication of your browser software) allow you to decide on acceptance of each new Cookie in a variety of ways. You can also delete all Cookies that are already on your device. If you do this, however, you may have to manually adjust some preferences every time you visit our website and some of the Services and functionalities may not work. To explore what Cookie settings are available to you, look in the “preferences” or “options” section of your browser’s menu. To find out more information about Cookies, including information about how to manage and delete Cookies, please visit allaboutcookies.org. Data Security and Retention We seek to protect your Personal Data from unauthorized access, use and disclosure using appropriate physical, technical, organizational and administrative security measures based on the type of Personal Data and how we are processing that data. You should also help protect your data by appropriately selecting and protecting your password and/or other sign-on mechanism; limiting access to your computer or device and browser; and signing off after you have finished accessing your account. Although we work to protect the security of your account and other data that we hold in our records, please be aware that no method of transmitting data over the internet or storing data is completely secure. We retain Personal Data about you for as long as you have an open account with us or as otherwise necessary to provide you with our Services. In some cases we retain Personal Data for longer, if doing so is necessary to comply with our legal obligations, resolve disputes or collect fees owed, or is otherwise permitted or required by applicable law, rule or regulation. We may further retain information in an anonymous or aggregated form where that information would not identify you personally. Accessing and Correcting Your Information   You can review and change some of the Personal Data that we have collected about you by logging into the Services and visiting your account profile page. You may also send us an email at support@duckbillhq.com to request access to, correct, or delete any Personal Data that you have provided to us. We may not accommodate a request to change information if we believe the change would violate any law or legal requirement or cause the information to be incorrect. Personal Data of Children The Services are not intended for children under 13 years of age and you are not allowed to use the Services or provide information on it if you are under 13 years of age. We do not knowingly collect or solicit Personal Information from anyone under the age of 13. If we learn that we have collected Personal Information from a child under age 13, we will delete that information as quickly as possible. If you believe that a child under 13 may have provided us Personal Data, please contact us at support@duckbillhq.com. European Union Data Subject Rights EU Residents If you are a resident of the European Union (“EU”), United Kingdom, Lichtenstein, Norway or Iceland, you may have additional rights under the EU General Data Protection Regulation (the “GDPR”) with respect to your Personal Data, as outlined below. For this section, we use the terms “Personal Data” and “processing” as they are defined in the GDPR, but “Personal Data” generally means information that can be used to individually identify a person, and “processing” generally covers actions that can be performed in connection with data such as collection, use, storage and disclosure. Duckbill will be the controller of your Personal Data processed in connection with the Services. If there are any conflicts between this section and any other provision of this Privacy Policy, the policy or portion that is more protective of Personal Data shall control to the extent of such conflict. If you have any questions about this section or whether any of the following applies to you, please contact us at support@duckbillhq.com. Note that we may also process Personal Data of our customers’ end users or employees in connection with our provision of certain services to customers, in which case we are the processor of Personal Data. If we are the processor of your Personal Data (i.e., not the controller), please contact the controller party in the first instance to address your rights with respect to such data. Personal Data We Collect The “Categories of Personal Data We Collect” section above details the Personal Data that we collect from you. Personal Data Use and Processing Grounds The “Our Commercial or Business Purposes for Collecting Personal Data” section above explains how we use your Personal Data. We will only process your Personal Data if we have a lawful basis for doing so. Lawful bases for processing include consent, contractual necessity and our “legitimate interests” or the legitimate interest of others, as further described below. Contractual Necessity:  We process your Personal Data as a matter of “contractual necessity”, meaning that we need to process the data to perform under our Terms of Use with you, which enables us to provide you with the Services. When we process data due to contractual necessity, failure to provide such Personal Data will result in your inability to use some or all portions of the Services that require such data.  Legitimate Interest:  We process your Personal Data when we believe it furthers the legitimate interest of us or third parties. We may also de-identify or anonymize Personal Data to further our legitimate interests. Examples of these legitimate interests include: Providing, customizing and improving the Services. Marketing the Services. Corresponding with you. Meeting legal requirements and enforcing legal terms. Completing corporate transactions. Consent:  In some cases, we process Personal Data based on the consent you expressly grant to us at the time we collect such data. When we process Personal Data based on your consent, it will be expressly indicated to you at the point and time of collection. Other Processing Grounds:  From time to time we may also need to process Personal Data to comply with a legal obligation, if it is necessary to protect the vital interests of you or other data subjects, or if it is necessary for a task carried out in the public interest. Sharing Personal Data The “How We Share Your Personal Data” section above details how we share your Personal Data with third parties. EU Data Subject Rights You have certain rights with respect to your Personal Data, including those set forth below. For more information about these rights, or to submit a request, please email us at support@duckbillhq.com. Please note that in some circumstances, we may not be able to fully comply with your request, such as if it is frivolous or extremely impractical, if it jeopardizes the rights of others, or if it is not required by law, but in those circumstances, we will still respond to notify you of such a decision. In some cases, we may also need you to provide us with additional information, which may include Personal Data, if necessary to verify your identity and the nature of your request. Access:  You can request more information about the Personal Data we hold about you and request a copy of such Personal Data. You can also access certain of your Personal Data by logging on to your account. Rectification:  If you believe that any Personal Data we are holding about you is incorrect or incomplete, you can request that we correct or supplement such data. You can also correct some of this information directly by logging on to your account. Erasure:  You can request that we erase some or all of your Personal Data from our systems. Withdrawal of Consent:  If we are processing your Personal Data based on your consent (as indicated at the time of collection of such data), you have the right to withdraw your consent at any time. Please note, however, that if you exercise this right, you may have to then provide express consent on a case-by-case basis for the use or disclosure of certain of your Personal Data, if such use or disclosure is necessary to enable you to utilize some or all of our Services. Portability:  You can ask for a copy of your Personal Data in a machine-readable format. You can also request that we transmit the data to another controller where technically feasible. Objection:  You can contact us to let us know that you object to the further use or disclosure of your Personal Data for certain purposes, such as for direct marketing purposes. Restriction of Processing:  You can ask us to restrict further processing of your Personal Data. Right to File Complaint:  You have the right to lodge a complaint about Duckbill’s practices with respect to your Personal Data with the supervisory authority of your country or EU Member State. A list of Supervisory Authorities is available here: https://edpb.europa.eu/about-edpb/board/members_en. Transfers of Personal Data The Services are hosted and operated in the United States (“U.S.”) through Duckbill and its service providers, and if you do not reside in the U.S., laws in the U.S. may differ from the laws where you reside. By using the Services, you acknowledge that any Personal Data about you, regardless of whether provided by you or obtained from a third party, is being provided to Duckbill in the U.S. and will be hosted on U.S. servers, and you authorize Duckbill to transfer, store and process your information to and in the U.S., and possibly other countries.  You hereby consent to the transfer of your data to the U.S. pursuant to: (i) a data processing agreement incorporating standard data protection clauses promulgated by the European Commission, a copy of which can be obtained at https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A32010D0087, (ii) binding corporate rules for data protection that align with the GDPR’s requirements, or (iii) adherence to an industry- or technology-specific approved code of conduct blessed by the European Commission. Changes to this Privacy Policy We’re constantly trying to improve our Services, so we may need to change this Privacy Policy from time to time, but we will alert you to any such changes by placing a notice on the Duckbill website, by sending you an email, and/or by some other means. Please note that if you’ve opted not to receive legal notice emails from us (or you haven’t provided us with your email address), this Privacy Policy will still govern your use of the Services, and you are still responsible for reading and understanding this Privacy Policy. If you use the Services after any changes to the Privacy Policy have been posted, that means you agree to all of the changes. Use of information we collect is subject to the Privacy Policy in effect at the time such information is collected. Contact Information If you have any questions or comments about this Privacy Policy, the ways in which we collect and use your Personal Data or your choices and rights regarding such collection and use, please do not hesitate to contact us at: Email (including Data Protection Officer): support@duckbillhq.com Address: 548 Market St #88528, San Francisco, CA 94110 #### Procurement Use Cases | Procurement Master compute contract procurement You're accountable for compute contracts worth tens to hundreds of millions, and you're measured on the savings and cost avoidance you deliver against them. The mix is getting harder, not easier: hyperscaler commitments now sit alongside model-lab agreements, inference pricing, and neocloud capacity deals, each layered and harder to benchmark than anything else in your portfolio. Duckbill pairs Skyway, our contract-aware compute position management platform, with hands-on negotiation expertise, so you can source, benchmark, and negotiate every deal from a position of strength. Benchmarks built from real deals Generic benchmark data won't tell you whether your deal is competitive for your usage and scale. Skyway maintains the market's pricing data, public rates, and proprietary private benchmarks alike, and scores your agreements against it. Know exactly where your deal stands and what to ask for before the vendor tells you what to think. Decide where to buy, then leave nothing on the table The same model or GPU capacity can often be sourced several ways: direct from a provider, through a hyperscaler, or against commitments you already hold. That choice can move the economics more than the rate itself. Skyway shows you which purchases count toward which commitments, including marketplace purchases, so you can weigh the sourcing decision on real numbers, then negotiate knowing which levers drive pricing and value. Walk into every negotiation prepared like a specialist Skyway assembles your negotiation data room automatically, pulling your usage, forecasts, and benchmarks into one place. It prioritizes your asks by expected value and models counter-proposals in real time as the vendor responds. It's the preparation a seven-figure consulting engagement would produce. And when you want experts in the room with you, our negotiators work from the same playbook. Make your results impossible to miss Savings and cost avoidance only count when the business sees them. Skyway validates every invoice against contract terms, recovering misapplied discounts and credits that never landed, and quantifies the value of every deal you close in language leadership respects. Your wins, documented and defensible. Related case studies The best contract for you and your business — at all times Your job is getting the most out of every contract, not chasing data across siloed teams, spreadsheets, and ever-changing provider pricing. Let us handle the technical complexity so you can focus on the best deal for your business. Contact us #### San Francisco FinOps Meetup San Francisco FinOps Meetup Lively discussion with your fellow cost management nerds.  Hosted by Duckbill, this gathering brings together practitioners from companies all throughout the Bay Area to talk about cost management. Join us! This meetup is for anyone navigating the messy reality of cloud financial management – where engineering wants to move fast and finance wants to understand why the bill doubled last month. We'll tackle real-world challenges like getting engineers to care about cost without killing velocity, building chargeback models that don't make everyone hate you, and explaining Reserved Instance strategies to executives who think "the cloud" means it's free. Who should attend? Anyone tired of being the "no" person when engineers want to spin up new services Finance professionals who've been handed AWS bills and told to "figure it out" Engineering leaders who know their team's cloud spending is out of control but aren't sure where to start FinOps practitioners who want to commiserate with people facing the same battles No need to bring spreadsheets or cost allocation strategies – just your war stories and questions. We'll provide food, honest conversation, and the relief of knowing you're not the only one dealing with this. Subscribe on Lu.ma #### Scribd Case Study | SCRIBD Scribd developed a cost model for their infrastructure migration to AWS Scribd is a born-in-the-datacenter SaaS company making a major leap forward in both the engineering capabilities and their business overall. Not wanting to make an expensive mistake, they sought out Duckbill for assistance with modeling the cost of a migration to AWS and ongoing architecture+cost assistance. R. Tyler Croy, Director of Platform Engineering, had this to say: We only get one shot at this migration We’ve been running our app in a managed datacenter and now need to move to AWS. We’re only going to do this migration once and we want to get it right the first time without burning through cash as we do it. When you’re talking about millions of dollars a year in investment in AWS, making mistakes is expensive. Plus, we only have the capacity to pull away engineers from product development for this migration for a very limited time, so that means we have to know what the hell we’re doing pretty clearly ahead of time. Internally, Scribd didn’t have the skills or exposure to AWS architecture to do this on our own, so we engaged Duckbill to help us plan our migration. They have the expertise in advanced AWS architecture and the understanding of what it should look like for big companies like ours to help us avoid making expensive mistakes. Duckbill started with an assessment of what we have in the datacenter. They spent a ton of time with us on-site and remotely to understand our application and all the unique constraints we have with it, ultimately assessing its cloud readiness and pointing out where we needed to focus improvement efforts. Based on their assessment of the data, Duckbill developed a cost model for us (on a per-team basis and even down to the machine) so we could see what the cost was going to look like in AWS when we migrate and as we grow over the next few years.  When your labor costs are 5x your infrastructure spend We’re now ready to proceed with the migration and feel confident that we’ve gotten it right. Without Duckbill’s involvement, we would not have been able to get to this point, certainly not this year. That has a huge monetary impact on us. As part of the cost modelling, Duckbill came up with a meaningful unit economic KPI for us to use: cost per thousand document views. It’s helped bridge the gap between how Finance perceives what we’re doing and how Engineering and Product perceives it. Finance can tie that metric to their annualized budget, while I can come from the other direction to identify how infrastructure changes we plan to make will impact that number. It just gives us a meeting place in the middle as we talk about costs. If, for example, you present an engineer or engineering lead with a choice of the myriad ways in which containers can be deployed in AWS, they may not have a full appreciation of why you would choose one of the other, not just in the raw cost impacts of each but also the labor impacts of scaling and managing each one. With Duckbill, it’s like we’re timesharing an AWS Architect. They’ve got that experience to tell us the cost impacts—including the undocumented costs—and why we should choose one over the other. We’re keeping Duckbill on retainer for our migration and it’s going to continue to pay dividends. As we move forward with the migration and have to make architectural changes for some of our applications, we have someone more experienced that we trust to go to with questions when we’re not sure if we’re missing something. Having them confirm “yes, that’s how you have to do it” or “no, you don’t want to use that service because of these hidden costs”, it gives us tremendously valuable confidence and moral support. Overview Client: Scribd, a digital library and document-sharing platform Number of employees: Approximately 200-300 globally Situation This born-in-the-datacenter SaaS company needed to migrate to AWS without making expensive mistakes or pulling engineers away from product development for extended periods. With millions in annual infrastructure investment at stake, they lacked internal AWS architecture expertise for a one-time-only migration. Solution Duckbill conducted extensive onsite and remote assessments to develop a detailed cost model (per-team and per-machine), created a meaningful unit economic KPI (cost per thousand document views), and provided ongoing retainer support. This enabled confident migration planning while optimizing for labor cost savings—their largest expense at 5x infrastructure costs. Ready to lower your AWS bill? Yes, Please! #### Services Retainer Services | Services REtainer Build better FinOps Whether you’re unsure if your cost management program is effective, or you’re trying to figure out how to start one yourself Duckbill can help. We’re experienced engineers and finance experts who can tailor a cost management strategy to your needs. Our solutions are designed to fit your environment and evolve with your business goals. The experts We offer skilled professionals who provide targeted expertise tailored to your needs. We have cloud economists for cloud cost optimization, FinOps analysts for cloud cost management, and technical program managers to keep your FinOps initiatives on track. The tools Cost management tools are only as good as their implementation and maintenance. We handle the heavy lifting: integration, customization, and upkeep. If you already have tools in place, we optimize them for your environment. The operational focus We streamline your processes, enabling your team to stay productive without being distracted by cost optimization tasks. Stay focused on growing your core business while we handle the complexities of cloud cost management. What we provide Accelerated progress We specialize in helping organizations build a solid foundation for their FinOps team, whether that means embedding experts to jumpstart your efforts or coaching your team and helping them scale for future growth. We can even help you recruit and train new hires as you expand. Industry expertise Benefit from the knowledge of Duckbill experts who have worked with some of the largest and most complex cloud environments in the world. By leveraging best practices honed across diverse industries, we bring insights and solutions tailored to your unique challenges. Data you can act on Achieve precise forecasting, detailed unit economics, and effective tagging governance. We deliver clear insights and models tailored to your business, equipping teams at all levels to make decisions with confidence. FinOps experts at every level Cloud economists Cloud economists analyze and optimize cloud costs with insights tailored to your environment, providing executive-level guidance and strategy when needed. FinOps analysts FinOps analysts deliver precise cost tracking, robust governance, and actionable financial reporting to keep your budgets aligned.  Technical program managers Technical program managers oversee your FinOps initiatives, ensuring seamless coordination across teams, timely execution, and continuous efficiency improvements. Flexibility for today’s enterprise Engage our team full-time or part-time, with remote flexibility across time zones. Need on-site support? We’ve got you covered. Whether for short- or long-term engagements, we adapt to your operational needs. Contact us #### Terms & Policies Terms & Policies Commercial Terms of Service Website Terms of Service Data Processing Agreement Privacy Policy AI Addendum #### Website Terms Website Terms of Service #### Why Duckbill Why Duckbill Deep expertise for your biggest tech commitments Duckbill gives you the technical and financial specialization to negotiate, manage, and forecast compute. As your largest and fastest-growing area of strategic spend, you need a strategic, experienced partner trusted by not only the AI-native market leaders, but the hyperscalers, AI labs, and neoclouds themselves too. Technology-driven finance, not financial-led tech We're a software company with deep financial, cloud, and AI expertise, not a financial firm with a technology bolt-on. Duckbill builds the financial control layer that AI-native companies use to negotiate, manage, and forecast their spend across every cloud, model, neocloud, and inference provider. This is the discipline we pioneered, and it's all we do. We have the perfect marriage of software and expertise People struggle to make sense of complex compute and contract data without great software thanks to its size and complexity. Software, in turn, can untangle the complexity but can't make the judgment calls that high-stakes commitments require. Duckbill delivers both the purpose-built software that handles the data complexity, and seasoned practitioners who make the critical calls. It's the only combination that turns overwhelming compute data into clear next moves. Compute contracts require specialized expertise Generalist consultancies and analyst firms lack the technical depth and provider-specific specialization to level the playing field with hyperscalers, model labs, inference providers, and neoclouds. We can help you identify your best options for negotiating costs, providers, and availability. Best of the best,
every day Our solution delivers the software and world-class expertise you need. We hire the top talent in the market, and they're who you work with alongside product purpose-built for AI-native complexity. Instead of training up people new to these fields, we seek out the hidden experts and turn them loose on your toughest commitments. We act as a translator and connector between your teams and stakeholders Compute spend is now shared ground. Duckbill sits at the intersection of strategic finance, engineering, and procurement, making sure each team's needs are reflected equally — so the whole organization works from the same numbers across every provider. Built on trust - in our product and our expertise Companies hand us their largest, most complex commitments — that only happens on trust. It's earned two ways: expertise proven across billions in negotiated contracts, and a product that ties your contracts, consumption, and pricing together across every provider, so the numbers you act on are numbers you believe. Our customers and press coverage attest to both. Manage your compute at the level your business deserves Contact us ### Events #### AWS re:Invent Pre-Keynote Breakfast URL: https://www.duckbillhq.com/events/aws-reinvent-pre-keynote-breakfast/ #### AWS Summit LA URL: https://www.duckbillhq.com/events/aws-summit-la/ #### AWS Summit NYC URL: https://www.duckbillhq.com/events/aws-summit-nyc/ #### COP203: What’s New with AWS Cost Management URL: https://www.duckbillhq.com/events/cop203-whats-new-with-aws-cost-management/ #### Duckbill & RedMonk Atomic Liquors Drinkup @ re:Invent 2025 URL: https://www.duckbillhq.com/events/aws-reinvent-beers-with-redmonk/ #### For the AWS Bill is Dark and Full of Terrors URL: https://www.duckbillhq.com/events/for-the-aws-bill-is-dark-and-full-of-terrors/ #### In-Memory Data Reimagined URL: https://www.duckbillhq.com/events/in-memory-data-reimagined/ #### Negotiating with Hyperscalers URL: https://www.duckbillhq.com/events/negotiating-with-hyperscalers/ #### San Francisco FinOps Meetup URL: https://www.duckbillhq.com/events/san-francisco-finops-meetup/ #### San Francisco FinOps Meetup: Cloud Cost Insights & Networking URL: https://www.duckbillhq.com/events/san-francisco-finops-meetup-cloud-cost-insights-networking/ #### The myth of portability: why your cloud native app is married to your provider at KubeCon URL: https://www.duckbillhq.com/events/the-myth-of-portability-why-your-cloud-native-app-is-married-to-your-provider/